Regression Methods.

Slides:

Advertisements

Similar presentations

The Software Infrastructure for Electronic Commerce Databases and Data Mining Lecture 4: An Introduction To Data Mining (II) Johannes Gehrke

Advertisements

Kin 304 Regression Linear Regression Least Sum of Squares

CHAPTER 9: Decision Trees

Classification with Multiple Decision Trees

Stat 112: Lecture 7 Notes Homework 2: Due next Thursday The Multiple Linear Regression model (Chapter 4.1) Inferences from multiple regression analysis.

Logistic Regression.

6-1 Introduction To Empirical Models 6-1 Introduction To Empirical Models.

Regression Analysis Module 3. Regression Regression is the attempt to explain the variation in a dependent variable using the variation in independent.

Chapter 7 – Classification and Regression Trees

Chapter 7 – Classification and Regression Trees

Regression. So far, we've been looking at classification problems, in which the y values are either 0 or 1. Now we'll briefly consider the case where.

9. SIMPLE LINEAR REGESSION AND CORRELATION

Chapter 11 Multiple Regression.

Prediction with Regression Analysis (HK: Chapter 7.8) Qiang Yang HKUST.

PSY 307 – Statistics for the Behavioral Sciences Chapter 7 – Regression.

Classification and Prediction: Basic Concepts Bamshad Mobasher DePaul University Bamshad Mobasher DePaul University.

Simple Linear Regression Analysis

Classification and Prediction: Regression Analysis

Ensemble Learning (2), Tree and Forest

Microsoft Enterprise Consortium Data Mining Concepts Introduction to Directed Data Mining: Decision Trees Prepared by David Douglas, University of ArkansasHosted.

Introduction to Directed Data Mining: Decision Trees

1 1 Slide Simple Linear Regression Chapter 14 BA 303 – Spring 2011.

1 FORECASTING Regression Analysis Aslı Sencer Graduate Program in Business Information Systems.

Chapter 6 & 7 Linear Regression & Correlation

1 1 Slide Evaluation. 2 2 n Interactive decision tree construction Load segmentchallenge.arff; look at dataset Load segmentchallenge.arff; look at dataset.

Classification and Prediction (cont.) Pertemuan 10 Matakuliah: M0614 / Data Mining & OLAP Tahun : Feb

Chapter 9 – Classification and Regression Trees

LOGO Ensemble Learning Lecturer: Dr. Bo Yuan

Jeff Howbert Introduction to Machine Learning Winter Regression Linear Regression.

Multiple Linear Regression. Purpose To analyze the relationship between a single dependent variable and several independent variables.

Overview of Supervised Learning Overview of Supervised Learning2 Outline Linear Regression and Nearest Neighbors method Statistical Decision.

Regression Chapter 16. Regression >Builds on Correlation >The difference is a question of prediction versus relation Regression predicts, correlation.

Simple Linear Regression. The term linear regression implies that  Y|x is linearly related to x by the population regression equation  Y|x =  +  x.

Computational Intelligence: Methods and Applications Lecture 20 SSV & other trees Włodzisław Duch Dept. of Informatics, UMK Google: W Duch.

STATISTICS 12.0 Correlation and Linear Regression “Correlation and Linear Regression -”Causal Forecasting Method.

Chapter 11 Statistical Techniques. Data Warehouse and Data Mining Chapter 11 2 Chapter Objectives  Understand when linear regression is an appropriate.

Jeff Howbert Introduction to Machine Learning Winter Regression Linear Regression Regression Trees.

MACHINE LEARNING 10 Decision Trees. Motivation  Parametric Estimation  Assume model for class probability or regression  Estimate parameters from all.

Christoph Eick: Learning Models to Predict and Classify 1 Learning from Examples Example of Learning from Examples  Classification: Is car x a family.

Classification (slides adapted from Rob Schapire) Eran Segal Weizmann Institute.

CHAPTER 5 CORRELATION & LINEAR REGRESSION. GOAL : Understand and interpret the terms dependent variable and independent variable. Draw a scatter diagram.

Chapter1: Introduction Chapter2: Overview of Supervised Learning

STATISTICS 12.0 Correlation and Linear Regression “Correlation and Linear Regression -”Causal Forecasting Method.

Combining multiple learners Usman Roshan. Decision tree From Alpaydin, 2010.

Regression Methods. Linear Regression  Simple linear regression (one predictor)  Multiple linear regression (multiple predictors)  Ordinary Least Squares.

Method 3: Least squares regression. Another method for finding the equation of a straight line which is fitted to data is known as the method of least-squares.

Machine Learning in Practice Lecture 8 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute.

Tree and Forest Classification and Regression Tree Bagging of trees Boosting trees Random Forest.

Overfitting, Bias/Variance tradeoff. 2 Content of the presentation Bias and variance definitions Parameters that influence bias and variance Bias and.

Data Mining: Concepts and Techniques1 Prediction Prediction vs. classification Classification predicts categorical class label Prediction predicts continuous-valued.

Linear model. a type of regression analyses statistical method – both the response variable (Y) and the explanatory variable (X) are continuous variables.

Combining Models Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya.

Multiple Regression.

The simple linear regression model and parameter estimation

Data Transformation: Normalization

REGRESSION G&W p

Noisy Data Noise: random error or variance in a measured variable.

Generalized regression techniques

Kin 304 Regression Linear Regression Least Sum of Squares

Table 1. Advantages and Disadvantages of Traditional DM/ML Methods

BPK 304W Regression Linear Regression Least Sum of Squares

Overview of Supervised Learning

Regression Computer Print Out

Multiple Regression.

Data Mining for Business Analytics

Introduction to Predictive Modeling

Linear Model Selection and regularization

Statistical Learning Dong Liu Dept. EEIS, USTC.

Logistic Regression.

Advisor: Dr.vahidipour Zahra salimian Shaghayegh jalali Dec 2017

Presentation transcript:

Regression Methods

Linear Regression Simple linear regression (one predictor) Multiple linear regression (multiple predictors) Ordinary Least Squares estimation Computed directly from the data Lasso regression selects features by setting parameters to 0

Coefficient of Determination Indicates how well a model fits the data R2 (R squared) R2 = 1−SSres/SStot SSres = Σ(yi−fi)2 difference between actual and predicted SStot = Σ(yi−y)2 difference between actual and horizontal line Between 0 and 1, if least squares model. Bigger range if other models are used Explained variance what percentage of the variance is explained by the model Linear least squares regression: R2 = r2

R Squared visual interpretation of R2 SStot SSres Source Wikipedia CC BY-SA 3.0 SStot SSres

Regression Trees Regression variant of decision tree Top-down induction 2 options: Constant value in leaf (piecewise constant) regression trees Local linear model in leaf (piecewise linear) model trees

M5 algorithm (Quinlan, Wang) M5’, M5P in Weka (classifiers > trees > M5P) Offers both regression trees and model trees Model trees are default -R option (buildRegressionTree) for piecewise constant

M5 algorithm (Quinlan, Wang) Splitting criterion: Standard Deviation Reduction SDR = sd(T) – Σ sd(Ti)|Ti|/|T| Stopping criterion: Standard deviation below some threshold (0.05sd(D)) Too few examples in node (e.g. ≤ 4) Pruning (bottom-up): Estimate error: (n+v)/(n−v)×absolute error in node n is examples in node, v is parameters in the model

Binary Splits All splits are binary Numeric as normal (in C4.5) Nominal: order all values according to average (prior to induction) introduce k-1 indicator variables in this order Example: database of skiing slopes avg(color = green) = 2.5% avg(color = blue) = 3.2% avg(color = red) = 7.7% avg(color = black) = 13.5% binary features: Green, GreenBlue, GreenBlueRed,

Regression tree on Servo dataset (UCI)

Model tree on Servo dataset (UCI) LM1: 0.0833 * motor=B,A + 0.0682 * screw=B,A + 0.2215 * screw=A + 0.1315 * pgain=4,3 + 0.3163 * pgain=3 − 0.1254 * vgain=1,2 + 0.3864 smaller tree because more expressivity in leafs

Regression in Cortana Regression a natural setting in Subgroup Discovery Local models, no prediction model Subgroups are piecewise constant subsets h = 3100 h = 2200

Subgroup Discover: regression A subgroup is a step-function (inside subgroup vs. outside) R2 of step function is an interesting quality measure (next to z-score) available in Cortana as Explained Variance

Other regression models Functions LinearRegression MultiLayerPerceptron (artificial neural network) SMOreg (Support Vector Machine) Lazy IBK (k-Nearest Neigbors) Rules M5Rule (decision list)

Approximating a smooth function Experiment: take a mathematical function f (with infinite precision) generate a dataset by sampling x and y, and computing z = f(x,y) learn f by M5’ (regression tree)

k-Nearest Neighbor k-Nearest Neighbor can also be used for regression with all advantages and disadvantages