Data mining and statistical learning, lecture 3 Outline  Ordinary least squares regression  Ridge regression

Data mining and statistical learning, lecture 3 Ordinary least squares regression (OLS) x1x1 x2x2 xpxp … y Model: Terminology:  0 : intercept (or bias)  1, …,  p : regression coefficients (or weights) The response variable responds directly and linearly to changes in the inputs

Data mining and statistical learning, lecture 3 Least squares regression Assume that we have observed a training set of data Estimate the  coefficients by minimizing the residual sum of squares

Data mining and statistical learning, lecture 3 Matrix formulation of OLS regression Differentiating the residual sum of squares and setting the first derivatives equal to zero we obtain where and

Data mining and statistical learning, lecture 3 Parameter estimates and predictions Least squares estimates of the parameters Predicted values

Data mining and statistical learning, lecture 3 Different sources of inputs Quantitative inputs Transformations of quantitative inputs Numeric or dummy coding of the levels of qualitative inputs Interactions between variables (e.g. X 3 = X 1 X 2 ) Example of dummy coding:

Data mining and statistical learning, lecture 3 An example of multiple linear regression Response variable: Requested price of used Porsche cars (1000 SEK) Inputs: X 1 = Manufacturing year X 2 = Milage (km) X 3 = Model (0 or 1) X 4 = Equipment (1 2, 3) X 5 = Colour (Red Black Silver Blue Black White Green)

Data mining and statistical learning, lecture 3 Price of used Porsche cars Response variable: Requested price of used Porsche cars (1000 SEK) Inputs: X 1 = Manufacturing year X 2 = Milage (km)

Data mining and statistical learning, lecture 3 Interpretation of multiple regression coefficients Assume that and that the regression coefficients are estimated by ordinary least squares regression Then the multiple regression coefficient represents the additional contribution of x j on y, after x j has been adjusted for x 0, x 1, …, x j-1, x j+1, …, x p

Data mining and statistical learning, lecture 3 Confidence intervals for regression parameters Assume that where the X- variables are fixed and the error terms are i.i.d. and N(0,  ) Then where v j is the jth diagonal element of

Data mining and statistical learning, lecture 3 Interpretation of software outputs Adding new independent variables to a regression model alters at least one of the old regression coefficients unless the columns of the X -matrix are orthogonal, i.e.

Data mining and statistical learning, lecture 3 Stepwise Regression: Price (1000SEK) versus Year, Milage (km),... The p -value refers to a t -test of the hypothesis that the regression coefficient of the last entered x -variable is zero Classical statistical model selection techniques are model-based. In data-mining the model selection is data-driven.

Data mining and statistical learning, lecture 3 Stepwise Regression: Price (1000SEK) versus Year, Milage (km),... - model validation by visual inspection of residuals Residual = Observed - Predicted

Data mining and statistical learning, lecture 3 The Gram-Schmidt procedure for regression by successive orthogonalization and simple linear regression 1. Intialize z 0 = x 0 = 1 2. For j = 1, …, p, compute where  depicts the inner product (the sum of coordinate-wise products) 3. Regress y on z p to obtain the multiple regression coefficient

Data mining and statistical learning, lecture 3 Prediction of a response variable using correlated explanatory variables - daily temperatures in Stockholm, Göteborg, and Malmö

Data mining and statistical learning, lecture 3 Absorbance records for ten samples of chopped meat 1 response variable (protein) 100 predictors (absorbance at 100 wavelengths or channels) The predictors are strongly correlated to each other

Data mining and statistical learning, lecture 3 Absorbance records for 240 samples of chopped meat The target is poorly correlated to each predictor

Data mining and statistical learning, lecture 3 Ridge regression The ridge regression coefficients minimize a penalized residual sum of squares: or Normally, inputs are centred prior to the estimation of regression coefficients

Data mining and statistical learning, lecture 3 Matrix formulation of ridge regression for centred inputs If the inputs are orthogonal, the ridge estimates are just a scaled version of the least squares estimates Shrinking enables estimation of regression coefficients even if the number of parameters exceeds the number of cases Figure 3.7

Data mining and statistical learning, lecture 3 Ridge regression – pros and cons Ridge regression is particularly useful if the explanatory variables are strongly correlated to each other. The variance of the estimated regression coefficient is reduced at the expensive of (slightly) biased estimates

Data mining and statistical learning, lecture 3 The Gauss-Markov theorem Consider a linear regression model in which: –the inputs are regarded as fixed –the error terms are i.i.d. with mean 0 and variance  2. Then, the least squares estimator of a parameter a T  has variance no bigger than any other linear unbiased estimator of a T  Biased estimators may have smaller variance and mean squared error!

Data mining and statistical learning, lecture 3 SAS code for an ordinary least squares regression proc reg data=mining.dailytemperature outest = dtempbeta; model daily_consumption = stockholm g_teborg malm_; run;

Data mining and statistical learning, lecture 3 SAS code for ridge regression proc reg data=mining.dailytemperature outest = dtempbeta ridge=0 to 10 by 1; model daily_consumption = stockholm g_teborg malm_; proc print data=dtempbeta; run;