1. Multiple Linear Regression
Learning Objectives
Extend Simple Linear Regression concepts to regression with multiple explanatory variables
Apply the Matlab regression tools and interpret their output
Choose the variables to use in a multiple regression
Quantify the uncertainty of MLR predictions

2. Readings Kottegoda and Rosso, Chapter 6 (6.2)
Helsel and Hirsch, Chapters 9 and 11
Hastie, Tibshirani and Friedman, Chapter 3
Matlab Statistics Toolbox Users Guide, Chapter 6.

3. Multiple Linear Regression Model

4. Data for Multiple Linear Regression
Output
Input
Carrier Matrix
Residuals

5. Solving Multiple Linear Regression
Minimizing results in
(KR )
Vector of estimated mean values at each observation
Vector of Residuals
(KR )

6. Error Variance
(KR )
Sum of squares of observation deviations from the mean
Sum of squares of regression estimates deviations from the mean

7. Significance Tests on the Regression
Overall Significance
(KR )
Nested/Partial F Test (Significance of ‘new’ parameters)
(KR )
(HH p297)
Complicated model with p1 parameters versus simpler model with p0 parameters

8. Significance and confidence limits on regression parameters
(KR )
(KR )
[b,bint,r,rint,stats]=regress(Y,X);
b,bint
b =
bint =

9. Confidence limits on mean response
(KR )
(KR )

10. Confidence limits on individual future value
(KR )

11. Regression Diagnostics
Do not rely only on R2, F, SSE and T statistics.
(Read Helsel and Hirsch page 244 and 300)
Use graphical tools to diagnose MLR deficiencies
Partial Residual Plot

12. Diagnostic Plots

13. These figures have the same R2, SSE, and regression parameters, but (b) suffers from curvature, (c) an outlier and (d) high influence and leverage.
Diagnosed from plots of y vs x or e vs y^. Difficult to diagnose in MLR
Helsel and Hirsch page 245

14. The Hat Matrix [,1] [,2] [,3] [,4] [,5] [,6] [,7]
H is independent of the observed outputs (y). Linear regression predictions are a weighted average of the original y-values
[,1] [,2] [,3] [,4] [,5] [,6] [,7]
[1,]
[2,]
[3,]
[4,]
[5,]
[6,]
[7,]

15. Weights from the Hat matrix
Weights from the Hat matrix. Each line in the plot represents the weights used to determine the fitted y-value at the indicated point x y 7 1 7 6 5 4 2 6 3 2 1 3 5
Weight 4
x-value

16. Diagonals of the Hat Matrix Quantify the Leverage that a point has on the regression

17. Leverage MLR: Hat matrix H = SLR
outlier with high leverage but low influence
outlier with high leverage and high influence
Helsel and Hirsch page 246

18. Outliers are harder to detect in MLR
Standardized residual (Compare to Normal or t distribution)
(KR )
Prediction residual (leave one out estimate)
(HH p247)
Prediction Error Sum of Squares
(HH p247)
Studentized residual (compare to t distribution)
(HH p247)

19. Cook’s Distance: Leverage v. Actual Influence
The Hat matrix (hii) indicates the leverage of point i.
The leverage is not the same as the actual influence.
Actual influence is only realized if the predicted value is very different than the observed point.
Cook’s Distance (Outlier if > 1)
(KR )

20. Choosing Variables in MLR (Helsel and Hirsch p 309)
Stepwise regression (forward or backward based on F or t statistic). Best model not guaranteed
Plausible theory why variable should influence response
Evaluate all possibilities using overall measure of quality (HH p313)

21. Overall Measures of Quality
Mallow’s Cp
Prediction Error Sum of Squares
Adjusted R2
(HH p313)
(HH p247)
(HH p313)