1. 1 Experimental Statistics - week 12 Chapter 12: Multiple Regression Chapter 13: Variable Selection Model Checking

2. 2 Y X 1 X 2 4.3 4.2 5.5 5.2 6.8 6.2 8.0 7.2 4.0 4.3 5.2 5.3 6.6 6.3 7.5 7.3 2.0 4.4 4.0 5.4 5.7 6.4 6.5 7.4 Data – Page 628 Y = weight loss (wtloss) X 1 = exposure time (exptime) X 2 = relative humidity (humidity) Weight loss in a chemical compound as a function of exposure time and humidity

3. 3 The REG Procedure Dependent Variable: wtloss Number of Observations Read 12 Number of Observations Used 12 Analysis of Variance Sum of Mean Source DF Squares Square F Value Pr > F Model 2 31.12417 15.56208 104.13 <.0001 Error 9 1.34500 0.14944 Corrected Total 11 32.46917 Root MSE 0.38658 R-Square 0.9586 Dependent Mean 5.50833 Adj R-Sq 0.9494 Coeff Var 7.01810 Parameter Estimates Parameter Standard Variable DF Estimate Error t Value Pr > |t| Intercept 1 0.66667 0.69423 0.96 0.3620 exptime 1 1.31667 0.09981 13.19 <.0001 humidity 1 -8.00000 1.36677 -5.85 0.0002 Chemical Weight Loss – MLR Output

4. 4 Examining Contributions of Individual X variables Use t -test for the X variable in question. - this tests the effect of that particular independent variable while all other independent variables stay constant. Parameter Estimates Parameter Standard Variable DF Estimate Error t Value Pr > |t| Intercept 1 0.66667 0.69423 0.96 0.3620 exptime 1 1.31667 0.09981 13.19 <.0001 humidity 1 -8.00000 1.36677 -5.85 0.0002 Note: In this equation, weight loss is positively related to exposure time and negatively to humidity.

5. 5 Residual Analysis in Multiple Regression Examination of residuals to help determine if: - assumptions are met - regression model is appropriate Residual Plots: - each indep var. in final model vs residuals - predicted Y vs residuals - run order vs residuals

6. 6 PROC REG; MODEL wtloss=exptime humidity; output out=new r=resid2 p=predict2; RUN; PROC GPLOT; Title 'Plot of Residuals - MLR Model'; PLOT resid2*exptime; PLOT resid2*humidity; PLOT resid2*predict2; RUN;

7. 7 Infant Length Data (Probability and Statistics for Engineers and Scientists – Walpole, Myers, Myers, and Ye, page 433) Data Set: 9 infants (2-3 months of age) Dependent Variable (Y): Current Infant length (cm) Independent Variables: X1 = age (in days) X2 = length at birth (cm) X3 = weight at birth (kg) X4 = chest size at birth (cm) Goal: Obtain an estimating equation relating length of an infant to all or a subset of these independent variables. DATA infant; INPUT id y x1 x2 x3 x4; DATALINES; 1 57.5 78 48.2 2.75 29.5 2 52.8 69 45.5 2.15 26.3 3 61.3 77 46.3 4.41 32.2 4 67.0 88 49.0 5.52 36.5 5 53.5 67 43.0 3.21 27.2 6 62.7 80 48.0 4.32 27.7 7 56.2 74 48.0 2.31 28.3 8 68.5 94 53.0 4.30 30.3 9 69.2 102 58.0 3.71 28.7 ; PROC CORR; Var y x1 x2 x3 x4; RUN; PROC REG; MODEL y=x1 x2 x3 x4; output out=new r=resid; RUN;

8. 8 SAS PROC CORR Output Pearson Correlation Coefficients, N = 9 Prob > |r| under H0: Rho=0 y x1 x2 x3 x4 y 1.00000 0.94709 0.81867 0.76114 0.56033 0.0001 0.0070 0.0172 0.1166 x1 0.94709 1.00000 0.95227 0.53402 0.38999 0.0001 <.0001 0.1386 0.2995 x2 0.81867 0.95227 1.00000 0.26267 0.15491 0.0070 <.0001 0.4947 0.6907 x3 0.76114 0.53402 0.26267 1.00000 0.78447 0.0172 0.1386 0.4947 0.0123 x4 0.56033 0.38999 0.15491 0.78447 1.00000 0.1166 0.2995 0.6907 0.0123 Note: x1, x2, and x3 are significantly correlated with y while x4 is not. Recall, this indicates that the simple linear regression of y using either x1, x2, or x3 will be significant. Standard SAS PROC REG Output for all 4 Independent Variables X1, X2, X3, and X4 Dependent Variable: Y Analysis of Variance Sum of Mean Source DF Squares Square F Value Prob>F Model 4 318.27442 79.56860 107.323 0.0003 Error 4 2.96558 0.74140 C Total 8 321.24000 Root MSE 0.86104 R-square 0.9908 Dep Mean 60.96667 Adj R-sq 0.9815 C.V. 1.41232 Parameter Estimates Parameter Standard T for H0: Variable DF Estimate Error Parameter=0 Prob > |T| INTERCEP 1 7.147532 16.45961128 0.434 0.6865 X1 1 0.100094 0.33970898 0.295 0.7829 X2 1 0.726417 0.78590156 0.924 0.4076 X3 1 3.075837 1.05917874 2.904 0.0439 X4 1 -0.030042 0.16646232 -0.180 0.8656 Note: Even though the overall p-value is small (.0003), there is much confusion concerning the contribution of the individual X variables - this is probably due to collinearity

9. 9 Setting: We have a dependent variable Y and several candidate independent variables. Question: Should we use all of them?

10. 10 Why do we run Multiple Regression? 1. Obtain estimates of individual coefficients in a model (+ or -, etc.) 2. Screen variables to determine which have a significant effect on the model 3. Arrive at the most effective (and efficient) prediction model

11. 11 The problem: Collinearity among the independent variables -- high correlation between 2 independent variables -- one independent variable nearly a linear combination of other independent variables -- etc. Example: x 1 = total income x 2 = bonus x 3 = monthly income Note: x 1 = 12 x 3 + x 2 -- singularity -- SAS cannot use all 3

12. 12 Effects of Collinearity parameter estimates are highly variable and unreliable - parameter estimates may even have the opposite sign from what is reasonable may have significant F but none of the t-tests are significant Variable Selection Techniques Techniques for “being careful” about which variables are put into the model

13. 13 Variable Selection Procedures Forward selection Backward Elimination Stepwise Best subset

14. 14 Forward Selection: Step 1: Choose X j that has highest R 2 (i.e. has the highest correlation with Y) -- call it X 1 Step 2: Choose another X j to go along with X 1 by finding the one that maximizes R 2 Note: This new R 2 will be at least as large as the one in Step 1. Problem: Has the new variable increased R 2 enough to be “useful”? Solution: Examine the significance level (p) of the new variable -- keep variable if p < SLENTRY (I used SLENTRY =.15 in example) Procedure continues until no new variables satisfy entry criteria

15. 15 FORWARD SELECTION RESULTS FROM SAS Stepwise Procedure for Dependent Variable Y Step 1 Variable X1 Entered R-square = 0.89698302 C(p) = 39.63635101 DF Sum of Squares Mean Square F Prob>F Regression 1 288.14682495 288.14682495 60.95 0.0001 Error 7 33.09317505 4.72759644 Total 8 321.24000000 Parameter Standard Type II Variable Estimate Error Sum of Squares F Prob>F INTERCEP 19.01108007 5.42271930 58.10583282 12.29 0.0099 X1 0.51797020 0.06634651 288.14682495 60.95 0.0001 Note: These F values are the squares of the usual t-values in SAS Bounds on condition number: 1, 1 -------------------------------------------------------------------------------- Step 2 Variable X3 Entered R-square = 0.98821914 C(p) = 2.10454082 DF Sum of Squares Mean Square F Prob>F Regression 2 317.45551809 158.72775905 251.65 0.0001 Error 6 3.78448191 0.63074698 Total 8 321.24000000 Parameter Standard Type II Variable Estimate Error Sum of Squares F Prob>F INTERCEP 20.10845029 1.98725776 64.58088391 102.39 0.0001 X1 0.41362967 0.02866328 131.34899803 208.24 0.0001 X3 2.02533400 0.29711598 29.30869314 46.47 0.0005 Bounds on condition number: 1.398946, 5.595783 -------------------------------------------------------------------------------- All variables left in the model are significant at the 0.1500 level. No other variable met the 0.1500 significance level for entry into the model. Summary of Stepwise Procedure for Dependent Variable Y Variable Number Partial Model Step Entered Removed In R**2 R**2 C(p) F Prob>F 1 X1 1 0.8970 0.8970 39.6364 60.9500 0.0001 2 X3 2 0.0912 0.9882 2.1045 46.4666 0.0005 This is the end of the SAS FORWARD SELECTION output. The final regression equation is: We can see from the model that an increase in age or in the weight at birth predicts longer current length. NOTICE: SAS picked 2 independent variables and then stopped. PROC reg; MODEL y=x1 x2 x3x4 /selection=forward slentry=.15; RUN; The next pages show SAS output from standard PROC REG. Each set of output on the following pages is from a separate running of PROC REG.

16. 16 Standard SAS PROC REG Printout for 3 Features - to show why STEPWISE Procedure stopped with 2 features X1, X3, and X4 Dependent Variable: Y Analysis of Variance Sum of Mean Source DF Squares Square F Value Prob>F Model 3 317.64101 105.88034 147.097 0.0001 Error 5 3.59899 0.71980 C Total 8 321.24000 Root MSE 0.84841 R-square 0.9888 Dep Mean 60.96667 Adj R-sq 0.9821 C.V. 1.39160 Parameter Estimates Parameter Standard T for H0: Variable DF Estimate Error Parameter=0 Prob > |T| INTERCEP 1 21.873528 4.07388552 5.369 0.0030 X1 1 0.412771 0.03066663 13.460 0.0001 X3 1 2.202668 0.47198905 4.667 0.0055 X4 1 -0.078945 0.15551477 -0.508 0.6333 Note: p-value for X4 is too large. X1, X3, and X2 Dependent Variable: Y Analysis of Variance Sum of Mean Source DF Squares Square F Value Prob>F Model 3 318.25027 106.08342 177.413 0.0001 Error 5 2.98973 0.59795 C Total 8 321.24000 Root MSE 0.77327 R-square 0.9907 Dep Mean 60.96667 Adj R-sq 0.9851 C.V. 1.26835 Parameter Estimates Parameter Standard T for H0: Variable DF Estimate Error Parameter=0 Prob > |T| INTERCEP 1 5.629827 12.70680159 0.443 0.6762 X1 1 0.080984 0.28988008 0.279 0.7911 X3 1 3.069358 0.95066097 3.229 0.0232 X2 1 0.771498 0.66918975 1.153 0.3011 Note: X2 really messes up the p-values, and the p-value for X2 is too large

17. 17 Standard SAS PROC Reg Output for X1 and for X1 & X3 X1 Dependent Variable: Y Analysis of Variance Sum of Mean Source DF Squares Square F Value Prob>F Model 1 288.14682 288.14682 60.950 0.0001 Error 7 33.09318 4.72760 C Total 8 321.24000 Root MSE 2.17430 R-square 0.8970 Dep Mean 60.96667 Adj R-sq 0.8823 C.V. 3.56638 Parameter Estimates Parameter Standard T for H0: Variable DF Estimate Error Parameter=0 Prob > |T| INTERCEP 1 19.011080 5.42271930 3.506 0.0099 X1 1 0.517970 0.06634651 7.807 0.0001 X1 and X3 Dependent Variable: Y Analysis of Variance Sum of Mean Source DF Squares Square F Value Prob>F Model 2 317.45552 158.72776 251.650 0.0001 Error 6 3.78448 0.63075 C Total 8 321.24000 Root MSE 0.79420 R-square 0.9882 Dep Mean 60.96667 Adj R-sq 0.9843 C.V. 1.30267 Parameter Estimates Parameter Standard T for H0: Variable DF Estimate Error Parameter=0 Prob > |T| INTERCEP 1 20.108450 1.98725776 10.119 0.0001 X1 1 0.413630 0.02866328 14.431 0.0001 X3 1 2.025334 0.29711598 6.817 0.0005

18. 18 Plots for Residual Analysis for the Final Model, i.e. x1 x3 IDPredicted Values

19. 19 Begin with all independent variables in the model Find the independent variable that is “least useful” in predicting the dependent variable (i.e. smallest R 2, F (or t), etc.) –delete this variable if p < SLSTAY Continue the process until no further variables are deleted Backward Elimination

20. 20 Add independent variables one at a time as in Forward Selection (if p < SLENTRY) At each stage perform backward elimination to see whether any variables should be removed (if p < SLSTAY) Stepwise Selection

21. 21 Examine criteria for all acceptable subsets of each “size”, i.e. # of independent variables Criteria: R 2, adjusted R 2, C p Best Subset Regression

22. 22 -- adjusts for the number of independent variables -- penalizes excessive use of independent variables -- useful for comparing competing models with differing number of independent variables Adjusted R 2 - C p statistic plays a similar role

23. 23 Multiple Regression – Analysis Suggestions 1. Examine pairwise correlations among variables 2. Examine pairwise scatterplots among variables

24. 24 SPSS Output from INFANT Data Set

25. 25 SPSS Output from CAR Data Set

26. 26 Multiple Regression – Analysis Suggestions 1. Examine pairwise correlations among variables 2. Examine pairwise scatterplots among variables - identify nonlinearity - identify unequal variance problems - identify possible outliers 3. Try transformations of variables for - correcting nonlinearity - stabilizing the variances - inducing normality of residuals

27. 27 Examples of Nonlinear Data “Shapes” and Linearizing Transformations

28. 28 Original Model  1 > 0  1 < 0 Transformed Into: Exponential Transformation (Log-Linear)

29. 29 Transformed Multiplicative Model (Log-Log)

30. 30  1 > 0  1 < 0 Square Root Transformation

31. 31 Note: - transforming Y using the log or square root transformation can help with unequal variance problems - these transformations may also help induce normality

32. 32 hmpg vs hp hmpg vs sqrt(hp) log(hmpg) vs hp log(hmpg) vs log(hp)