1. Multiple Regression

2. Objectives

3. Explanation The most direct interpretation of the regression variate is a determination of the relative importance of each independent variable in the prediction of the dependent measure. Assess the nature of the relationships between the independent variables and the dependent variable. Provide insight into the relationships among independent variables. Y’ X1X1 X2X2 X3X3

4. Sample Problem (Leslie Salt Property):Finding Fair Price of a Land Variable NameDescription PRICESale price in $000 per acre COUNTYSan Mateo=0, Santa Clara=1 SIZESize of the property in acres ELEVATIONAverage elevation in feet above sea level SEWERDistance (in feet) to nearest sewer connection DATEDate of sale counting backward from current time (in months) FLOODSubject to flooding by tidal action =1; otherwise=0 DISTANCEDistance in miles from Leslie property

5. PRICECOUNTYSIZEELEVATIONSEWERDATEFLOODDISTANCE 4.51138.4103000-10300.3 10.615240-10302.5 1.7016.102640-98110.3 501695.213500-93014 5084511000-92114 3.316.9210000-8600 5.71105.940-6800 6.2156.640-6400 19.4151.4201300-6301.2 3.2122.106000-6200 4.7122.106000-6100 6.9127.734500-6000 8.1118.655000-5900.5 11.6169.980-5904.4 19.31145.7100-5904.2 11.7177.290-5904.5 13.3126.280-5904.7 15.11102.360-5904.9 12.4149.5110-5904.6 15.3112.280-5905 12.20320.604000-54016.5 18.119.950-5405.2 16.8115.320-5305.5 5.9055.201320-49111.9 40116.22900-4515.5 37.201550-3907.2 18.2023.454420-3905.5 15.10132.822640-35010.2 22.901253400-1605.5 15.20672900-515.5 21.9030.82900-405.5

6. SEWER FLOOD SIZE COUNTY DISTANCE ELEVATION DATE PRICE SEWER FLOOD SIZECOUNTYDISTANCEELEVATION DATEPRICE COUNTYSIZEELEVATIONSEWERDATEFLOODDISTANCE PRICE100.00%-18.22%-23.97%35.18%-39.12%59.47%-32.31%9.33% COUNTY-18.22%100.00%-33.94%47.52%-5.00%-36.98%-55.18%-74.22% SIZE-23.97%-33.94%100.00%-20.95%5.34%-34.95%10.89%55.69% ELEVATION35.18%47.52%-20.95%100.00%-35.94%-5.65%-37.31%-36.25% SEWER-39.12%-5.00%5.34%-35.94%100.00%-15.15%-11.31%-15.87% DATE59.47%-36.98%-34.95%-5.65%-15.15%100.00%1.54%4.44% FLOOD-32.31%-55.18%10.89%-37.31%-11.31%1.54%100.00%42.33% DISTANCE9.33%-74.22%55.69%-36.25%-15.87%4.44%42.33%100.00%

8. PRICECOUNTYSIZEELEVATIONSEWERDATEFLOODDISTANCE PRICE100.00%-18.22%-23.97%35.18%-39.12%59.47%-32.31%9.33% COUNTY-18.22%100.00%-33.94%47.52%-5.00%-36.98%-55.18%-74.22% SIZE-23.97%-33.94%100.00%-20.95%5.34%-34.95%10.89%55.69% ELEVATION35.18%47.52%-20.95%100.00%-35.94%-5.65%-37.31%-36.25% SEWER-39.12%-5.00%5.34%-35.94%100.00%-15.15%-11.31%-15.87% DATE59.47%-36.98%-34.95%-5.65%-15.15%100.00%1.54%4.44% FLOOD-32.31%-55.18%10.89%-37.31%-11.31%1.54%100.00%42.33% DISTANCE9.33%-74.22%55.69%-36.25%-15.87%4.44%42.33%100.00%

9. summary(model) Call: lm(formula = leslie_salt[, 1] ~ leslie_salt[, 4] + leslie_salt[, 5] + leslie_salt[, 6]) Residuals: Min 1Q Median 3Q Max -9.6076 -3.2506 -0.0281 2.8770 20.2776 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 21.2787636 2.9203157 7.286 7.75e-08 *** leslie_salt[, 4] 0.5614588 0.2515472 2.232 0.034107 * leslie_salt[, 5] -0.0005871 0.0004460 -1.316 0.199129 leslie_salt[, 6] 0.1836824 0.0421712 4.356 0.000172 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 5.559 on 27 degrees of freedom Multiple R-squared: 0.5327,Adjusted R-squared: 0.4807 F-statistic: 10.26 on 3 and 27 DF, p-value: 0.000111

10. Assumptions

11. Linearity (cts.) A higher order term of the dependent variable should be included. In that case define a new variable by taking the square (for this case) of that independent variable and use squared values in the regression. Use: Visual inspection

12. More troublesome is MODERATOR effect If an independent-dependent variable relationship is effected by another independent variable this situation is termed a moderator effect. The most common moderator effect in multiple regression is the bilinear moderator in which the slope of the relationship of one independent variable (X 1 ) changes across values of the moderator variable (X 2 ).

13. Example

14. Adding Moderator Effect The idea comes from observing a self moderator effect. If a variable has a moderator effect onto itself then we would assume a nonlinear (second degree) relationship with the dependent variable. Thus if there is a moderator effect add X 1 X 2 as an independent variable to regression equation. But we will return back to this!!!

15. Assumption: Homoscedasticity Constant variance of the error terms.

16. Heteroscedasticity (cts.) in residuals within variables

17. Heteroscedasticity (cts.) Use: Levene Test. Levene Test: Tests the equality of variance. Levene's test works by testing the null hypothesis that the variances of the group are the same. The output probability is the probability that at least one of the samples in the test has a significantly different variance. If this is greater than a selected percentage (usually 5%) then it is considered too great to be able to usefully apply parametric tests. Variances In SPSS it is reported. In R: In «lawstat» library use levene.test() function.

18. Use F test for more than 2 groups…

20. Assumptions Independence of the error terms. Check the coordinates!!!

21. Independence of Error Terms Use: Durbin-Watson The value of the Durbin-Watson statistic ranges from 0 to 4. As a general rule of thumb, the residuals are uncorrelated is the Durbin- Watson statistic is approximately 2. A value close to 0 indicates strong positive correlation, while a value of 4 indicates strong negative correlation.

23. In SPSS Durbin Watson is reported. In R under «lmtest» library use dwtest() dwtest(formula, order.by = NULL, alternative = c("greater", "two.sided", "less"), iterations = 15, exact = NULL, tol = 1e-10, data = list()) For our regression model. > dwtest(model) Durbin-Watson test data: model DW = 2.3762, p-value = 0.7783 alternative hypothesis: true autocorrelation is greater than 0

24. Assumptions Normality of the error term distribution.

25. qqPlot(model)

26. Diagonistics Call: lm(formula = leslie_salt[, 1] ~ leslie_salt[, 4] + leslie_salt[, 5] + leslie_salt[, 6]) Residuals: Min 1Q Median 3Q Max -9.6076 -3.2506 -0.0281 2.8770 20.2776 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 21.2787636 2.9203157 7.286 7.75e-08 *** leslie_salt[, 4] 0.5614588 0.2515472 2.232 0.034107 * leslie_salt[, 5] -0.0005871 0.0004460 -1.316 0.199129 leslie_salt[, 6] 0.1836824 0.0421712 4.356 0.000172 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 5.559 on 27 degrees of freedom Multiple R-squared: 0.5327,Adjusted R-squared: 0.4807 F-statistic: 10.26 on 3 and 27 DF, p-value: 0.000111

27. Identifying Influential Observations observations that lie outside the general patterns of the data set observations that strongly influence regression results Types of Influential Observations 1.Outliers – observations that have large residuals (based on dependent variables) 2.Leverage points – observations that are distinct from the remaining observations based on their independent variable values. 3.Influential observations – including all observations that have a disproportionate effect on the regression results.

28. Outliers Typical boxplot test. In «car» library  outlierTest(model) rstudent unadjusted p-value Bonferonni p 2 3.704906 0.0010527 0.032634

29. Leverage

31. 051015202530 0.00 0.15 0.30 Index cooks.distance(model)

32. R-Code # Influential Observations # added variable plots av.Plots(model) # Cook's D plot # identify D values > 4/(n-k-1) cutoff <- 4/((nrow(leslie_salt)- length(model$coefficients)-2)) plot(fit, which=4, cook.levels=cutoff) # Influence Plot influencePlot(model, id.method="identify", main="Influence Plot", sub="Circle size is proportial to Cook's Distance" )

33. 0.00.10.20.30.4 -2 0 1 2 3 Leverage Standardized residuals lm(leslie_salt[, 1] ~ leslie_salt[, 4] + leslie_salt[, 6] + leslie_salt[, 7... Cook's distance 0.5 1 Residuals vs Leverage 2 9 4

34. Assessing Multicollinearity***** A key issue in interpreting the regression variate is the correlation among the independent variables. Our task in a regression analysis includes the following: 1.Assess the degree of multicollinearity 2.Determine its impact on results 3.Apply the necessary remedies if needed

35. Assess the degree of multicollinearity The simplest and most obvious way: Identifying collinearity in correlation matrix. Check for correlation >90%. A direct measure of multicollinearity is tolerance (1/VIF). The amount of variability of the selected independent variable not explained by the other independent variables. Computation: Take each independent variable. Assume it as the dependent variable. Compute adjusted R 2. Tolerance is then 1-R 2. For example if other variables explain 25% of an independent variable then tolerence of this variable is 75%. Tolerence should be more than 10% > 1/vif(model) leslie_salt[, 4] leslie_salt[, 6] leslie_salt[, 7] leslie_salt[, 8] 0.8081325 0.9959058 0.7650806 0.7715437

36. Further… see page http://www.statmethods.net/stats/rdiagnostics.html for diagonistic tests with Rhttp://www.statmethods.net/stats/rdiagnostics.html

37. Partial Correlation A partial correlation coefficient is a way of expressing the unique relationship between the criterion and a predictor. Partial correlation represents the correlation between the criterion and a predictor after common variance with other predictors has been removed from both the criterion and the predictor of interest. t.values <- model$coeff / sqrt(diag(vcov(model))) partcorr <- sqrt((t.values^2) / ((t.values^2) + model$df.residual)) partcorr ***************************************************** leslie_salt[, 4] leslie_salt[, 6] leslie_salt[, 7] leslie_salt[, 8] 0.6562662 0.8043296 0.6043579 0.5740840

39. Part (Semi-partial) Correlation A semipartial correlation coefficient represents the correlation between the criterion and a predictor that has been residualized with respect to all other predictors in the equation. Note that the criterion remains unaltered in the semipartial. Only the predictor is residualized. After removing variance that the predictor has in common with other predictors, the semipartial expresses the correlation between the residualized predictor and the unaltered criterion. An important advantage of the semipartial is that the denominator of the coefficient (the total variance of the criterion, Y) remains the same no matter which predictor is being examined. This makes the semipartial very interpretable. The square of the semipartial can be interpreted as the proportion of the criterion variance associated uniquely with the predictor. It is also possible to use the semipartial to fully deconstruct the variance components in a regression analysis.

40. Project (Step1): Go to web page: http://luna.cas.usf.edu/~mbrannic/files/regression/Partial.html Replicate the results there using a dataset of your own. Be creative in problem formulation. Data may be imaginary. Use at least 5 independent variables.

41. Comparing Regression Models

42. Stepwise Regression Start with the most basic model. Pick your favourite independent variable and construct the model. Test it. Remember correlation matrix (price in logs) PRICECOUNTYSIZEELEVATIONSEWERDATEFLOODDISTANCE PRICE100.00%-18.22%-23.97%35.18%-39.12%59.47%-32.31%9.33% COUNTY-18.22%100.00%-33.94%47.52%-5.00%-36.98%-55.18%-74.22% SIZE-23.97%-33.94%100.00%-20.95%5.34%-34.95%10.89%55.69% ELEVATION35.18%47.52%-20.95%100.00%-35.94%-5.65%-37.31%-36.25% SEWER-39.12%-5.00%5.34%-35.94%100.00%-15.15%-11.31%-15.87% DATE59.47%-36.98%-34.95%-5.65%-15.15%100.00%1.54%4.44% FLOOD-32.31%-55.18%10.89%-37.31%-11.31%1.54%100.00%42.33% DISTANCE9.33%-74.22%55.69%-36.25%-15.87%4.44%42.33%100.00%

43. Call: lm(formula = leslie_salt[, 1] ~ leslie_salt[, 6]) Residuals: Min 1Q Median 3Q Max -1.12046 -0.34364 0.04853 0.39719 1.00081 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 3.322336 0.269975 12.306 4.9e-13 *** leslie_salt[, 6] 0.018124 0.004257 4.257 0.000198 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 0.5719 on 29 degrees of freedom Multiple R-squared: 0.3846,Adjusted R-squared: 0.3634 F-statistic: 18.12 on 1 and 29 DF, p-value: 0.0001982 Our focus is the improvement in RSS. So we need residual sum of squares. But it is not given in the report directly (given in SPSS). > anova(m1) Analysis of Variance Table Response: leslie_salt[, 1] Df Sum Sq Mean Sq F value Pr(>F) leslie_salt[, 6] 1 5.9282 5.9282 18.124 0.0001982 *** Residuals 29 9.4858 0.3271 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

44. Now lets add another variable say SEWER and assume we have done all testing Call: lm(formula = leslie_salt[, 1] ~ leslie_salt[, 6] + leslie_salt[, 5]) Residuals: Min 1Q Median 3Q Max -1.21681 -0.21980 0.08597 0.29875 0.81520 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 3.442e+00 2.442e-01 14.093 3.07e-14 *** leslie_salt[, 6] 1.643e-02 3.841e-03 4.278 0.000199 *** leslie_salt[, 5] -1.105e-04 3.797e-05 -2.910 0.007013 ** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 0.51 on 28 degrees of freedom Multiple R-squared: 0.5275,Adjusted R-squared: 0.4937 F-statistic: 15.63 on 2 and 28 DF, p-value: 2.766e-05 Analysis of Variance Table Response: leslie_salt[, 1] Df Sum Sq Mean Sq F value Pr(>F) leslie_salt[, 6] 1 5.9282 5.9282 22.7903 5.146e-05 *** leslie_salt[, 5] 1 2.2024 2.2024 8.4671 0.007013 ** Residuals 28 7.2833 0.2601 ---

45. How much improvement do we have?

46. In our case

47. Back to moderator effect.

48. Mediation

49. Project (Step2,3 and 4): Find the best regression equation for your Project. Test moderator effects Test mediation effects.