LECTURE 12 Multiple regression analysis Epsy 640 Texas A&M University

Multiple regression analysis The test of the overall hypothesis that y is unrelated to all predictors, equivalent to H 0 :  2 y  123… = 0 H 1 :  2 y  123… = 0 is tested by F = [ R 2 y  123… / p] / [ ( 1 - R 2 y  123… ) / (n – p – 1) ] F = [ SS reg / p ] / [ SS e / (n – p – 1)]

Multiple regression analysis SOURCEdfSum of SquaresMean SquareF x 1, x 2 …pSS reg SS reg / pSS reg / 1 SS e /(n-p-1) e (residual) n-p-1SS e SS e / (n-p-1) total n-1SS y SS y / (n-1) Table 8.2: Multiple regression table for Sums of Squares

Multiple regression analysis predicting Depression LOCUS OF CONTROL, SELF-ESTEEM, SELF-RELIANCE

ss x 1 ss x 2 SSy SSe Fig. 8.4: Venn diagram for multiple regression with two predictors and one outcome measure SS reg

Type I ss x 1 Type III ss x 2 SSy SSe Fig. 8.5: Type I contributions SSx 1 SSx 2

Type III ss x 1 Type III ss x 2 SSy SSe Fig. 8.6: Type IIII unique contributions SSx 1 SSx 2

Multiple Regression ANOVA table SOURCEdfSum of SquaresMean SquareF (Type I) Model2SS reg SS reg / 2SS reg / 2 SS e / (n-3) x 1 1 SS x1 SS x1 / 1SS x1 / 1 SS e /(n-3) x 21 SS x2  x1 SS x2  x1 SS x2  x1 / 1 SS e /(n-3) e n-3SS e SS e / (n-3) total n-1SS y SS y / (n-3) Table 8.3: Multiple regression table for Sums of Squares of each predictor

X1X1 X2X2 Y e  =.5  =.6 r =.4 R 2 = (.74)(.8)(.4)  ( ) = PATH DIAGRAM FOR REGRESSION

Depression DEPRESSION LOC. CON. SELF-EST SELF-REL R 2 =.60 e .4

Shrinkage R 2 Different definitions: ask which is being used: –What is population value for a sample R 2 ? R 2 s = 1 – (1- R 2 )(n-1)/(n-k-1) –What is the cross-validation from sample to sample? R 2 sc = 1 – (1- R 2 )(n+k)/(n-k)

Estimation Methods Types of Estimation: –Ordinary Least Squares (OLS) Minimize sum of squared errors around the prediction line –Generalized Least Squares A regression technique that is used when the error terms from an ordinary least squares regression display non-random patterns such as autocorrelation or heteroskedasticity.ordinary least squares –Maximum Likelihood

Maximum Likelihood Estimation Maximum likelihood estimation There is nothing visual about the maximum likelihood method - but it is a powerful method and, at least for large samples, very preciseMaximum likelihood estimation begins with writing a mathematical expression known as the Likelihood Function of the sample data. Loosely speaking, the likelihood of a set of data is the probability of obtaining that particular set of data, given the chosen probability distribution model. This expression contains the unknown model parameters. The values of these parameters that maximize the sample likelihood are known as the Maximum Likelihood Estimatesor MLE's. Maximum likelihood estimation is a totally analytic maximization procedure. MLE's and Likelihood Functions generally have very desirable large sample properties: –they become unbiased minimum variance estimators as the sample size increases –they have approximate normal distributions and approximate sample variances that can be calculated and used to generate confidence bounds –likelihood functions can be used to test hypotheses about models and parameters With small samples, MLE's may not be very precise and may even generate a line that lies above or below the data pointsThere are only two drawbacks to MLE's, but they are important ones: –With small numbers of failures (less than 5, and sometimes less than 10 is small), MLE's can be heavily biased and the large sample optimality properties do not apply Calculating MLE's often requires specialized software for solving complex non-linear equations. This is less of a problem as time goes by, as more statistical packages are upgrading to contain MLE analysis capability every year.

Outliers Leverage (for a single predictor): L i = 1/n + (Xi –Mx) 2 /  x 2 (min=1/n, max=1) Values larger than 1/n by large amount should be of concern Cook’s Di =  (Y – Yi) 2 / [(k+1)MSres] –the difference between predicted Y with and without Xi   

Outliers In SPSS Regression, under the SAVE option, both leverage and Cook’s D will be computed and saved as new variables with values for each case