Regression 10/29

Prediction Correlation can tell how to predict one variable from another What about multiple predictor variables? Explaining Y using X1, X2, … Xm Income based on age, education Memory based on pre-test, IQ Social behavior based on personality dimensions Regression Finds best combination of predictors to explain outcome variable Determines unique contribution of each predictor Can test whether each predictor has reliable influence

Linear Prediction One predictor Multiple predictors Intercept: b0 Draw a line through data Multiple predictors Each predictor has linear effect Effects of different predictors simply add together Intercept: b0 Value of Y when all Xs are zero Regression coefficients: bi Influence of Xi Sign tells direction; magnitude tells strength Can be any value (not standardized like correlation)

Example Predict income from education and intelligence Y = $k/yr X1 = years past HS X2 = IQ Regression equation: 4-year college, IQ = 120 -70 + 10*4 + 1*120 = 90  $90k/yr 2-year college, IQ = 90 -70 + 10*2 + 1*90 = 40  $40k/yr

Finding Regression Coefficients Goal: regression equation with best fit to data Minimize error between Y and Math solution Matrix algebra Treats X as a matrix Rows for subjects, columns for variables Transposes, matrix inverse, and other fun Practical solution Use a computer R: linear model function: lm(Y~X1+X2+X3)

Regression vs. Correlation One predictor: regression = correlation Multiple predictors Predictors affect each other Each coefficient depends on what other predictors are included Example: 3rd-variable problem Math achievement and verbal achievement are correlated Math helps with English? Add IQ as a predictor: verbal ~ math + IQ bmath will be near 0 No effect of math once IQ is accounted for Regression finds best combination of predictors to explain outcome Best values of b1, b2, etc. taken together Each coefficient shows contribution of predictor beyond effects of others Allows identification of most important predictors

Explained Variability Sum of squares Same as mean squared error, but without dividing by df Easier because not complicated by degrees of freedom Total sum of squares Variability in Y SSY = S(Y – MY)2 Residual sum of squares Deviation between predicted and actual scores SSresidual = S(Y – )2

Explained Variability SSY SSresidual SSregression SSregression Variability explained by regression Difference between total and residual sums of squares SSregression = SSY - SSresidual R2 ("R-squared") Fraction of variability explained by the regression SSregression/SSY Measures how well the predictors (Xs) can predict or explain the outcome (Y) Same as r2 from correlation when only one predictor

Hypothesis Testing with Regression Does the regression explain meaningful variance in the outcome? Null hypothesis All regression coefficients equal zero in population Regression only “explaining” random error in sample Approach Find how much variance the predictors explain: SSregression Compare to amount expected by chance Ratio gives F statistic, which we can use to get p-value If F is larger than expected, regression explains more variance than expected by chance Reject null hypothesis if F is large enough

Hypothesis Testing with Regression SSY SSresidual SSregression Likelihood function for F is an F distribution Comes from ratio of two chi-square variables Two df values, from numerator and denominator p-value is probability of a result greater than F

Degrees of Freedom SSY dfY = n - 1 dfresidual = n – m – 1 SSregression dfregression = m SSY dfY = n - 1 dfresidual = n – m – 1

Testing Individual Predictors Does predictor variable have any effect on outcome? Does it provide any information beyond other predictors? Null hypothesis: bi = 0 in population Other bs might be nonzero Compute standard error for bi Uses SSresidual (MSE) to estimate t statistic: Get p-value from t distribution with dfresidual Can do one-tailed test if predicted direction of effect Sign of bi