1. 1 G89.2229 Lect 4W Multiple regression in matrix terms Exploring Regression Examples G89.2229 Multiple Regression Week 4 (Wednesday)

2. 2 G89.2229 Lect 4W Some Matrix Language A vector is a list of numbers. »It is assumed to be a column. »The number of elements is its row order. »A vector X can be represented geometrically as a single point in high dimensional space. Its direction is determined by drawing a line from the zero point to X. A matrix is a collection of vectors. »The number of rows is row order »The # of columns is column order »The rank of the matrix is the number of dimensions needed to represent the vectors. It will be equal or less than the smaller of the row and column order.

3. 3 G89.2229 Lect 4W Matrix Multiplication An Identity matrix, I, is a square matrix with ones on the diagonal and zeros on the off diagonal »A*I = A When the matrix product of two matrices is I, the matrices are orthogonal. If a matrix A is square and full rank (nonsingular), then its unique inverse A -1 exists such that »A*A -1 = I

4. 4 G89.2229 Lect 4W Matrix Inverse In general, the elements of an inverse are not intuitive. As the number of rows/columns gets larger, they are more complex. For 2x2 matrix A E.G.

5. 5 G89.2229 Lect 4W Some facts about matrix multiplication Distributive principle holds »A(B+C) = AB + AC In general, AB ~= BA »Commutative principle does not hold When A and B are square and full rank »(A*B) -1 = B -1 *A -1 A matrix A can be multiplied by a single number, called a scalar, which sets the unit of the new matrix: »kA = [kA ij ]

6. 6 G89.2229 Lect 4W Regression equations in Matrix Terms Basic Regression equation »For randomly chosen observation Y = x T B + e »For sample of n subjects Y = XB + e e = Y  XB Y = [1 X 1 X 2 ] B 0 + e B 1 B 2 Y 1 1 X 11 X 12 B 0 e 1 Y 2 = 1 X 21 X 22 B 1 + e 2 Y 3 1 X 31 X 32 B 2 e 3

7. 7 G89.2229 Lect 4W Least Squares Estimates of B The OLS estimates of B make e T e as small as possible. »This happens when the geometric representation of e is shortest. »e will be shortest when it is orthogonal to the predictors, X X T e = X T (Y- XB) = 0 X T (Y  XB) = X T Y  X T XB = 0 X T Y  X T XB When (X T X) -1 exists:

8. 8 G89.2229 Lect 4W Some Useful Matrix Facts Regression for one observation Regression for sample Y = X B + e Regression estimates If data are centered Standardized regression

9. 9 G89.2229 Lect 4W Matrix Expectation Operators Matrix expectation operators follow the same rules we have studied previously Assume that E(Y|X) = XB »E(B|X) = E[(X T X) -1 X T Y | X] = (X T X) -1 X T E[ Y | X] = (X T X) -1 X T XB = B »V(B|X) = V[(X T X) -1 X T Y | X] = (X T X) -1 X T V[ Y | X][(X T X) -1 X T ] T = (X T X) -1 X T [I    X  (X T X) -1 = (X T X) -1 X T X  (X T X) -1   =  (X T X) -1   ^ ^ (OLS estimate is unbiased)

10. 10 G89.2229 Lect 4W Example

11. 11 G89.2229 Lect 4W The diagonal of R -1 and (X T X) -1 The i th diagonal element of R -1 is 1/(1-R 2 i ) where R 2 i is the proportion of variance of the i th predictor accounted by the other predictors. The i th diagonal element of (X T X) -1 is [1/(1-R 2 i )](n-1)S 2 i Both get larger as the predictors become multicollinear

12. 12 G89.2229 Lect 4W Comparing Two Regression Coefficients Suppose we have »Y=b 0 +b 1 X 1 +b 2 X 2 +e Suppose one theory states that b 1 =b 2, but another theory says that they should differ. How do we carry out the test? Create a Wald statistic »In the numerator, »In the denominator, the standard error of the numerator. Recall for two vars W and Z: V(k 1 W+ k 2 Z) = k 1 2  w 2 + k 2 2  z 2 + 2 k 1 k 2  wz

13. 13 G89.2229 Lect 4W Example: Predicting Depressed mood day 29 In bar exam study, let's revisit the prediction of depression on day 29 as a function of depression and anxiety on day 28. What can we say about »The residual distribution? »Homoscedasticity? »Adequacy of the linear model? »Alternative scaling of depression? »Testing whether the two effects are identical?