1. Part 3: Least Squares Algebra 3-1/27 Econometrics I Professor William Greene Stern School of Business Department of Economics

2. Part 3: Least Squares Algebra 3-2/27 Econometrics I Part 3 – Least Squares Algebra

3. Part 3: Least Squares Algebra 3-3/27 Vocabulary  Some terms to be used in the discussion. Population characteristics and entities vs. sample quantities and analogs Residuals and disturbances Population regression line and sample regression  Objective: Learn about the conditional mean function. Estimate  and  2  First step: Mechanics of fitting a line (hyperplane) to a set of data

4. Part 3: Least Squares Algebra 3-4/27 Fitting Criteria  The set of points in the sample  Fitting criteria - what are they: LAD Least squares and so on  Why least squares?  A fundamental result: Sample moments are “good” estimators of their population counterparts We will spend the next few weeks using this principle and applying it to least squares computation.

5. Part 3: Least Squares Algebra 3-5/27 An Analogy Principle for Estimating  In the population E[y | X ] = X so E[y - X |X] = 0 Continuing E[x i  i ] = 0 Summing, Σ i E[x i  i ] = Σ i 0 = 0 Exchange Σ i and E[] E[Σ i x i  i ] = E[ X ] = 0 E[X(y - X) ]= 0 Choose b, the estimator of  to mimic this population result: i.e., mimic the population mean with the sample mean Find b such that As we will see, the solution is the least squares coefficient vector.

6. Part 3: Least Squares Algebra 3-6/27 Population and Sample Moments We showed that E[ i |x i ] = 0 and Cov[x i, i ] = 0. If so, and if E[y|X] = X, then  = (Var[x i ]) -1 Cov[x i,y i ]. This will provide a population analog to the statistics we compute with the data.

7. Part 3: Least Squares Algebra 3-7/27 U.S. Gasoline Market, 1960-1995

8. Part 3: Least Squares Algebra 3-8/27 Least Squares  Example will be, G i on x i = [1, PG i, Y i ]  Fitting criterion: Fitted equation will be y i = b 1 x i1 + b 2 x i2 +... + b K x iK.  Criterion is based on residuals: e i = y i - b 1 x i1 + b 2 x i2 +... + b K x iK Make e i as small as possible. Form a criterion and minimize it.

9. Part 3: Least Squares Algebra 3-9/27 Fitting Criteria  Sum of residuals:  Sum of squares:  Sum of absolute values of residuals:  Absolute value of sum of residuals  We focus on now and later

10. Part 3: Least Squares Algebra 3-10/27 Least Squares Algebra

11. Part 3: Least Squares Algebra 3-11/27 Least Squares Normal Equations

12. Part 3: Least Squares Algebra 3-12/27 Least Squares Solution

13. Part 3: Least Squares Algebra 3-13/27 Second Order Conditions

14. Part 3: Least Squares Algebra 3-14/27 Does b Minimize e’e?

15. Part 3: Least Squares Algebra 3-15/27 Sample Moments - Algebra

16. Part 3: Least Squares Algebra 3-16/27 Positive Definite Matrix

17. Part 3: Least Squares Algebra 3-17/27 Algebraic Results - 1

18. Part 3: Least Squares Algebra 3-18/27 Residuals vs. Disturbances

19. Part 3: Least Squares Algebra 3-19/27 Algebraic Results - 2  A “residual maker” M = (I - X(X’X) -1 X’)  e = y - Xb= y - X(X’X) -1 X’y = My  My = The residuals that result when y is regressed on X  MX = 0 (This result is fundamental!) How do we interpret this result in terms of residuals? When a column of X is regressed on all of X, we get a perfect fit and zero residuals.  (Therefore) My = MXb + Me = Me = e (You should be able to prove this.  y = Py + My, P = X(X’X) -1 X’ = (I - M). PM = MP = 0.  Py is the projection of y into the column space of X.

20. Part 3: Least Squares Algebra 3-20/27 The M Matrix  M = I- X(X’X) -1 X’ is an nxn matrix  M is symmetric – M = M’  M is idempotent – M*M = M (just multiply it out)  M is singular; M -1 does not exist. (We will prove this later as a side result in another derivation.)

21. Part 3: Least Squares Algebra 3-21/27 Results when X Contains a Constant Term  X = [1,x 2,…,x K ]  The first column of X is a column of ones  Since X’e = 0, x 1 ’e = 0 – the residuals sum to zero.

22. Part 3: Least Squares Algebra 3-22/27 Least Squares Algebra

23. Part 3: Least Squares Algebra 3-23/27 Least Squares

24. Part 3: Least Squares Algebra 3-24/27 Residuals

25. Part 3: Least Squares Algebra 3-25/27 Least Squares Residuals

26. Part 3: Least Squares Algebra 3-26/27 Least Squares Algebra-3 M is NxN potentially huge

27. Part 3: Least Squares Algebra 3-27/27 Least Squares Algebra-4 MX =