1. Lecture 6 Preview: Ordinary Least Squares Estimation Procedure  The Properties Clint’s Assignment: Assess the Effect of Studying on Quiz Scores General Properties: Describing the Coefficient Estimate’s Probability Distribution Mean (Center) of the Coefficient Estimate’s Probability Distribution Variance (Spread) of the Error Term’s Probability Distribution Sample Size: The Number of Observations Reliability of the Coefficient Estimate Range of the Explanatory Variable Review: Ordinary Least Squares (OLS) Estimation Procedure Review: The Regression Model Variance (Spread) of the Coefficient Estimate’s Probability Distribution Strategy: General Properties and a Specific Application Best Linear Unbiased Estimation Procedure (BLUE) Theory: Additional Studying Increases Quiz Scores New Equation for the Ordinary Least Squares (OLS) Coefficient Estimate OLS Estimation Procedures and the Estimate’s Probability Distribution Importance of the Variance (Spread) Importance of the Mean (Center) Standard Ordinary Least Squares (OLS) Premises Question: How much confidence should we have in Clint’s results?

2. Clint’s Assignment: Assess the Effect of Studying on Quiz Scores The Regression Model: y t =  Const +  x x t + e t Econometrician’s Philosophy: If you lack the information to determine the value directly, do the best you can by estimating the value using the information you do have. Strategy: Apply the ordinary least square (OLS) procedure to the data from the first quiz to estimate the values of  Const and  x. Find the best fitting line, the equation that minimizes the sum of squared residuals. Assignment: Clint wishes to find the values of  Const and  x.  Const = Points given for showing up  x = Points earned for each minute studied y t = Actual quiz score x t = Minutes studied e t = Error term First Quiz Student x y 1 5 66 2 15 87 3 25 90 Ordinary Least Squares (OLS) Estimates b Const = Estimated points given for showing up = 63 b x = Estimated points for each minute studied = 1.2 Clint can never determine the actual values of  Const and  x. Theory: Additional studying increases quiz scores. But, he can’t;  Const and  x are not observable. Notation:  ’s denote the actual values; b’s denote the estimates. How can he proceed? Esty = 63 + 1.2x b x = b Const = e t reflects random influences: Mean[e t ] = 0

3. Error Term Reflects the Random Influences  e 1 has no systematic effect on Student 1’s quiz score Mean[e 1 ] = 0 Random influences should not systematically raise or lower a student’s quiz score; the mean of the probability distribution for each student’s error term must equal 0:  e 2 has no systematic effect on Student 2’s quiz score Mean[e 2 ] = 0  e 3 has no systematic effect on Student 3’s quiz score Mean[e 3 ] = 0 What do we know about the ordinary least squares (OLS) estimates? b x = b Const = = 1.2 The OLS estimate for the value of the coefficient is 1.2; Clint estimates that an additional minute of studying results in 1.2 additional points suggesting that the theory is correct. But, since random influences are present in the real world: The coefficient estimate is a random variable; even if we knew the actual value of the coefficient we could not predict the estimate’s value with certainty beforehand. We are all but certain that the numerical value of the coefficient estimate, 1.2, does NOT equal the actual value of the coefficient. The Model: y t =  Const +  x x t + e t  Const = Points given for showing up  x = Points earned for each minute studied y t = Actual quiz score x t = Minutes studied e t = Error term e t reflects random influences: Mean[e t ] = 0

4. Coefficient Reliability: How reliable is the coefficient estimate calculated from the results of the first quiz? That is, how confident should Clint be that the coefficient estimate, 1.2, will be close to the actual value? Theory Confidence: How much confidence should Clint have in the theory that studying more leads to higher quiz scores? Dividing Clint’s assignment into two related parts: Standard Ordinary Least Squares (OLS) Premises Error Term Equal Variance Premise: The variance of the error term’s probability distribution for each observation is the same; all the variances equal Var[e]: Var[e 1 ] = Var[e 2 ] = Var[e 3 ] = Var[e] Error Term/Error Term Independence Premise: The error terms are independent: Cov[e i, e j ] = 0. Knowing the value of one student’s (observation’s) error term would not help us predict value of the error term for any other student (observation). Long Term Plan: These premises allow us to consider the most straightforward case. After analyzing this case, we shall relax these premises to consider more complex scenarios. Explanatory Variable/Error Term Independence Premise: The explanatory variables, the x t ’s, and the error terms, the e t ’s, are not correlated. Question: How much confidence should we have in Clint’s results? Knowing the value of a student’s (observation’s) explanatory variable does not help you predict the value of that student’s (observation’s) error term.

5. The OLS Estimation Procedure: The General and the Specific Estimate  Const and  x by finding the b Const and b x that minimize the sum of squared residuals ——  Apply the estimation procedure once to the first quiz data:  Before Experiment (Quiz)  Estimate: Numerical Value  After Experiment (Quiz)  Mean[b x ] = ?  Mean and variance describe the center and spread of the estimate’s probability distribution Model: y t =  Const +  x x t + e t Var[b x ] = ?  Random Variable: Distribution b x = b Const = b x = b Const = = 1.2 = 81  18 = 63 OLS Equations As a consequence of random influences (the error term), we can not predict the numerical value of b x with certainty even if we knew the value of  x. General Properties versus One Specific Application How reliable are the estimates? Strategy: Use the general properties of the estimation procedure to assess the reliability. Use algebra to derive the equations for the probability distribution’s mean and variance. Check the algebra with a simulation by exploiting the relative frequency interpretation of probability. General properties of the estimation procedure: Probability distribution of the estimate. Mean and variance describe the center and spread of the probability distribution.

6. Review: Arithmetic of Means Mean of a constant plus a variable: Mean[c + x] = c + Mean[x] Mean of a constant times a variable: Mean[cx] = c Mean[x] Mean of the sum of two variables: Mean[x + y] = Mean[x] + Mean[y] Calculating the Mean and Variance: A New OLS Equation b x = Review: Error Term Reflects the Random Influences The mean of the probability distribution for each student’s error term equals 0: Mean[e 1 ] = Mean[e 2 ] = Mean[e 3 ] = 0 Calculating the Mean of the Coefficient Estimate’s Probability Distribution

7. Mean of the Coefficient Estimate’s Probability Distribution Mean[c + x] = c + Mean[x] Mean[cx] = cMean[x] Mean[x+y] = Mean[x] + Mean[y] Mean[cx] = cMean[x] Mean[e 1 ] = Mean[e 2 ] = Mean[e 3 ] = 0 Rewrite fraction as a product Mean[b x ] =  x bxbx

8. Question: Why is the mean of the estimate’s probability distribution important? An unbiased estimation procedure does not systematically underestimate or overestimate the actual value. If the probability distribution is symmetric we have even more intuition. the chances that the estimate is too low the chances that the estimate is too high equal Formally, an estimation procedure is unbiased whenever the mean of the estimate’s probability distribution equals the actual value. Mean[b x ] Probability Distribution of b x xx bxbx In one repetition: Equation: Simulations: Mean of Random Mean (Average) of Actual Variable b x Simulation Numerical Values from β x Mean[b x ] Repetitions the Experiments >1,000,000 22 44 66 22 44 66 The simulation confirms that OLS estimation procedure is unbiased. Question: Is our algebra correct?Answer: Yes  Lab 6.1 Mean[b x ] =  x  Average of the estimate’s numerical values after many, many repetitions Unbiased Estimation Procedure Relative Frequency Interpretation of Probability Average of the estimate’s numerical values after many, many repetitions =  x = Intuition

9. Variance largeVariance small  Small probability that the numerical value of the coefficient estimate from one repetition of the experiment will be close to actual value  Large probability that the numerical value of the coefficient estimate from one repetition of the experiment will be close to actual value  Estimate is unreliable  Estimate is reliable Unbiased Estimation Procedures: Importance of the Spread (Variance) The variance (spread) of distribution indicates the reliability of the estimate. Variance largeVariance small Probability Distributions of Estimates Actual Value The variance tells us how likely it is that the numerical value of the estimate resulting from one repetition of the experiment will be close to the actual value. Estimate When an estimation procedure is unbiased, the variance (spread) of the coefficient estimate’s probability distribution reveals the estimate’s reliability. The variance tells us how likely it is that the numerical value of the estimate resulting from one repetition of the experiment will be “close to” the actual value:

10. Review: Arithmetic of Variance Variance of the sum of a constant and a variable: Var[c + x] = Var[x] Variance of a constant times a variable: Var[cx] = c 2 Var[x] Variance of the sum of two variables: Var[x + y] = Var[x] + 2Cov[x, y] + Var[y] Relationships Used to Calculate the Variance of the Distribution of Coefficient Estimates Variance of the sum of two independent variables: Var[x + y] = Var[x] + Var[y] When the variables are independent we do not need to worry about the covariance. Review: Standard Ordinary Least Squares (OLS) Premises Error Term Equal Variance Premise: The variance of the error term’s probability distribution for each observation is the same; all the variances equal Var[e]: Var[e 1 ] = Var[e 2 ] = Var[e 3 ] = Var[e] Error Term/Error Term Independence Premise: The error terms are independent: Cov[e i, e j ] = 0. Knowing the value of the error term from one student would not help you predict the error term on any other student.

11. = Variance of the Coefficient Estimate’s Probability Distribution Var[c + x] = Var[x] Var[cx] = c 2 Var[x] e’s independent Var[cx] = c 2 Var[x] Var[e t ] = Var[e] Factor out Var[e] Rewrite the fraction as a product bxbx

12. Checking the Equation for the Variance of the Coefficient Estimate’s Probability Distribution Mean[b x ] =  x = 2 x 1 = 5x 2 = 15x 3 = 25 = 15 = (5  15) 2 + (15  15) 2 + (25  15) 2 = (  10) 2 + (0) 2 + (10) 2 = 100 + 0 + 100= 200 Var[e] = 500 = 2.50 22  2.50  x = 2 200 500 = 2.50 Question: Is our algebra correct?Answer: Yes 5002 2  Lab 6.2 Var[b x ] = Simulations: After Many, Many Repetitions Mean of Variance of Percent of Actual Numerical Numerical Estimates Values Sample x x Equations: Values of b x Values of b x Between β x Var[e] Size Min Max Mean[b x ] Var[b x ] from Exp’s. From Exp’s 1.0 and 3.0 3 0 30  47%

13. What Affects the Reliabilty of the Estimate Variance of the Error Term’s Probability Distribution Sample Size Sample Range Mean[b x ] = Estimation procedure is unbiased Simulations: After Many, Many Repetitions Mean of Variance of Percent of Actual Numerical Numerical Estimates Values Sample x x Equations: Values of b x Values of b x Between β x Var[e] Size Min Max Mean[b x ] Var[b x ] from Exp’s. From Exp’s 1.0 and 3.0 2 500 3 0 30 2 2.50  2.0  2.50  47% 50 200 2 50 5 0 30 .14  2.0 .14…  99% 2 50 5 10 201.25  2.0  1.25  63% 36040 Intuition: Will the estimate become more or less reliable? = x 1 = 5 x 2 = 15 x 3 = 25x 1 = 3 x 2 = 9 x 3 = 15 x 4 = 21 x 5 = 27x 1 = 11 x 2 = 13 x 3 = 15 x 4 = 17 x 5 = 19 = 200= 360= 40 =.25 = .14 = 1.25  x = 2 Var[e] = 50 More and/or Higher Quality Information  Less Variance  More Reliable Estimates 2 2  Lab 6.4  Lab 6.5 2 50 3 0 30 2.25  2.0 .25  95%  Lab 6.3 500 = 2.50Var[e] = 500 Less and/or Lower Quality Information  More Variance  Less Reliable Estimates Var[b x ] = As the variance of the error term’s probability distribution decreases: Increase More Does the quality of the information increase or decrease? Is an estimate more or less reliable? As sample size increases: MoreIs more or less information available? MoreIs an estimate more or less reliable? As the sample range decreases: DecreaseDoes the diversity of information increase or decrease? LessIs more or less information available? LessIs an estimate more or less reliable?

14. Best Linear Unbiased Estimation Procedure (BLUE) Why do we spend so much time on the ordinary least squares (OLS) estimation procedure when there are other procedures we could use to find the best fitting line? What is so special about the ordinary least square (OLS) procedure? Consider two other possible procedures to use to find the best fitting line: Any TwoMin-Max

15. OLS 2.0  2.0  1.4  60% Any Two Min-Max 2.0  2.0  13.9  29% 2.0  2.0  1.7  55% How are the procedures similar? All three procedures appear to be unbiased. After many, many repetitions of the experiment, the average of the estimates equals the actual value. How do they differ? Fraction of repetitions that are “close to” the actual value is the greatest. Probability that the estimate from one repetition of the experiment will be “close to” the actual value is the greatest. Consequently, the ordinary least squares (OLS) procedure is the most reliable of the three. Sample Size = 5 Simulations: After Many, Many Repetitions Mean (Average) Variance Percent of Experiments’ of Experiments’ of Estimates Estimation Actual Numerical Values Numerical Values Between Procedure of  x (Estimates) (Estimates) 1.0 and 3.0 Gauss-Markov Theorem: Of all linear unbiased estimation procedures, the ordinary least squares (OLS) estimation procedure has the smallest variance whenever the standard ordinary least squares (OLS) regression premises are satisfied.  Lab 6.6 The OLS procedure has the smallest variance. Why is this important? The variances differ.