1. Topic 3: Simple Linear Regression

2. Outline Simple linear regression model –Model parameters –Distribution of error terms Estimation of regression parameters –Method of least squares –Maximum likelihood

3. Data for Simple Linear Regression Observe i=1,2,...,n pairs of variables Each pair often called a case Y i = i th response variable X i = i th explanatory variable

4. Simple Linear Regression Model Y i = b 0 + b 1 X i + e i b 0 is the intercept b 1 is the slope e i is a random error term –E( e i )=0 and s 2 ( e i ) =s 2 –e i and e j are uncorrelated

5. Simple Linear Normal Error Regression Model Y i = b 0 + b 1 X i + e i b 0 is the intercept b 1 is the slope e i is a Normally distributed random error with mean 0 and variance σ 2 e i and e j are uncorrelated → indep

6. Model Parameters β 0 : the intercept β 1 : the slope σ 2 : the variance of the error term

7. Features of Both Regression Models Y i = β 0 + β 1 X i + e i E (Y i ) = β 0 + β 1 X i + E(e i ) = β 0 + β 1 X i Var(Y i ) = 0 + var(e i ) = σ 2 –Mean of Y i determined by value of X i –All possible means fall on a line –The Y i vary about this line

8. Features of Normal Error Regression Model Y i = β 0 + β 1 X i + e i If e i is Normally distributed then Y i is N(β 0 + β 1 X i, σ 2 ) (A.36) Does not imply the collection of Y i are Normally distributed

9. Fitted Regression Equation and Residuals Ŷ i = b 0 + b 1 X i –b 0 is the estimated intercept –b 1 is the estimated slope e i : residual for i th case e i = Y i – Ŷ i = Y i – (b 0 + b 1 X i )

10. X=82 Ŷ 82 =b 0 + b 1 82 e 82 =Y 82 -Ŷ 82

11. Plot the residuals proc gplot data=a2; plot resid*year vref=0; where lean ne.; run; Continuation of pisa.sas Using data set from output statement vref=0 adds horizontal line to plot at zero

12. e 82

13. Least Squares Want to find “best” b 0 and b 1 Will minimize Σ(Y i – (b 0 + b 1 X i ) ) 2 Use calculus: take derivative with respect to b 0 and with respect to b 1 and set the two resulting equations equal to zero and solve for b 0 and b 1 See KNNL pgs 16-17

14. Least Squares Solution These are also maximum likelihood estimators for Normal error model, see KNNL pp 30-32

15. Maximum Likelihood

16. Estimation of σ 2

17. Standard output from Proc REG Analysis of Variance SourceDF Sum of Squares Mean SquareF ValuePr > F Model115804 904.12<.0001 Error11192.2857117.48052 Corrected Total1215997 Root MSE4.18097R-Square0.9880 Dependent Mean693.69231Adj R-Sq0.9869 Coeff Var0.60271 MSE df e s

18. Properties of Least Squares Line The line always goes through Other properties on pgs 23-24

19. Background Reading Chapter 1 –1.6 : Estimation of regression function –1.7 : Estimation of error variance –1.8 : Normal regression model Chapter 2 –2.1 and 2.2 : inference concerning  ’s Appendix A –A.4, A.5, A.6, and A.7