1. Evgeniya Anatolievna Kolomak, Professor
Econometrics III
Evgeniya Anatolievna Kolomak,
Professor

2. Theme 1. The classical multiple linear regression

3. The classical multiple linear regression. Model specification
yi=f(xi1, xi2,…, xiK)+εi
yi=β1xi1+β2xi2+…+βKxiK+εi, i=1,…,n
Where
y – dependent (explained) variable or regressand;
x1,x2,…, xK - independent (explanatory) variables or regressors;
εi - random disturbance (errors of the variables).

4. The classical multiple linear regression. Model specification
We assume that each observation (yi, xi1, xi2,…,xiK), i=1,…,n is generated by the process described by yi=β1xi1+β2xi2+…+βkxiK + εi deterministic part random part

5. The classical multiple linear regression. Assumptions of the model
Linear functional form
Identifiability of the model parameters
Expected value of the disturbance given observed information
Variance and covariance of the disturbances given observed information
Nature of the independent variables
Probability distribution of the stochastic part of the model

6. Assumptions of the model. 1. Linear functional form
Let
y - column vector (nx1) of the observations y1,y2,..,yn
xk – column vector (nx1) of the observations on variable xk;
X – matrix (nxK) assemble xk;
ε - column vector (nx1) containing the n disturbances.
The model yi=β1xi1+β2xi2+…+βkxiK+εi can be written
y= x1β1+…..+xkβk+ε
y=Xβ+ε
Our primary interest is estimation and inference about β

7. Assumptions of the model. 2. Identifiability of the model parameters
Identification condition: X is nxK matrix with rank K The columns of X are linearly independent and there are at least K observations.

8. Assumptions of the model. 3. Expected value of the disturbance εi
The disturbance conditioned on observations X is assumed to have expected value 0 at every observation E[εi│X]=0 for all i=1,…,n The assumption implies E[y│X]=Xβ

9. Assumptions of the model. 4
Assumptions of the model. 4. Variance and covariance of the disturbances
Constant variance or homoscedasticity Var[εi│X]=σ2 for all i=1,…,n Absence of correlation across observations or absence of autocorrelation Cov[εi, εj │X]=0 for all i≠j Summarizing E[εεT│X]= σ2 I I – identity matrix

10. Assumptions of the model. 5. Non-stochastic regressors
X is known nxK matrix of constants X is non-stochastic matrix X and ε are uncorrelated

11. Assumptions of the model. 6. Probability distribution of ε
The disturbances εi are normally distributed ε│X ~ N[0, σ2 I] Normality enables to obtain several exact statistical results and to construct test statistics.

12. Assumptions of the classical regression model. Summary
Number
Assumption A1
y=Xβ+ε A2
X is nxK matrix with rank K A3
E[ε│X]=0 A4
E[εεT│X]= σ2 I A5
X is non-stochastic matrix A6
ε│X ~ N[0, σ2 I]

13. Least squares regression
The population quantities are β and εi The sample estimations are b and ei For any value of b we estimate ei = yi - xiTb yi = xiTb + ei Vector b is chosen so that the fitted line xiTb is close to the data points. The fitting criterion is least squares.

14. Least squares regression
The solving problem is: Minimize b eTe = (y – Xb)T(y – Xb) b satisfies the least squares normal equations: XTXb=XTy If the inverse of XTX exists, which follows from the full rank assumption, then b=(XTX)-1XTy

15. Least squares regression
Algebraic properties of the least squares Let the first column of X is 1s, then 1. The least squares residuals sum to zero. Σiei=0 2. The regression passes through the point of means of the data ͞y=͞xTb 3. The mean of fitted values from the regression equals the mean of the actual values ̂y=Xb

16. Partitioned regression
Suppose the regression involves two set of variables X1 and X2. However we are interested in estimates for β2 only.
The normal equations are:
We first solve for b1
b1 is the coefficient vector in the regression of (y-x2Tb2) on x1 .

17. Partitioned regression
Given the solution for b1 we solve the second equation I – identity matrix and Matrix M1∙is: 1) symmetric ( ) and 2) idempotent ( ). Proof: 1. 2.

18. Partitioned regression
Properties of matrixes M1 and P
M1∙y – vector of residuals in the regression of y on x1
M1∙x2 – matrix of residuals of the regressions of x2 on x1
Let x2*=M1∙x2 and y*=M1∙y.
Then

19. Partitioned regression
Theorem Frisch-Waugh. The sub-vector b2 is the set of coefficients obtained when the residuals from a regression of y on X1 are regressed on the set of residuals obtained when each column of X2 is regressed on X1.
The algorithm is as follows:
To regress y on X1 and to estimate residuals ey .
To regress each column of X2 on X1 and estimate residuals ex2.
To regress ey on ex2 and estimate coefficients b2 and residuals e.

20. Goodness of fit and the analysis of variance
The total variation in y: TSS=Σi=1n(yi - ͞y)2
In terms of regression equation: yi =xiTb+ei
yi - ͞y = (xi - ͞x)Tb ei
regression part error part
For the full set of observations:
Total sum of squares = regression sum of squares + residuals sum of squares
TSS=RegSS+RSS

21. Goodness of fit and the analysis of variance
Coefficient of determination:

22. Goodness of fit and the analysis of variance
Source
Degrees of freedom
Mean Square
Regression
𝑏 𝑇 𝑋 𝑇 𝑦−𝑛 𝑦 2
K-1 (assuming a constant term)
Residuals
𝑒 𝑇 𝑒
n-k
𝑠 2
Total
𝑦 𝑇 𝑦−𝑛 𝑦 2
n-1
𝑆 𝑦𝑦 𝑛−1 = 𝑠 𝑦 2
Coefficient of determination
𝑅 2 =1− 𝑒 𝑇 𝑒/( 𝑦 𝑇 𝑦−𝑛 𝑦 2 )

23. Statistical properties of the Least Squares Estimator in finite sample
Gaus-Markov Theorem. In the classical liner regression model y=Xβ+ε, the least squares estimator b is the minimum variance linear unbiased estimator of β

24. Statistical properties of the Least Squares Estimator in finite sample
If the disturbances εi are normally distributed ε│X ~ N[0, σ2 I] then b│X ~ N[β, σ2 (XTX)-1] bk│X ~ N[βk, σ2 (XTX)-1kk] Rao-Blackwell Theorem. In the classical liner regression model y=Xβ+ε with normally distributed disturbance the least squares estimator b has the minimum variance of all unbiased estimators.

25. Statistical properties of the Least Squares Estimator in finite sample
If we wish to test hypotheses about β or to construct confidence intervals, we need an estimate of the covariance matrix Var[b]= σ2 (XTX)-1 Since ei is an estimate of εi, estimation of σ2 is And Est. Var[b]= s2 (XTX)-1

26. Statistical properties of the Least Squares Estimator in finite sample
Theorem. If the disturbance ε is normally distributed, then the ratio 𝑏 𝑘 − 𝛽 𝑘 𝑠 2 𝑆 𝑘𝑘 ~ 𝑡 𝑛−𝐾 Where Skk – kth diagonal element of (XTX)-1. We can use t-distribution to form confidence interval or test hypotheses about individual elements of β.

27. Statistical properties of the Least Squares Estimator in finite sample
A common test is whether a parameter βk is significantly different from 0.
𝑏 𝑘 𝑠 2 𝑆 𝑘𝑘 ~ 𝑡 𝑛−𝐾
A confidence interval for βk is
𝑃𝑟𝑜𝑏(𝑏 𝑘 − 𝑡 𝛼 2 𝑠 𝑏 𝑘 ≤ 𝛽 𝑘 ≤ 𝑏 𝑘 + 𝑡 𝛼 2 𝑠 𝑏 𝑘 )=1−𝛼
Where 𝑠 𝑏 𝑘 = 𝑠 2 𝑆 𝑘𝑘 - estimate of standard error of βk

28. Statistical properties of the Least Squares Estimator in finite sample
Testing the significance of the regression equation as a whole (β2= β3 =…= βK =0) 𝑅 2 /(𝐾−1) (1− 𝑅 2 )/(𝑛−𝐾) ~ 𝐹 𝐾−1,𝑛−𝐾 If the disturbances are normally distributed, then we may carry out tests and construct confidence intervals for the parameters without making any change in our procedures