1. Lecturer Dr. Veronika Alhanaqtah
ECONOMETRICS
Lecturer
Dr. Veronika Alhanaqtah

2. Topic 2. Theory of the regression analysis
2.1 Numerical characteristics of random variables:
Mean
Expected value
Standard deviation
Variance
Covariance
Covariance matrix
2.2 Econometric data
2.3 Action plan in regression analysis. OLS
2.4 OLS assumptions. Gauss-Markov theorem
2.5 Confidence intervals and hypothesis testing
2.6 Interpretation of a standard table in any statistical package

3. 2.1 Numerical characteristics of random variables
Mean (measure of center of a distribution) is the arithmetic average.
Example: Compute the mean for the variable

4. 2.1 Numerical characteristics of random variables
Expected value (mathematical expectation, 1st moment, mean) is the probability-weighted average for all possible values. 

5. Example: Expected value
There are 2 business projects with the same prognostic sum of capital investments. Which of the projects is more preferable upon the criteria of expected value of income?
Project A
Project B
Income (x)
Probability (p) 30
0,10 20 35
0,20
0,15 40
0,40
0,30 45 50
0,35 80
Answer: Project B is favorable upon the criteria of the expected value of income.
At the same time it is more risky due to the higher range of incomes than in the Project A.

6. 2.1 Numerical characteristics of random variables
Standard deviation (SD) is the average distance to the mean. It quantifies the amount of variation of values in a data set. 
Example: Compute SD for the variable
In statistical formulas stands for the population SD;
stands for the sample SD.

7. 2.1 Numerical characteristics of random variables
Variance (Var) measures how far a set of values is spread out.
Variance is the squared standard deviation ( ).
Variance of zero indicates that all the values are identical.
Variance is always non-negative: a small variance indicates that the data points tend to be very close to the mean (expected value) and hence to each other, while a high variance indicates that the data points are very spread out around the mean and from each other.
In regression analysis SD represents the magnitude of the noise in the data. The standard error (se) is an estimator of the population SD. In other words, se is equivalent to SD. Thus, standard error is the squared root from the variance:

8. 2.1 Numerical characteristics of random variables
Covariance is a measure showing of how much two random variables change together.
If the greater values of one variable mainly correspond with the greater values of the other variable, and the same holds for the smaller values, i.e., the variables tend to show similar behavior, the covariance is positive.
In the opposite case, when the greater values of one variable mainly correspond to the smaller values of the other, i.e., the variables tend to show opposite behavior, the covariance is negative.
The sign of the covariance therefore shows the tendency in the linear relationship between the variables.
The magnitude of the covariance is not easy to interpret.
The normalized version of the covariance – the linear correlation coefficient (r) – shows by its magnitude the strength of the linear relation.

9. 2.1 Numerical characteristics of random variables
Covariance is a measure showing of how much two random variables change together.
Notably:
Correlation is dimensionless while covariance is in units obtained by multiplying the units of the two variables.
The covariance of a variable with itself ( ) is called the variance; it is commonly denoted as (squared SD).
The correlation of a variable with itself always equals to one (except in the degenerate case where the two variances are zero, in which case the correlation does not exist).

10. Average annual income, %
Example: Covariance
We consider information on financial funds with low level of investment risk. In the table below is the information on average annual incomes during 6 years and the share of expenses. Compute covariance and linear correlation coefficient. Interpret the results.
Average annual income, %
Share of expenses
11,00
0,59
18,20
1,09
15,10 1
12,30
0,81
12,00
0,8
12,10
0,78

11. Example: Covariance Solution: -2,45 -0,255 0,62475 4,75 0,245 1,16375
1,65
0,155
0,25575
-1,15
-0,035
0,04025
-1,45
-0,045
0,06525
-1,35
-0,065
0,08775
Sum
2,2375
Linear correlation coefficient was computed in Excel (function CORREL).
We observe strong positive linear relationship.
The magnitude of the covariance we don’t interpret.

12. 2.1 Numerical characteristics of random variables
Covariance matrix (dispersion matrix, variance–covariance matrix) generalizes the notion of variance for multiple dimensions.
It is a matrix whose element in the i, j position is the covariance  between the i th and j th elements of a random vector. 
Because the covariance of the i th random variable with itself is simply that random variable's variance, each element on the principal diagonal of the covariance matrix is the variance of one of the random variables.
Because the covariance of the i th random variable with the j th one is the same thing as the covariance of the j th random variable with the i th one, every covariance matrix is symmetric.
Computation of a covariance matrix in R-Studio: vcov(model)

13. 2.2 Econometric data
Econometrics is the science that enables to discover and analyze patterns in data.
This science tries to answer such questions as:
How can one variable (x) influence another (y)?
How can we predict values of y? What will be tomorrow?
Answers to these and similar questions we get with the help of models. For example:

14. What types of data are used in Econometrics?
2.2 Econometric data
What types of data are used in Econometrics?

15. Population (thousands)
One-dimensional time series:
we observe the dynamics of just one random variable in time
Multidimensional time series:
we observe several variables in different moments of time
Year
GNP (mln JD)
2004
7320.8
2005
8285.1
2006
9163.9
2007
2008
2009
2010
2011
2012
2013
2014
2015
Year
Population (thousands)
Unemployment rate (%)
2010
7.4
2011
6.5
2012
2013
5.5
(1) Time series is some ordered data points for different moments of time

16. (2) CROSS-sectional data
Represent observations for different objects (or individuals) at a single point in time.
In cross data there are:
one dependent variable, y
several independent variables (regressors): x1, x2, …, xn
a number of observations n for every variable: y1, y2, …, yn
Country
Gold
Silver
Bronze
Russian 13 11 9
Norway 5 10
Canada US 7 12

17. (3) PANEL data (longitudinal data)
combination of cross data and time series
Sample
Start-Year
Households
Persons A
West-German (residents)
1984
n=4,528
n=12,245 B
Foreigners
n=1,393
German reunification C
East-Germans
1990
n=2,179
n=4,453 D
Immigrants
1994/1995
n=522
n=1,078 E
Refreshment
1998
n=1,067
n=1,923 F
Innovation
2000
n=6,052
n=10,890 G
High Income
2002
n=1,224
n=2,671 H
2006
n=1,506
n=2,616 I
Incentive/Refreshment
2009
n=1,531
n=2,509 J
2011
n=3.136
n=5.161
Example: The German Socio-Economic Panel is a  longitudinal panel dataset of the population in Germany. It is a household based study which started in 1984 and which reinterviews adult household members annually.

18. 2.3 Action plan in regression analysis
Regression analysis allows us to find out the estimators of beta-coefficients.
Think on an adequate model (before, strongly recommended to plot data points on the graph to visualize a relationship);
Choose the method which would allow to get estimators of unknown parameters β1 and β2;
We can estimate unknown β1 and β2 by different methods. Nevertheless, the most simple (from the computation standpoint) and popular is the Ordinary Least Squares method (OLS). Among other methods of estimation of coefficients of a regression are method of moments and maximum likelihood method.
Predict, replacing unknown parameters β1 and β2 by its estimators. In other words, if we want to predict or interpret these coefficients, we can use instead of unknown β1 and β2 its estimators.

19. 2.3 Ordinary least squares (OLS)
OLS is the method that allows to obtain estimators of unknown parameters β1 and β2 on the basis of real data.
Recall:
To each point in the data set there is associated an “error (the positive or negative vertical distance from the point to the line).
The line that best fits the data in the sense of minimizing the sum of squared errors (SSE) is called the least squares regression line.
The idea of OLS is to solve the following optimization task: to find out such beta-estimators so that SSE be minimized.

20. 2.3 Ordinary least squares (OLS)
Regression analysis allows to find out the estimators of beta- coefficients.
Remember: empirical coefficients of a regression line and
are just estimators of theoretical coefficients β1 and β2, and a regression line represents just the common tendency in behavior of data points. Beta-estimators don’t give us the answers to such questions as:
how reliable beta-estimators are;
how close beta-estimators are to their prototypes β1 and β2;
how close the empirical regression line is to the line of a population;
how close y-estimator ( ) is to the conditional expected value
To answer these questions additional investigations are needed. Nevertheless, the regression line serves as a powerful qualitative instrument of analysis and prediction.
After interpreting the results of analysis econometricians question the quality of estimators of coefficients.

21. 2.4 OLS assumptions. Gauss-Markov theorem
From the theoretical regression model it follows that y is dependent on a regressor (x) and an error ( ) .
Consequently, variable y is a random variable directly dependent on the error. Thus, until we don’t know the probabilistic behavior of the error term we can’t be sure in a good quality of beta-estimators.
Estimators of coefficients β1 and β2 as well as the quality of the whole regression line are sufficiently dependent on qualities of an error.
In order to get the best, reliable, qualitative beta- estimators with the help of OLS-method assumptions in relation to the error must be hold.

22. OLS assumptions
(1) The expected value of a residual is equal to zero for all observations:
(2) The variance of each residual is constant (equal, homoscedastic) for every observation:
Feasibility of this assumption is called homoscedasticity.
Infeasibility of this assumption is called heteroscedasticity.
(3) Residuals are uncorrelated between observations
(4) Residuals are independent on regressors (x)
(5) Model is linear in relation to its parameters
It means that beta-estimators are linear in relation to yi:
where cn are values which are dependent only on regressors xi but not on the dependent variable y.

23. Gauss-Markov theorem
If the set of OLS assumptions is hold, then beta- estimators possess the following good qualities: unbiasedness, consistency and efficiency.
In the special literature such qualitative beta-estimators are called as BLUE-estimators (Best Linear Unbiased Estimators).

24. Unbiasedness, Consistency, Efficiency
Estimators are unbiased.
It means that on average beta-estimators fall into unknown coefficient β: or
Estimators are consistent.
It means that when the number of observations grows infinitely (asymptotically), variance of beta-estimators tends to zero (unbiased).
In other words, when the number of observations grows, beta-estimators become more and more reliable.
Estimators are efficient.
It means they have the least variance in comparison with other estimators of a certain parameter.
In other words, variance of beta-coefficients is the least possible.

25. Other assumptions for a classical linear regression:
Regressors are not random variables.
Residuals have normal distribution (applied for small samples).
The number of observations is much bigger than the number of regressors.
There are no specification problems (the model is adequate to the data).
There is no perfect multicollinearity (ideal linear relationship between regressors of the model).

26. 2.5 Confidence intervals and hypothesis testing
In order to test hypothesis and construct confidence intervals in small samples we must assume normality of residuals. For large samples (when n is asymptotical), this assumption is not necessarily must be hold.
In other words, when we conduct regression analysis we should have either large sample or assume normality of residuals. Otherwise, we can’t test hypothesis.
Consequently, confidence intervals for coefficients we may construct using one of the two approaches:
Asymptotically:
When the number of observations is large, the outcome of calculations has standard normal distribution.
Normality of residuals:
When the number of observations is rather small the outcome of calculations has t-distribution (Student distribution) with (n-k) degrees of freedom; where n - number of observations, k – number of linear relationships in a model.

27. 2.5 Confidence intervals
Standard approach to construct confidence intervals for coefficients: The 95 % -confidence interval is roughly calculated:

28. 2.5 Hypothesis testing. Critical value approach
Algorithm of any test:
Make assumptions (for example, asymptotical or the normality of residuals is required).
State an initial hypothesis H0 and an alternative hypothesis Ha. 
Choose a level of significance α (type I error).
Calculate observable test statistic.
Look at the critical value of statistics (apply statistical table).
Compare observable and critical values statistics.
Make a statistical inference (conclusion), whether H0 is rejected (in favor of Ha) or doesn’t.
If test statistic more than critical point statistic,
then null-hypothesis is rejected.
Interpret the result in the context of the problem.

29. 2.5 Hypothesis testing. p-value approach
Algorithm :
Make assumptions (for example, asymptotical or the normality of residuals is required).
State an initial hypothesis H0 and an alternative hypothesis Ha. 
Choose a level of significance α (type I error).
Compute probability value (p-value) using a statistical software.
Compare level of significance α with p-value.
Make a statistical inference, whether H0 is rejected (in favor of Ha) or doesn’t. If
then null-hypothesis is rejected.
Interpret the result in the context of the problem.

30. 2.6. Interpretation of a standard table in any statistical package
Example: Data comprises fuel consumption and 10 aspects of automobile design and performance for 32 automobiles. Estimate a regression for fuel consumption (variable ‘mpg’) and verify the significance of variables.
Estimate
Std. Error
t value
Pr(>|t|)
(Intercept)
12.119
1.98e-12 ***
Displacement
0.240
Horsepower
-3.163 **
Weight
-2.633 *
Transmission
1.505
Multiple R-squared:
Adjusted R-squared:
Intercept is significant (<1%)
Variable ‘Horsepower’ is significant (1%)
Variable ‘Weight’ is significant (5%)
Variables ‘Displacement’ and ‘Transmission’ are not significant

31. 2.6. Interpretation of a standard table in any statistical package
Estimate
Std. Error
t value
Pr(>|t|)
(Intercept)
12.119
1.98e-12 ***
Displacement
0.240
Horsepower
-3.163 **
Weight
-2.633 *
Transmission
1.505
Multiple R-squared:
Adjusted R-squared:
1st column (estimate) shows beta-estimators.
2nd column (Std.Error) shows standard errors for every estimated β coefficient.
3rd column (t value) shows t-statistic which we compare with t-critical.
4th column (Pr) shows p-values.
* means that a coefficient is significant when the level of significance is 5 %.
** means that a coefficient is significant when the level of significance is 1 %.
*** means that a coefficient is significant when the level of significance is less than 1 % .