1. Statistical Inference and Regression Analysis: GB.3302.30
Professor William Greene
Stern School of Business
IOMS Department
Department of Economics

2. Statistics and Data Analysis
Part 7 – Regression Model Regression Diagnostics

3. Using the Residuals How do you know the model is “good?”
Various diagnostics to be developed over the semester.
But, the first place to look is at the residuals.

4. Residuals Can Signal a Flawed Model
Standard application: Cost function for output of a production process.
Compare linear equation to a quadratic model (in logs)
(123 American Electric Utilities)

5. Electricity Cost Function

6. Candidate Model for Cost
Log c = a + b log q + e
Most of the points in this area are above the regression line.
Most of the points in this area are above the regression line.
Most of the points in this area are below the regression line.

7. A Better Model?
Log Cost = α + β1 logOutput + β2 [logOutput] ε

8. Candidate Models for Cost
The quadratic equation is the appropriate model.
Logc = a + b1 logq + b2 log2q + e

9. Missing Variable Included
Residuals from the quadratic cost model
Residuals from the linear cost model

10. Unusual Data Points
Outliers have (what appear to be) very large disturbances, ε
Wolf weight vs. tail length The 500 most successful movies

11. Outliers
99.5% of observations will lie within mean ± 3 standard deviations. We show (a+bx) ± 3se below.)
Titanic is 8.1 standard deviations from the regression!
Only 0.86% of the 466 observations lie outside the bounds. (We will refine this later.)
These observations might deserve a close look.

12. logPrice = a + b logArea + e
Prices paid at auction for Monet paintings vs. surface area (in logs)
logPrice = a + b logArea + e
Not an outlier: Monet chose to paint a small painting.
Possibly an outlier: Why was the price so low?

13. What to Do About Outliers
(1) Examine the data
(2) Are they due to mismeasurement error or obvious “coding errors?” Delete the observations.
(3) Are they just unusual observations? Do nothing.
(4) Generally, resist the temptation to remove outliers. Especially if the sample is large. (500 movies is large. 10 wolves is not.)
(5) Question why you think it is an outlier. Is it really?

14. Regression Options

15. Diagnostics

16. On Removing Outliers
Be careful about singling out particular observations this way.
The resulting model might be a product of your opinions, not the real relationship in the data.
Removing outliers might create new outliers that were not outliers before.
Statistical inferences from the model will be incorrect.

17. Statistics and Data Analysis
Part 7 – Regression Model Statistical Inference

18. b As a Statistical Estimator
What is the interest in b?
 = dE[y|x]/dx
Effect of a policy variable on the expectation of a variable of interest.
Effect of medication dosage on disease response
… many others

19. Application: Health Care Data
German Health Care Usage Data, There are altogether 27,326 observations on German households,
DOCTOR = 1(Number of doctor visits > 0) HOSPITAL = 1(Number of hospital visits > 0)
HSAT =  health satisfaction, coded 0 (low) - 10 (high)  
DOCVIS =  number of doctor visits in last three months HOSPVIS =  number of hospital visits in last calendar year PUBLIC =  insured in public health insurance = 1; otherwise = ADDON =  insured by add-on insurance = 1; otherswise = 0
INCOME =  household nominal monthly net income in German marks / HHKIDS = children under age 16 in the household = 1; otherwise = EDUC =  years of schooling
AGE = age in years
MARRIED = marital status
EDUC = years of education 19

20. Regression? Population relationship Income =  + Health + 
(For this population, Income = Health +  E[Income | Health] = Health

21. Distribution of Health

22. Distribution of Income

23. Average Income | Health
Nj =

24. b is a statistic Random because it is a sum of the ’s.
It has a distribution, like any sample statistic

25. Sampling Experiment
500 samples of N=52 drawn from the 27,326 (using a random number generator to simulate N observation numbers from 1 to 27,326)
Compute b with each sample
Histogram of 500 values

27. Conclusions Sampling variability Seems to center on 
Appears to be normally distributed

28. Distribution of slope estimator, b
Assumptions:
(Model) Regression: yi =  + xi + i
(Crucial) Exogenous data: data x and noise  are independent; E[|x]=0 or Cov(,x)=0
(Temporary) Var[|x] = 2, not a function of x (Homoscedastic)
Results: What are the properties of b?

30. (1) b is unbiased and linear in 

31. (2) b is efficient
Gauss – Markov Theorem: Like Rao Blackwell. (Proof in Greene)
Variance of b is smallest among linear unbiased estimators.

32. (3) b is consistent

33. Consistency: N=52 vs. N=520

34. a is unbiased and consistent

35. Covariance of a and b

36. Inference about  Have derived expected value and variance of b.
b is a ‘point’ estimator
Looking for a way to form a confidence interval.
Need a distribution and a pivotal statistic to use.

37. Normality

38. Confidence Interval

39. Estimating sigma squared

40. Useable Confidence Interval
Use s instead of s.
Use t distribution instead of normal.
Critical t depends on degrees of freedom
b - ts <  < b + ts

41. Slope Estimator

42. Regression Results
Ordinary least squares regression
LHS=BOX Mean =
Standard deviation =
No. of observations = DegFreedom Mean square
Regression Sum of Squares =
Residual Sum of Squares =
Total Sum of Squares =
Standard error of e = Root MSE
Fit R-squared = R-bar squared
Model test F[ 1, 60] = Prob F > F*
| Standard Prob % Confidence
BOX| Coefficient Error t |t|>T* Interval
Constant| **
CNTWAIT3| ***
Note: ***, **, * ==> Significance at 1%, 5%, 10% level.

43. Hypothesis Test about 
Outside the confidence interval is the rejection for hypothesis tests about 
For the internet buzz regression, the confidence interval is to
The hypothesis that  equals zero is rejected.

44. Statistics and Data Analysis
Part 7-3 – Prediction

45. Predicting y Using the Regression
Actual y0 is  + x0 + 0
Prediction is y0^ = a + bx0 + 0
Error is y0 – y0^ = (a-) + (b-)x0 + 0
Variance of the error is Var[a] + x02 Var[b] + 2x0 Cov[a,b] + Var[0]

46. Prediction Variance

47. Quantum of Solace Actual Box = $67.528882M
a=-14.36, b= , N=62, sb = , s2 =
buzz = 0.76, prediction =
Mean buzz =
(buzz – mean)2 =
Sforecast =
Confidence interval = / ( ) = to (Note: The confidence interval contains the value)

48. Forecasting Out of Sample
Regression Analysis: G versus Income
The regression equation is
G = Income
Predictor Coef SE Coef T P
Constant
Income
S = R-Sq = 88.0% R-Sq(adj) = 87.8%
How to predict G for 2020? You would need first to predict Income for 2020.
How should we do that?
Per Capita Gasoline Consumption vs. Per Capita Income,

49. The Extrapolation Penalty
The interval is narrowest at x* = , the center of our experience. The interval widens as we move away from the center of our experience to reflect the greater uncertainty. (1) Uncertainty about the prediction of x (2) Uncertainty that the linear relationship will continue to exist as we move farther from the center.

50. An Enduring Art Mystery
Graphics show relative sizes of the two works.
The Persistence of Econometrics. W. Greene, 2018
Why do larger paintings command higher prices?
The Persistence of Memory. Salvador Dali, 1931

51. The Data
Note: Using logs in this context. This is common when analyzing financial measurements (e.g., price) and when percentage changes are more interesting than unit changes. (E.g., what is the % premium when the painting is 10% larger?)

52. Monet in Large and Small
Sale prices of 328 signed Monet paintings
The residuals do not show any obvious patterns that seem inconsistent with the assumptions of the model.
Log of $price = a + b log surface area + e

53. Monet Regression: There seems to be a regression. Is there a theory?

55. "Morning", Claude Monet , oil on canvas 200 x 425 cm, Musée de l Orangerie, Paris France. Left panel
167” (13 feet 11 inches)
78.74” (6 Feet 7 inch)
32.1” (2 feet 8 inches)
26.2” (2 feet 2.2”)

56. Predicted Price for a Huge Painting

57. Prediction Interval for Price

58. Use the Monet Model to Predict a Price for a Dali?
118” (9 feet 10 inches)
32.1” (2 feet 8 inches)
26.2” (2 feet 2.2”)
157” (13 Feet 1 inch)
Average Sized Monet
Hallucinogenic Toreador

60. Normality Necessary for t statistics and confidence intervals
Residuals reveal whether disturbances are normal?
Standard tests and devices

61. Normally Distributed Residuals?
Kolmogorov-Smirnov test of F(E )
vs. Normal[ , ^2]
******* K-S test statistic =
******* 95% critical value =
******* 99% critical value =
Normality hyp. should be rejected.

62. Nonnormal Disturbances
Appeal to the central limit theorem
Use standard normal instead of t
t is essentially normal if N > 50.

63. Statistics and Data Analysis
Part 7-4 – Multiple Regression

64. Box Office and Movie Buzz

65. Box Office and Budget

66. Budget and Buzz Effects

69. An Enduring Art Mystery
Graphics show relative sizes of the two works.
The Persistence of Statistics. Rice, 2007
Why do larger paintings command higher prices?
The Persistence of Memory. Salvador Dali, 1931

70. Monet Regression: There seems to be a regression. Is there a theory?

71. How much for the signature?
The sample also contains 102 unsigned paintings
Average Sale Price
Signed $3,364,248
Not signed $1,832,712
Average price of signed Monet’s is almost twice that of unsigned

72. Can we separate the two effects?
Average Prices
Small Large
Unsigned 346, ,795,000
Signed , ,556,490
What do the data suggest?
(1) The size effect is huge
(2) The signature effect is confined to the small paintings.

73. A Multiple Regression b2
Ln Price = a + b1 ln Area + b2 (0 if unsigned, 1 if signed) + e

74. Monet Multiple Regression
Regression Analysis: ln (US$) versus ln (SurfaceArea), Signed
The regression equation is
ln (US$) = ln (SurfaceArea) Signed
Predictor Coef SE Coef T P
Constant
ln (SurfaceArea)
Signed
S = R-Sq = 46.2% R-Sq(adj) = 46.0%
Interpretation (to be explored as we develop the topic): (1) Elasticity of price with respect to surface area is – very large
(2) The signature multiplies the price by exp(1.2618) (about 3.5), for any given size.

75. Ceteris Paribus in Theory
Demand for gasoline: G = f(price,income)
Demand (price) elasticity: eP = %change in G given %change in P holding income constant.
How do you do that in the real world?
The “percentage changes”
How to change price and hold income constant?

76. The Real World Data

77. U.S. Gasoline Market,

78. Shouldn’t Demand Curves Slope Downward?

79. A Thought Experiment
The main driver of gasoline consumption is income not price
Income is growing over time.
We are not holding income constant when we change price!
How do we do that?

80. How to Hold Income Constant?
Multiple Regression Using Price and Income
Regression Analysis: G versus GasPrice, Income
The regression equation is
G = GasPrice Income
Predictor Coef SE Coef T P
Constant
GasPrice
Income
It looks like the theory works.

81. Statistics and Data Analysis
Linear Multiple Regression Model

82. Classical Linear Regression Model
The model is y = f(x1,x2,…,xK,1,2,…K) + 
= a multiple regression model. Important examples:
Marginal cost in a multiple output setting
Separate age and education effects in an earnings equation.
Denote (x1,x2,…,xK) as x. Boldface symbol = vector.
Form of the model – E[y|x] = a linear function of x.
‘Dependent’ and ‘independent’ variables.
Independent of what? Think in terms of autonomous variation.
Can y just ‘change?’ What ‘causes’ the change?

83. Model Assumptions: Generalities
Linearity means linear in the parameters. We’ll return to this issue shortly.
Identifiability. It is not possible in the context of the model for two different sets of parameters to produce the same value of E[y|x] for all x vectors. (It is possible for some x.)
Conditional expected value of the deviation of an observation from the conditional mean function is zero
Form of the variance of the random variable around the conditional mean is specified
Nature of the process by which x is observed.
Assumptions about the specific probability distribution.

84. Linearity of the Model
f(x1,x2,…,xK,1,2,…K) = x11 + x22 + … + xKK
Notation: x11 + x22 + … + xKK = x.
Boldface letter indicates a column vector. “x” denotes a variable, a function of a variable, or a function of a set of variables.
There are K “variables” on the right hand side of the conditional mean “function.”
The first “variable” is usually a constant term. (Wisdom: Models should have a constant term unless the theory says they should not.)
E[y|x] = 1*1 + 2*x2 + … + K*xK.
(1*1 = the intercept term).

85. Linearity
Linearity means linear in the parameters, not in the variables
E[y|x] = 1 f1(…) + 2 f2(…) + … + K fK(…).
fk() may be any function of data.
Examples:
Logs and levels in economics
Time trends, and time trends in loglinear models – rates of growth
Dummy variables
Quadratics, power functions, log-quadratic, trig functions, interactions and so on.

86. Linearity Simple linear model, E[y|x] =x’β
Quadratic model: E[y|x] = α + β1x + β2x2
Loglinear model, E[lny|x] = α + Σk lnxkβk
Semilog, E[y|x] = α + Σk lnxkβk
All are “linear.” An infinite number of variations.

87. Matrix Notation

88. Notation
Define column vectors of N observations on y and the K x variables.
The assumption means that the rank of the matrix X is K.
No linear dependencies => FULL COLUMN RANK of the matrix X.

89. An Unidentified (But Valid) Theory of Art Appreciation
Enhanced Monet Area Effect Model: Height and Width Effects
Log(Price) = β1 + β2 log Area +
β3 log Aspect Ratio +
β4 log Height +
β5 Signature + ε
(Aspect Ratio = Width/Height)
This X does not have full column rank. At least one column is a linear function of the others.

90. Conditional Homoscedasticity and Nonautocorrelation
Disturbances provide no information about each other.
Var[i | X ] = 2
Cov[i, j |X] = 0

91. Heteroscedasticity
Countries are ordered by the standard deviation of their 19 residuals.
Regression of log of per capita gasoline use on log of per capita income, gasoline price and number of cars per capita for 18 OECD countries for 19 years. The standard deviation varies by country.

92. Autocorrelation logG=β1 + β2logPg + β3logY + β4logPnc + β5logPuc + ε

93. Autocorrelation Results from an Incomplete Model

94. Normal Distribution of ε
Used to facilitate finite sample derivations of certain test statistics.
Observations are independent
Assumption will be unnecessary – we will use the central limit theorem for the statistical results we need.

95. The Linear Model
y = X+ε, N observations, K columns in X, (usually including a column of ones for the intercept).
Standard assumptions about X
Standard assumptions about ε|X
E[ε|X]=0, E[ε]=0 and Cov[ε,x]=0
Regression: E[y|X] = X

96. Statistics and Data Analysis
Least Squares

97. Vocabulary Some terms to be used in the discussion.
Population characteristics and entities vs. sample quantities and analogs
Residuals and disturbances
Population regression line and sample regression
Objective: Learn about the conditional mean function. Estimate  and 2
First step: Mechanics of fitting a line to a set of data

98. Least Squares

99. Matrix Results

101. Moment Matrices

102. Least Squares Normal Equations

103. -----------------------------------------------------------------------------
Ordinary least squares regression
LHS=G Mean =
Standard deviation =
No. of observations = DegFreedom Mean square
Regression Sum of Squares =
Residual Sum of Squares =
Total Sum of Squares =
Standard error of e = Root MSE
Fit R-squared = R-bar squared
Model test F[ 2, 33] = Prob F > F*
| Standard Prob % Confidence
G| Coefficient Error t |t|>T* Interval
Constant|
PG| ***
Y| ***

104. Second Order Conditions

105. Does b Minimize e’e?

106. Positive Definite Matrix