1. Violations of Regression Assumptions
Copyright (c) 2008 by The McGraw-Hill Companies. This spreadsheet is intended solely for educational purposes by licensed users of LearningStats. It may not be copied or resold for profit.

2. Minitab Copyright Notice
Copyright Notice Portions of MINITAB Statistical Software input and output contained in this document are printed with permission of Minitab, Inc. MINITABTM is a trademark of Minitab Inc. in the United States and other countries and is used herein with the owner's permission.

3. Regression Assumptions
yi = b0 + b1x1i + b2x2i + … + bpxpi + ei
Correct model specified (no variables omitted)
Appropriate model form (e.g., linear)
Predictors are non-stochastic and independent
Errors (disturbances) are random
zero mean
normally distributed
homoscedastic (constant variance)
mutually independent (non-autocorrelated)

4. Errors are normally distributed Errors have constant variance s2
Standard Notation
ei ~ N(0,s2)
Errors are normally distributed
Errors have constant variance s2
Errors have zero mean

5. Violations of Assumptions
yi = b0 + b1x1i + b2x2i + … + bpxpi + ei
Relevant predictors were omitted
Wrong model form specified (e.g., linear)
Collinear predictors (i.e., correlated Xj and Xk)
Non-normal errors (e.g., skewed, outliers)
Heteroscedastic errors (non-constant variance)
Autocorrelated errors (non-independent)

6. What Is Specification Bias?
Wrong model form or the wrong variables
Example of Wrong Model Form:
You said Y = a + bX, but actually Y = a + bX + cX2
Example of Wrong Variables:
You said Y = a + bX but actually Y = a + bX +cZ

7. Specification Bias
a linear model was specified, but the data actually are non-linear

8. Detecting Specification Bias
In a bivariate model:
Plot Y against X
Plot residuals against estimated Y
In a multivariate model:
Plot residuals against actual Y
Plot fitted Y against actual Y
Look for patterns (there should be none)

9. Residuals Plotted on Y
residuals are correlated with Y (suggests incorrect specification)

10. What Is Multicollinearity?
The "independent" variables are related
Collinearity:
Correlation between any two predictors
Multicollinearity:
Relationship among several predictors

11. Effects of Multicollinearity
Estimates may be unstable
Standard errors may be misleading
Confidence intervals generally too wide
High R2 yet t statistics insignificant

12. Variance Inflation Factor
VIFs give a simple multicollinearity test. Each predictor has a VIF. For predictor j, the VIF is
where Rj2 is the coefficient of determination when predictor j is regressed against all the other predictors.

13. Variance Inflation Factor
Example A: If Rj2 =.00 then VIFj = 1:
MegaStat and MINITAB will calculate the VIF for each predictor if you request it
Example B: If Rj2 = .90 then VIFj = 10:

14. Evidence of Multicollinearity
Any VIF > 10
Sum of VIFs > 10
High correlation for pairs of predictors Xj and Xk
Unstable estimates
(i.e., the remaining coefficients change sharply when a suspect predictor is dropped from the model)

15. Example: Estimating Body Fat
Problem: Several VIFs exceed 10.

16. Correlation Matrix of Predictors
Age and Height are relatively independent of other predictors.
Problem: Neck, Chest, Abdomen, and Thigh are highly correlated.

17. Solution: Eliminate Some Predictors
R2 is reduced slightly, but all VIFs are now below 10.

18. Stability Check for Coefficients
There are large changes in estimated coefficients as high VIF predictors are eliminated, revealing that the original estimates were unstable. But the “fit” deteriorates when we eliminate predictors.

19. Example: College Graduation Rates
Minor problem?
The sum of the VIFs exceeds 10 (but few statisticians would worry since no single VIF is very large).

20. Remedies for Multicollinearity
Drop one or more predictors
But this may create specification error
Transform some variables (e.g., log X)
Enlarge the sample size (if you can)
Tip
If they feel the model is correctly specified, statisticians tend to ignore multicollinearity unless its influence on the estimates is severe.

21. What Is Heteroscedasticity?
Non-constant error variance
Homoscedastic:
Errors have the same variance for all values of the predictors (or Y)
Heteroscedastic
Error variance changes with the values of the predictors (or Y)

22. How to Detect Heteroscedasticity
Excel and MegaStat and MINITAB will do these residual plots if you request them
Plot residuals against each predictor (a bit tedious)
Plot residuals against estimated Y (quick check)
There are more general tests, but they are complex

23. Homoscedastic Residuals

24. Heteroscedastic Residuals
To detect heteroscedasticity, we plot the residuals against each predictor. Some predictors may show a problem, while others are O.K. A quick overall test is to plot the residuals only against estimated Y.

25. Effects of Heteroscedasticity
Happily ...
OLS coefficients bj are still unbiased
OLS coefficients bj are still consistent
But ...
Std errors of b's are biased (bias may be + or -)
t values and CI for b's may be unreliable
May indicate incorrect model specification

26. Remedies for Heteroscedasticity
Avoid totals (e.g., use per capita data)
Transform some variables (e.g., log X)
Don't worry about it (may not be serious)

27. What Is Autocorrelation?
The errors are not independent
Independent errors:
et does not depend on et-1 (r = 0)
Autocorrelated errors:
et depends on et (r 0)
Good News Autocorrelation is a worry in time-series models (the subscript t = 1, 2, ..., n denotes time) but generally not in cross-sectional data.

28. What Is Autocorrelation?
Assumed Model: et = r et-1 + ut
where ut is assumed non-autocorrelated
Independent errors:
et does not depend on et-1 (r = 0)
Autocorrelated errors:
et depends on et (r 0)
 The residuals will show a pattern over time

29. Autocorrelated Residuals
Common
Rare
When a residual tends to be followed by another of the same sign, we have positive autocorrelation
When a residual tends to be followed by another of opposite sign, we have negative autocorrelation

30. How to Detect Autocorrelation
Look for pattern in residuals plotted against time
Look for cycles of of followed by
Look for alternating pattern
Calculate the correlation between et and et-1
This is called the “autocorrelation coefficient”
It should not differ significantly from)
Check Durbin-Watson statistic
DW = 2 indicates absence of autocorrelation
DW < 2 indicates positive autocorrelation (common)
DW > 2 indicates negative autocorrelation (rare)

31. residuals are autocorrelated
Residual Time Plot
residuals are autocorrelated
problem: runs of
and

32. Durbin-Watson Test

33. Common Types of Autocorrelation
Errors are autocorrelated (relatively minor)
Lagged Y used as predictor (OK if large n)
First Order Autocorrelation
Yt = b0 + b1Xt1 + b2Xt2 + et where
et = ret-1 + ut and ut is N(0,s2)
Lagged Predictor
Yt = b0 + b1Xt-1 + b2Yt-1 + et

34. Effects of Simple First-Order Autocorrelation
OLS coefficients bj are still unbiased
OLS coefficients bj are still consistent
If r > 0 (the typical situation) then the
standard errors of bj is underestimated
computed t values will be too high
C.I. for bj will be too narrow

35. General Effects of Autocorrelation

36. Data Transformations for Autocorrelation
Use first differences: DY = f(DX1, DX2)
Use Cochrane-Orcutt transformation
DYt = g0 + b1DX1 + b2DX2 + et
Comment Simple, but only suffices when r is near 1.
Yt* = Yt - rYt-1
Xt* = Xt - rXt-1
Comment We must estimate the sample autocorrelation coeffficient and use it to estimate r.

37. The errors are normally distributed
What Is Non-Normality?
The errors are normally distributed
Normal errors:
The histogram of residuals is "bell-shaped"
There are no outliers in the residuals
The probability plot is linear
Non-normal errors
Any violations of the above

38. Residual Histogram histogram should be symmetric and bell-shaped
there are outliers beyond 3 s

39. Residual Probability Plot
If normal, dots should be linear (45o line)
possible outlier

40. Effects of Non-Normal Errors
Confidence intervals for Y may be incorrect
May indicate outliers
May indicate incorrect model specification
But usually not considered a serious problem

41. Detection of Non-Normal Errors
Look at histogram of residuals
Should be symmetric
Should be bell-shaped
Look for outliers or asymmetry
Outliers are a serious violation
Mild asymmetry is common
Look at probability plot of residuals
Should be linear
Look for outliers

42. Remedies for Non-Normal Errors
Avoid totals (e.g., use per capita data)
Transform some variables (e.g., log X)
Enlarge the sample (asymptotic normality)

43. Influential Observations
High "leverage" of certain data points
These are data points with extreme X values
Sometimes called “high leverage” observations
One case may strongly affect the estimates

44. How to Detect Influential Observations
In MINITAB, look for observations denoted "X"
(These are observations with unusual X values)
In MINITAB, look for observations denoted "R"
(These are observations with unusual residuals)
Do your own tests
(MINITAB does them automatically)

45. Rules for Finding Influential Observations
Unusual X: look at hat matrix for hii > 2p/n where
p is the number of coefficients in the model
n is the number of observations
Unusual Y: look for large studentized deleted residuals
Use z values as a reference if n is large
Use t values for d.f. = n - p - 1 if n is small
Unusual X and Y: use Cook’s distance measure
Use F(p,n-p) as critical value
Unusual X and Y: use MINITAB’s Dfits measure
Rule of thumb is Dfits > 2{p/n}.5

46. Remedies for Influential Observations
Discard the observation only if you have logical reasons for thinking the observation is flawed
Use method of least absolute deviations
(but Minitab and Excel don’t calculate absolute deviations)
Call a professional statistician

47. Assessing Fit The fit of a model can be assessed by the R2 and R2adj,
There are many ways
The fit of a model can be assessed by
the R2 and R2adj,
the F statistic in ANOVA table
the standard error syIx
plot of fitted Y against actual Y

48. Overall Fit: Actual Y versus Fitted Y
The correlation between actual Y and fitted Y is the multiple correlation coefficient
The closer to a 45o line, the better the fit

49. Summing It Up Computers do most of the work
Regression is somewhat robust
Be careful but don't panic
Excelsior!