1. Chapter 9 Multicollinearity
Linear Regression Analysis 5E Montgomery, Peck & Vining

2. Linear Regression Analysis 5E Montgomery, Peck & Vining
9.1 Introduction
Multicollinearity is a problem that plagues many regression models. It impacts the estimates of the individual regression coefficients.
Uses of regression:
1. Identifying the relative effects of the regressor variables
2.  Prediction and/or estimation, and
3.  Selection of an appropriate set of variables for the model.
Linear Regression Analysis 5E Montgomery, Peck & Vining

3. Linear Regression Analysis 5E Montgomery, Peck & Vining
9.1 Introduction
If all regressors are orthogonal, then multicollinearity is not a problem. This is a rare situation in regression analysis.
More often than not, there are near-linear dependencies among the regressors such that
is approximately true. If this sum holds exactly for a subset of regressors, then (X’X)-1 does not exist.
Linear Regression Analysis 5E Montgomery, Peck & Vining

4. 9.2 Sources of Multicollinearity
Four primary sources
The data collection method employed
Constraints on the model or in the population
Model specification
An overdefined model
Linear Regression Analysis 5E Montgomery, Peck & Vining

5. 9.2 Sources of Multicollinearity
Data collection method employed
Occurs when only a subsample of the entire sample space has been selected. (Soft drink delivery: number of cases and distance tend to be correlated. That is, we may have data where only a small number of cases are paired with short distances, large number of cases paired with longer distances. ) We may be able to reduce this multicollinearity through the sampling technique used. There is no physical reason why you can’t sample in that area.
Linear Regression Analysis 5E Montgomery, Peck & Vining

6. 9.2 Sources of Multicollinearity
Linear Regression Analysis 5E Montgomery, Peck & Vining

7. 9.2 Sources of Multicollinearity
Constraints on the model or in the population.
(Electricity consumption: two variables x1 – family income and x2 – house size). Physical constraints are present, multicollinearity will exist regardless of collection method.
Linear Regression Analysis 5E Montgomery, Peck & Vining

8. 9.2 Sources of Multicollinearity
Model Specification
Polynomial terms can cause ill-conditioning in the X’X matrix. This is especially true if range on a regressor variable, x, is small.
Linear Regression Analysis 5E Montgomery, Peck & Vining

9. 9.2 Sources of Multicollinearity
Overdefined model
More regressor variables than observations. The best way to counter this is to remove regressor variables. Recommendations: 1) Redefine the model using smaller set of regressors; 2) do preliminary studies using subsets of regressors; or 3) use principal components type regressor methods to remove regressors.
Linear Regression Analysis 5E Montgomery, Peck & Vining

10. 9.3 Effects of Multicollinearity
Strong multicollinearity can result in large variances and covariances for the least squares estimates of the coefficients. Recall from chapter 3, C = (X’X)-1 and
Strong multicollinearity between xj and any other regressor variable will cause Rj2 to be large, and thus Cjj to be large.
In other words, the variance of the least squares estimate of the coefficient will be very large.
Linear Regression Analysis 5E Montgomery, Peck & Vining

11. 9.3 Effects of Multicollinearity
Strong multicollinearity can also produce least-squares estimates of the coefficients that are too large in absolute value. The squared distance between the least squares estimate and the true parameter is denoted
Linear Regression Analysis 5E Montgomery, Peck & Vining

12. 9.3 Effects of Multicollinearity
Tr(X’X)-1 is the trace of a matrix – which is the sum of the main diagonal elements.
With multicollinearity present, some of the eigenvalues of X’X will be small.
Tr(matrix) = sum of the eigenvalues of the matrix.
Let j > 0 be the jth eigenvalue of X’X. Then
Linear Regression Analysis 5E Montgomery, Peck & Vining

13. Linear Regression Analysis 5E Montgomery, Peck & Vining

14. Linear Regression Analysis 5E Montgomery, Peck & Vining

15. 9.4 Multicollinearity Diagnostics
Ideal characteristics of a multicollinearity diagnostic:
We want the procedure to correctly indicate if multicollinearity is present; and,
We want the procedure to provide some insight as to which regressors are causing the problem.
Linear Regression Analysis 5E Montgomery, Peck & Vining

16. 9.4.1 Examination of the Correlation Matrix
If we scale and center the regressors in the X’X matrix, we have the correlation matrix. The pairwise correlation between two variables xi and xj is denoted rij. The off diagonal elements of the centered and scaled X’X matrix (X’X matrix in correlation form) are the pairwise correlations.
If |rij| is close to unity, then there may be an indication of multicollinearity. But, the opposite does not always hold. That is, there may be instances when multicollinearity is present, but the pairwise correlations do not indicate a problem. This can happen when using pairwise correlations in a problem with more than two variables involved.
Linear Regression Analysis 5E Montgomery, Peck & Vining

17. Linear Regression Analysis 5E Montgomery, Peck & Vining

18. Linear Regression Analysis 5E Montgomery, Peck & Vining
The correlation matrix fails to identify the multicollinearity problem in the Mason, Gunst & Webster data in Table 9.4, page 296.
Linear Regression Analysis 5E Montgomery, Peck & Vining

19. 9.4.2 Variance Inflation Factors
As discussed in Chapter 3, variance inflation factors are very useful in determining if multicollinearity is present.
VIFs > 5 to 10 are considered significant. The regressors that have high VIFs probably have poorly estimated regression coefficients
Linear Regression Analysis 5E Montgomery, Peck & Vining

20. Linear Regression Analysis 5E Montgomery, Peck & Vining

21. 9.4.2 Variance Inflation Factors
VIFs: A Second Look and Interpretation
The length of the normal-theory confidence interval on the jth regression coefficient can be written as
Linear Regression Analysis 5E Montgomery, Peck & Vining

22. 9.4.2 Variance Inflation Factors
VIFs: A Second Look and Interpretation
The length of the corresponding normal-theory confidence interval based on a design with orthogonal regressors (with same sample size, same root-mean square (rms) values) is
Linear Regression Analysis 5E Montgomery, Peck & Vining

23. 9.4.2 Variance Inflation Factors
VIFs: A Second Look and Interpretation
Take the ratio of these two: Lj/L* = That is, the square root of the jth VIF gives us a measure of how much longer the confidence interval for the jth regression coefficient is because of multicollinearity.
For example, say VIF3 = 10. Then This tells us that that the confidence interval is 3.3 times longer than if the regressors had been orthogonal (the best case scenario).
Linear Regression Analysis 5E Montgomery, Peck & Vining

24. 9.4.3 Eigensystem Analysis of X’X
The eigenvalues of X’X (denoted 1, 2, …, p) can be used to measure multicollinearity. Small eigenvalues are indications of multicollinearity.
  The condition number of X’X is
This number measures the spread in the eigenvalues.
 < 100, no serious problem
100 <  < 1000, moderate to strong multicollinearity
 > 1000, strong multicollinearity.
Linear Regression Analysis 5E Montgomery, Peck & Vining

25. 9.4.3 Eigensystem Analysis of X’X
A large condition number indicates multicollinearity exists. It does not tell us how many regressors are involved.
  The condition indices of X’X are
The number of condition indices that are large (greater than 1000) provide a measure of the number of near linear dependencies in X’X.
  In SAS, PROC REG, in the model statement of your program, you can use the option COLLIN; this will produce out eigenvalues, condition indices, etc.
Linear Regression Analysis 5E Montgomery, Peck & Vining

26. Linear Regression Analysis 5E Montgomery, Peck & Vining

27. Linear Regression Analysis 5E Montgomery, Peck & Vining

28. Linear Regression Analysis 5E Montgomery, Peck & Vining

29. Linear Regression Analysis 5E Montgomery, Peck & Vining

30. Linear Regression Analysis 5E Montgomery, Peck & Vining

31. Linear Regression Analysis 5E Montgomery, Peck & Vining

32. 9.5 Methods for Dealing with Multicollinearity
Collect more data
Respecify the model
Ridge Regression
Linear Regression Analysis 5E Montgomery, Peck & Vining

33. 9.5 Methods for Dealing with Multicollinearity
Least squares estimation gives an unbiased estimate,
with minimum variance – but this variance may still be very large, resulting in unstable estimates of the coefficients.
 Alternative: Find an estimate that is biased but with smaller variance than the unbiased estimator
Linear Regression Analysis 5E Montgomery, Peck & Vining

34. 9.5 Methods for Dealing with Multicollinearity
Ridge Estimator
k is a “biasing parameter” usually between 0 and 1.
Linear Regression Analysis 5E Montgomery, Peck & Vining

35. 9.5 Methods for Dealing with Multicollinearity
The effect of k on the MSE
Recall:
Now,
As k , Var , and bias 
Choose k such that the reduction in variance > increase in bias.
Linear Regression Analysis 5E Montgomery, Peck & Vining

36. 9.5 Methods for Dealing with Multicollinearity
Ridge Trace
Plots k against the coefficient estimates. If multicollinearity is severe, the ridge trace will show it. Choose k such that is stable and hope the MSE is acceptable
Ridge regression is a good alternative if the model user wants to have all regressors in the model.
Linear Regression Analysis 5E Montgomery, Peck & Vining

37. 9.5 Methods for Dealing with Multicollinearity
Linear Regression Analysis 5E Montgomery, Peck & Vining

38. Linear Regression Analysis 5E Montgomery, Peck & Vining

39. More About Ridge Regression
Methods for choosing k
Relationship to other estimators
Ridge regression and variable selection
Generalized ridge regression (a procedure with a biasing parameter k for each regressor
Linear Regression Analysis 5E Montgomery, Peck & Vining

40. Linear Regression Analysis 5E Montgomery, Peck & Vining
9.5.4 Principal-Component Regression
Linear Regression Analysis 5E Montgomery, Peck & Vining

41. Linear Regression Analysis 5E Montgomery, Peck & Vining

42. Linear Regression Analysis 5E Montgomery, Peck & Vining

43. Linear Regression Analysis 5E Montgomery, Peck & Vining
The eigenvalues suggest that a model based on 4 or 5 of the PCs would probably be adequate
Linear Regression Analysis 5E Montgomery, Peck & Vining

44. Linear Regression Analysis 5E Montgomery, Peck & Vining

45. Models D and E are pretty similar
Linear Regression Analysis 5E Montgomery, Peck & Vining