Chapter 9 Multicollinearity Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining 9.1 Introduction Multicollinearity is a problem that plagues many regression models. It impacts the estimates of the individual regression coefficients. Uses of regression: 1. Identifying the relative effects of the regressor variables 2.  Prediction and/or estimation, and 3.  Selection of an appropriate set of variables for the model. Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining 9.1 Introduction If all regressors are orthogonal, then multicollinearity is not a problem. This is a rare situation in regression analysis. More often than not, there are near-linear dependencies among the regressors such that is approximately true. If this sum holds exactly for a subset of regressors, then (X’X)-1 does not exist. Linear Regression Analysis 5E Montgomery, Peck & Vining

9.2 Sources of Multicollinearity Four primary sources The data collection method employed Constraints on the model or in the population Model specification An overdefined model Linear Regression Analysis 5E Montgomery, Peck & Vining

9.2 Sources of Multicollinearity Data collection method employed Occurs when only a subsample of the entire sample space has been selected. (Soft drink delivery: number of cases and distance tend to be correlated. That is, we may have data where only a small number of cases are paired with short distances, large number of cases paired with longer distances. ) We may be able to reduce this multicollinearity through the sampling technique used. There is no physical reason why you can’t sample in that area. Linear Regression Analysis 5E Montgomery, Peck & Vining

9.2 Sources of Multicollinearity Linear Regression Analysis 5E Montgomery, Peck & Vining

9.2 Sources of Multicollinearity Constraints on the model or in the population. (Electricity consumption: two variables x1 – family income and x2 – house size). Physical constraints are present, multicollinearity will exist regardless of collection method. Linear Regression Analysis 5E Montgomery, Peck & Vining

9.2 Sources of Multicollinearity Model Specification Polynomial terms can cause ill-conditioning in the X’X matrix. This is especially true if range on a regressor variable, x, is small. Linear Regression Analysis 5E Montgomery, Peck & Vining

9.2 Sources of Multicollinearity Overdefined model More regressor variables than observations. The best way to counter this is to remove regressor variables. Recommendations: 1) Redefine the model using smaller set of regressors; 2) do preliminary studies using subsets of regressors; or 3) use principal components type regressor methods to remove regressors. Linear Regression Analysis 5E Montgomery, Peck & Vining

9.3 Effects of Multicollinearity Strong multicollinearity can result in large variances and covariances for the least squares estimates of the coefficients. Recall from chapter 3, C = (X’X)-1 and   Strong multicollinearity between xj and any other regressor variable will cause Rj2 to be large, and thus Cjj to be large. In other words, the variance of the least squares estimate of the coefficient will be very large. Linear Regression Analysis 5E Montgomery, Peck & Vining

9.3 Effects of Multicollinearity Strong multicollinearity can also produce least-squares estimates of the coefficients that are too large in absolute value. The squared distance between the least squares estimate and the true parameter is denoted   Linear Regression Analysis 5E Montgomery, Peck & Vining

9.3 Effects of Multicollinearity Tr(X’X)-1 is the trace of a matrix – which is the sum of the main diagonal elements. With multicollinearity present, some of the eigenvalues of X’X will be small. Tr(matrix) = sum of the eigenvalues of the matrix. Let j > 0 be the jth eigenvalue of X’X. Then   Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining

9.4 Multicollinearity Diagnostics Ideal characteristics of a multicollinearity diagnostic: We want the procedure to correctly indicate if multicollinearity is present; and, We want the procedure to provide some insight as to which regressors are causing the problem. Linear Regression Analysis 5E Montgomery, Peck & Vining

9.4.1 Examination of the Correlation Matrix If we scale and center the regressors in the X’X matrix, we have the correlation matrix. The pairwise correlation between two variables xi and xj is denoted rij. The off diagonal elements of the centered and scaled X’X matrix (X’X matrix in correlation form) are the pairwise correlations. If |rij| is close to unity, then there may be an indication of multicollinearity. But, the opposite does not always hold. That is, there may be instances when multicollinearity is present, but the pairwise correlations do not indicate a problem. This can happen when using pairwise correlations in a problem with more than two variables involved. Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining The correlation matrix fails to identify the multicollinearity problem in the Mason, Gunst & Webster data in Table 9.4, page 296. Linear Regression Analysis 5E Montgomery, Peck & Vining

9.4.2 Variance Inflation Factors As discussed in Chapter 3, variance inflation factors are very useful in determining if multicollinearity is present. VIFs > 5 to 10 are considered significant. The regressors that have high VIFs probably have poorly estimated regression coefficients Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining

9.4.2 Variance Inflation Factors VIFs: A Second Look and Interpretation The length of the normal-theory confidence interval on the jth regression coefficient can be written as Linear Regression Analysis 5E Montgomery, Peck & Vining

9.4.2 Variance Inflation Factors VIFs: A Second Look and Interpretation The length of the corresponding normal-theory confidence interval based on a design with orthogonal regressors (with same sample size, same root-mean square (rms) values) is Linear Regression Analysis 5E Montgomery, Peck & Vining

9.4.2 Variance Inflation Factors VIFs: A Second Look and Interpretation Take the ratio of these two: Lj/L* = . That is, the square root of the jth VIF gives us a measure of how much longer the confidence interval for the jth regression coefficient is because of multicollinearity. For example, say VIF3 = 10. Then . This tells us that that the confidence interval is 3.3 times longer than if the regressors had been orthogonal (the best case scenario). Linear Regression Analysis 5E Montgomery, Peck & Vining

9.4.3 Eigensystem Analysis of X’X The eigenvalues of X’X (denoted 1, 2, …, p) can be used to measure multicollinearity. Small eigenvalues are indications of multicollinearity.   The condition number of X’X is This number measures the spread in the eigenvalues.  < 100, no serious problem 100 <  < 1000, moderate to strong multicollinearity  > 1000, strong multicollinearity. Linear Regression Analysis 5E Montgomery, Peck & Vining

9.4.3 Eigensystem Analysis of X’X A large condition number indicates multicollinearity exists. It does not tell us how many regressors are involved.   The condition indices of X’X are The number of condition indices that are large (greater than 1000) provide a measure of the number of near linear dependencies in X’X.   In SAS, PROC REG, in the model statement of your program, you can use the option COLLIN; this will produce out eigenvalues, condition indices, etc. Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining

9.5 Methods for Dealing with Multicollinearity Collect more data Respecify the model Ridge Regression Linear Regression Analysis 5E Montgomery, Peck & Vining

9.5 Methods for Dealing with Multicollinearity Least squares estimation gives an unbiased estimate, with minimum variance – but this variance may still be very large, resulting in unstable estimates of the coefficients.  Alternative: Find an estimate that is biased but with smaller variance than the unbiased estimator Linear Regression Analysis 5E Montgomery, Peck & Vining

9.5 Methods for Dealing with Multicollinearity Ridge Estimator k is a “biasing parameter” usually between 0 and 1. Linear Regression Analysis 5E Montgomery, Peck & Vining

9.5 Methods for Dealing with Multicollinearity The effect of k on the MSE Recall: Now, As k , Var , and bias  Choose k such that the reduction in variance > increase in bias. Linear Regression Analysis 5E Montgomery, Peck & Vining

9.5 Methods for Dealing with Multicollinearity Ridge Trace Plots k against the coefficient estimates. If multicollinearity is severe, the ridge trace will show it. Choose k such that is stable and hope the MSE is acceptable Ridge regression is a good alternative if the model user wants to have all regressors in the model. Linear Regression Analysis 5E Montgomery, Peck & Vining

9.5 Methods for Dealing with Multicollinearity Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining

More About Ridge Regression Methods for choosing k Relationship to other estimators Ridge regression and variable selection Generalized ridge regression (a procedure with a biasing parameter k for each regressor Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining 9.5.4 Principal-Component Regression Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining The eigenvalues suggest that a model based on 4 or 5 of the PCs would probably be adequate Linear Regression Analysis 5E Montgomery, Peck & Vining

Linear Regression Analysis 5E Montgomery, Peck & Vining

Models D and E are pretty similar Linear Regression Analysis 5E Montgomery, Peck & Vining