Part I – MULTIVARIATE ANALYSIS C3 Multiple Linear Regression II © Angel A. Juan & Carles Serrat - UPC 2007/2008

1.3.1: The General Linear Model  Model building is the process of developing an estimated regression equation that describes the relationship between a dependent variable and one or more predictors.  Two major issues in model building are: a)To find the proper functional form (linear, quadratic, etc.) of the relationship b)To select the predictors to be included in the model  Suppose we have data for one response y and k predictors x 1, x 2, …, x k. The General Linear Model (GLM) involving p predictors is:  Note: The regression techniques are not limited to linear relationships. The word “linear” in the term GLM refers only to the fact that β 0, β 1,…, β p all have exponents of one; it does not imply that the relationship between y and the x i ’s is linear. Multicollinearity in a regression model results from the common mistake of putting too many predictors into the model. Such a model is said to be “over-fit”. Inevitably many of the predictors will have effects that are too correlated and cannot be separately estimated

1.3.2: Modeling Curvilinear Relationships  Curvilinear relationships in RA can be handled by using powers of predictors.  File: SALESPEOPLE.MTW Although the output shows that a linear relationship explains a high percentage of the variability in sales, the standardized residual plot suggest that a curvilinear relationship is needed. This standardized residual plot shows that the previous curvilinear pattern has been removed. We want to investigate the relationship between months of employment of the salespeople and the number of units sold.

1.3.3: Interaction between Predictors  When interaction between two predictors is present, we cannot study the effect of one predictor on the response y independently of the other predictor.  File: SALESPRICE.MTW We want to investigate the relationship between price, advertising expenditure and the number of units sold of a product. Note that, at higher selling prices, the effect of increased advertising expenditure diminishes. This fact suggest existence of interaction between the price and advertising predictors. We will use the predictor price*advertising ( P*A ) to account for the effect of interaction. Since the p-value corresponding to the t test for P*A is 0.000, we conclude that interaction is significant, i.e.: the effect of advertising expenditure on sales depends on the price

1.3.4: Transformations of the Response  Often the problem of non-constant variance can be corrected by transforming the response variable to a different scale using a logarithmic transformation, Log(y), or a reciprocal transformation,1/y.  File: KMWEIGHT.MTW We want to investigate the relationship between Km per liter and the vehicle weight. This standardized residual plot shows that the variability in the residuals appears to increase as the fitted value increases. Furthermore, there is a large standardized residual. The wedge-shaped pattern has now disappeared. Moreover, there is not any large standardized residual.

1.3.5: “Linearizing” Nonlinear Models  Models in which the β parameters have exponents other than one are called nonlinear models.  In some cases we can easily transform the nonlinear model into a linear one, as in the case of the exponential model:  Many nonlinear models cannot be transformed into an equivalent linear model. However, such models have had limited use in practical applications.

1.3.6: Model Building: Initial Steps  File: MODELBUILDING.MTW The Correlation Matrix provides the sample correlation coefficients between each pair of variables and the p-value for the corresponding test on significant correlation. The best predictors for Sales seem to be: Accts (R 2 =57%), Time, Poten and Adv. There is multicollinearity between Time and Accts ; hence if Accts were used as a predictor, Time would not add much more explanatory power to the model. A similar problem occurs between Change and Rating. We want to investigate the relationship between Sales and the eight predictors.

1.3.7: Model Building: Predictors Selection  Objective: to select those predictors that provide the “best” regression model.  Alternative methods for selecting the predictors: 1.Stepwise (forward and backward) regression: At each step, a single variable can be added or deleted. The process continues until no more improvement can be obtained by adding or deleting a variable. 2.Forward selection: At each step, a single variable can be added, but variables are never deleted. The process continues until no more improvement can be obtained by adding a variable. 3.Backward elimination: The procedure starts with a model involving all the possible predictors. At each step a variable is deleted. The procedure stops when no more improvement can be obtained by deleting a variable. 4.Best-subsets regression: This is not a one-variable- at-a-time method. It evaluates regression models involving different subsets of the predictors. Iterative methods The stopping criterion is based on the F statistic They provide a “good” regression model with little multicollinearity problems, but not necessarily the “best” model (the one with the highest R 2 ) Iterative methods The stopping criterion is based on the F statistic They provide a “good” regression model with little multicollinearity problems, but not necessarily the “best” model (the one with the highest R 2 ) It provides the “best” regression model for the given data. Functions of the predictors (e.g.: z = x 1 * x 2 ) can be used to create new predictors for use with any of the methods presented here.

1.3.8: Model Building: Stepwise Regression  File: MODELBUILDING.MTW  Stat > Regression > Stepwise... In this example, Stepwise Regression takes 5 steps. At the end of the procedure, five predictors ( Accts, Adv, Poten, Share and Change ) have been selected for the regression model. Note that R-Sq(adj) = 88,94% for the last model. In Stepwise (Forward and Backward) Regression, a significance level of α=0.15 is recommended both for add or delete new variables to/from the model. At each step: 1. The already-in-the-model predictor with the highest non-significant p- value (p-value > α) will be deleted from the model 2. The not-already-in-the-model predictor with the lowest significant p-value (p-value <= α) will enter the model 3. P-values are recalculated At each step: 1. The already-in-the-model predictor with the highest non-significant p- value (p-value > α) will be deleted from the model 2. The not-already-in-the-model predictor with the lowest significant p-value (p-value <= α) will enter the model 3. P-values are recalculated

1.3.9: Model Building: Forward Selection  File: MODELBUILDING.MTW  Stat > Regression > Stepwise... In this example, Forward Selection takes 6 steps. At the end of the procedure, six predictors ( Accts, Adv, Poten, Share, Change and Time ) have been selected for the regression model. Note that R-Sq(adj) = 89,38% for the last model. In Forward Selection, a significance level of α=0.25 is recommended for add new variables to the model. Mallows’ C-p statistic represents a subsetting criterion to be used in selecting a reduced model without multicollinearity problems. A rule of thumb is to select a model in which the value of Cp is close to the number of terms, including any constant term, in the model. At each step: 1. The not-already-in-the-model predictor with the lowest significant p-value (p-value <= α) will enter the model 2. P-values are recalculated At each step: 1. The not-already-in-the-model predictor with the lowest significant p-value (p-value <= α) will enter the model 2. P-values are recalculated

1.3.10: Model Building: Backward Eliminat.  File: MODELBUILDING.MTW  Stat > Regression > Stepwise... In this example, Backward Elimination takes 4 steps. At the end of the procedure, five predictors ( Time, Poten, Adv, Share, and Change ) have been selected for the regression model. Note that R-Sq(adj) = 89,27% for the last model. In Backward Elimination, a significance level of α=0.1 is recommended for delete variables from the model. At each step: 1. The already-in-the-model predictor with the highest non- significant p-value (p-value > α) will be deleted from the model 2. P-values are recalculated At each step: 1. The already-in-the-model predictor with the highest non- significant p-value (p-value > α) will be deleted from the model 2. P-values are recalculated

1.3.11: Model Building: Best-Subsets Reg.  File: MODELBUILDING.MTW  Stat > Regression > Best Subsets... In this example, the adjusted coefficient of determination is largest for the model with six predictors ( Time, Poten, Adv, Share, Change and Accts ). However, the best model with four independent variables ( Poten, Adv, Share and Accts ) has an adjusted coefficient of determination almost as high. The Best-Subsets output identifies the two best one-predictor models, the two best two-predictor models, and so on. The criterion used which models are best for any number of predictors is the value of R 2. All other things being equal, a simpler model with fewer variables is usually preferred.