Polynomial Regression and Transformations STA 671 Summer 2008

Review The estimated residuals e 1,…,e n provide the best method for checking the assumptions. Remember the residuals ε i ~ N(0,σ). The estimated residuals should be close to that. In a residual plot, you are looking for outliers, curvature, or changing variance. In this lecture we will discuss polynomial regression and transformations, two separate methods. Both are possible solutions to curvature, and transformations have the added benefit they sometimes address changing variance.

Recall the Hooker data. There appears to be a small amount of curvature.

This curvature is seen more clearly in the residual plot.

Polynomial regression – one method for dealing with curvature. To account for curvature, we can perform something called “polynomial regression”, which consists of fitting a polynomial (a quadratic or cubic typically) instead of a line. Recall the linear model was Y i = β 0 + β 1 X i + ε i. The quadratic model is Y i = β 0 + β 1 X i + β 2 X i 2 + ε i. The cubic model is Y i = β 0 + β 1 X i + β 2 X i 2 + β 3 X i 3 + ε i. The higher the order of the polynomial, the more curvature it can account for.

Quadratic model accounts for the curvature Quadratic equation is – Temp Temp 2 If the quadratic model is better than the linear model, what about a cubic?

A cubic model produces no visual improvement equation is Pressure = – 1.69 Temp Temp Temp 3

Which to choose, quadratic or cubic? In general, choose the LOWEST order polynomial possible (i.e. prefer linear to quadratic, quadratic to cubic, etc.). This is aimed at 1) “Occam’s razor” meaning that simpler models are preferred, and 2) the higher the order, the more parameters to estimate. Statistically, it’s easier to estimate a few parameters than many.

P-values for selecting order The regression output provides a formal method for selecting the order of the polynomial. This method typically agrees with looking at the residual plot. The regression output provides p-values for each term in the regression. The p-value for the highest order term is the ONLY one that is used.

Using p-values to select order Begin by fitting the cubic model. If the cubic term is significant, use the cubic model (you can consider higher order models, but we do not in STA671) If the cubic term is NOT significant, remove it and RERUN the model (p-values change depending on what terms are in the model), then look to see if the quadratic term is significant. If the quadratic term is not significant, remove it and RERUN the model, resulting in a linear regression. If none of these models produce a reasonable residual plot, you may need another method.

For the boiling point data We first run the cubic model and acquire the following p-values The p-value is not significant, so remove the cubic term and RERUN the model (do NOT just remove the quadratic terms based on the p-value above) Parameter Estimates Parameter Standard Variable Label DF Estimate Error t Value Pr > |t| Type I SS Intercept Intercept Temperature Temperature_2 2nd power of TEMPERATURE Temperature_3 3rd power of TEMPERATURE

Quadratic model for boiling point data The quadratic model produces the following p-values The quadratic term is significant AND we observe a reasonable residual plot, so we stop here. This is our final model. Parameter Estimates Parameter Standard Variable Label DF Estimate Error t Value Pr > |t| Type I SS Intercept Intercept < Temperature < Temperature_2 2nd power of TEMPERATURE <

What if no polynomial model produces a reasonable residual plot? If none of our polynomial model produces a reasonable residual plot, we need another method. Another method to try is to transform the response variable. Transformations, like polynomial regression, can handle curvature, and in addition transformations have the potential to handle changing spread as well.

Example - the ethanol data Data comes from an engine exhaust study. NOx is a measure of the exhaust from the engine, while E is a measure of the fuel/air mixture (high values are almost all fuel, low values are almost all air) A cubic model does not fit the data. A quadratic of linear model would do worse.

A cubic fit to the ethanol data Scatterplot Residual plot shows clear curvature.

Transformations Instead of fitting Y as the response variable, we fit a function of Y as the response variable. Thus, instead of Y i = β 0 + β 1 X i + ε i, you can fit log(Y i ) = β 0 + β 1 X i + ε i, or sqrt(Y i ) = β 0 + β 1 X i + ε i, or cbrt(Y i ) = β 0 + β 1 X i + ε i, etc. Thus, you greatly expand the possible models you can fit. You can transform the X variable as well, but in the interest of time we do not discuss that in detail in STA671.

Transformations allow different errors structures. A quadratic regression looks like Y i = β 0 + β 1 X i + β 2 X i 2 + ε i. At any particular X, the variance is the same. Taking the square root transformation sqrt(Y i ) = β 0 + β 1 X i + ε i means that Y i = [β 0 + β 1 X i + ε i ] 2 = β β 1 2 X i 2 + ε i β 0 β 1 X i + 2 β 0 ε i + 2 β 1 X i ε i. There is a quadratic relationship between X and Y. Note the multiplication between X i and ε i, this allows the variance to change for each X i. Thus, in addition to handling curvature, transformations allow you to address changing variance.

Prototypical Data requiring transformation

After square root transformation

Which transformation? There are no hard and fast rules on which transformation to try, no guaranteed method for finding a good transformation (in some data, you seem to never find a great fit). Usually you have to perform trial and error, and remember you can combine polynomial regression with transformation. Thus for example, you can fit a cubic model in X to predict log(Y).

Some “typical” transformations If you have area data, a square root transform is often useful (converts area to something proportional to the radius or length). Similarly with volume, a cube root transformation may be appropriate. With financial data (incomes, etc.), a log transform may be appropriate. Logs change percentage increases to constant increases, thus if a unit increase in X results in a 10% increase in Y, it also results in a increase in Y.

A general strategy Fit the raw data (X and Y) with a least squares line. See if you get a good residual plot. If so, stop and be happy If not, try a polynomial regression (quadratic or cubic). If one of these fits, stop and be happy (remember, fit the smallest model possible). If a polynomial regression does not work, try transforming Y to log, sqrt, and cube root (i.e. perform three more regressions). Fit a cubic polynomial regression on each of these and determine the best outcome. Choose the transformation that provides the best residual plot. If none of those work, then regression might not be effective (there are more advanced techniques) or you may have to start transforming X as well. This becomes true trial and error. Consult your friendly local statistician.

Back to the ethanol data. We can see from the scatterplot that E and NOX are not linearly related. We tried a cubic regression and that didn’t work. Now off to the transformations. We fit cubic regressions with log(Y), sqrt(Y), and cbrt(Y) as the response variables. We may be able to get satisfactory results with something less than cubic, but if cubic doesn’t work the lower order models won’t either, thus we start with cubic models.

Square root transformation. Still clear curvature. ScatterplotResidual plot

Cube root transformation. Improved, but still some curvature.

Log transformation. Still some lack of fit, but best of the bunch.

Log transform is not perfect, but best we can do right now (I encourage you to play with the data on your own) After we have chosen the log transformation on the basis of the best residual plot (and decided it is “ok”, if certainly not a great residual plot), we look at the p-value for the cubic term to see if we can remove it. We can. Parameter Estimates Parameter Standard Variable Label DF Estimate Error t Value Pr > |t| Type I SS Intercept Intercept < E E_2 2nd power of E E_3 3rd power of E

Quadratic model for log(NOX) The quadratic model produces almost identical scatter and residual plots. The quadratic term is significant, so this is our final model. Parameter Estimates Parameter Standard Variable Label DF Estimate Error t Value Pr > |t| Type I SS Intercept Intercept < E < E_2 2nd power of E <

Extras There are more advanced ways of dealing with polynomial regression and transformations, which we do not address in STA671. Polynomial regression can be extended to handle more general curved models, such as splines (piecewise polynomials with desirable smoothness properties) Transformation can be selected automatically by using something called a Box-Cox transformation, which automatically determines the appropriate exponent to transform your data (with a tradeoff of some interpretability). Consult your friendly local statistician.