1. Model selection
Stepwise regression

2. Statement of problem
A common problem is that there is a large set of candidate predictor variables.
Goal is to choose a small subset from the larger set so that the resulting regression model is simple, yet have good predictive ability.

3. Stepwise regression: the idea
Start with no predictors in the “stepwise model.”
At each step, enter or remove a predictor based on the t-tests.
Stop when no more predictors can be justifiably entered or removed from the stepwise model.

4. Stepwise regression: Preliminary steps
Specify an Alpha-to-Enter (αE = 0.15) significance level.
Specify an Alpha-to-Remove (αR = 0.15) significance level.

5. Stepwise regression: Step #1
Fit each of the one-predictor models, that is, regress y on x1, regress y on x2, … regress y on xp-1.
The first predictor put in the stepwise model is the predictor that has the smallest P-value (below αE = 0.15).
If no P-value < 0.15, stop.

6. Stepwise regression: Step #2
Suppose x1 was the “best” one predictor.
Fit each of the two-predictor models with x1 in the model, that is, regress y on (x1, x2), regress y on (x1, x3), …, and y on (x1, xp-1).
The second predictor put in stepwise model is the predictor that has the smallest P-value (below αE = 0.15).
If no P-value < 0.15, stop.

7. Stepwise regression: Step #2 (continued)
Suppose x2 was the “best” second predictor.
Step back and check P-value for β1 = 0. If the P-value for β1 = 0 has become not significant (above αR = 0.15), remove x1 from the stepwise model.

8. Stepwise regression: Step #3
Suppose both x1 and x2 made it into the two-predictor stepwise model.
Fit each of the three-predictor models with x1 and x2 in the model, that is, regress y on (x1, x2, x3), regress y on (x1, x2, x4), …, and regress y on (x1, x2, xp-1).

9. Stepwise regression: Step #3 (continued)
The third predictor put in stepwise model is the predictor that has the smallest P-value (below αE = 0.15).
If no P-value < 0.15, stop.
Step back and check P-values for β1 = 0 and β2 = 0. If either P-value has become not significant (above αR = 0.15), remove the predictor from the stepwise model.

10. Stepwise regression: Stopping the procedure
The procedure is stopped when adding an additional predictor does not yield a P-value below αE = 0.15.

11. Drawbacks of stepwise regression
The final model is not guaranteed to be optimal in any specified sense.
The procedure yields a single final model, although in practice there are often several equally good models.
It doesn’t take into account a researcher’s knowledge about the predictors.