Byron Gangnes Econ 427 lecture 6 slides Selecting forecasting models— alternative criteria

Byron Gangnes Forecast Model Selection What are we trying to do? Find the model with the best likely forecast performance A practical approach: –Find the model with the smallest out-of- sample 1-step-ahead mean squared prediction error.

Byron Gangnes EViews output Of course the problem is that often we only have data from in-sample.

Byron Gangnes Mean Squared error One approach would be to pick the model that minimizes in-sample mean squared error This is the same as minimizing the sum of square resids, right?

Byron Gangnes R-squared Also the same as maximizing R 2 (R-squared) : What is the problem with these approaches?

Byron Gangnes Critique of MSE as a Model Selection Tool In-sample overfitting and data mining Technically, MSE is a (downward) biased estimator of out-of-sample 1-step-ahead prediction error variance. –The bias increases as you add variables. How to “fix” this problem? –Penalize for degrees of freedom used up in estimation (number of included variables)

Byron Gangnes Some alternatives that do this Adjusted R-squared

Byron Gangnes Some alternatives that do this Akaike Information Criteria: Schwatz Information Criteria:

Byron Gangnes How do penalize degrees of fr?

Byron Gangnes Evaluating Model Selection Criteria Consistency –When the true model is one of the ones evaluated, the probability of selecting that one approaches 1 as sample size becomes large. –When the true model is NOT one of the ones evaluated, the probability of selecting the best approximation among candidate models approaches 1 as sample size becomes large. Are any of these consistent?

Byron Gangnes Evaluating Model Selection Criteria Asymptotic efficiency –Chooses a sequence of models as the sample size becomes large whose 1-step-ahead forecast error variances approach the true one at least as fast as any other selection criterion. Do any of our candidate criteria meet that? What to do in practice?