FORECAST PERFORMANCE AND DATA MINING *Forecast is the ultimate challenge for any econometric model. *Any model that fails to predict is useless. Exceptions? *History may not repeat itself, but it rhythms.

Remarks on econometric forecast: (1)Forecast and decision- cost of forecast (forecast is irrelevant per se) Forecast without properly defined cost function means nothing. (2)Forecast standard deviation please… (3)Statistical measures for prediction accuracy can be problematic in the real world.

(4)Out-of-sample monitoring Is yesterday’s model capable of explaining today’s data? (5)Correcting data mining bias in a “historically adequate” model. Predictions based on historically adequate model resulting from extensive specification search are subject to data mining bias.

Data Mining A general modeling process that applies a sequence of statistical and non-statistical tools to a data set in an attempt to obtain a final model. References: Ye, J (1998) “On Measuring and Correcting the Effects of Data Mining and Model Selection.” JASA 93 p White, H (2000): “A Reality Check for Data Snooping.” Econometrica

Simulation Results Cost of Data Mining Generalized degree of freedom (GDF) is defined as the sum of sensitivity of each fitted value to perturbations in the corresponding value.

Reality Check Whenever a good forecasting model is obtained by an extensive specification search, there is always the danger that the observed good performance results not from actual forecasting ability, but instead of luck. Stock Investment News letter Simulation using Stationary bootstraps