MAPE Publication Neil McAndrews For Bob Ryan of Deutsche Bank

What makes a forecast a valid forecast? Unbiased – overshoots as many time as it undershoots Validation Metric - Percent Error Accurate – predicts the actual load Validation Metric – Mean Absolute Percentage Error Reliable – often predicts the load accurately Validation Metric – histogram of error 1. 2.

Update of forecast performance Sept 1 to Dec 3. ERCOT Peak Forecast and Actual (from PUC data) 1. MAPE 4.01% 2. Average Bias is 1010 MWs.

November 30 th PUC data Peak Load Forecast: 40,627 MW Actual Peak Load: 47,000 MW Forecast Error: 6,573 MW Percent Forecast Error: minus 13.6%

ERCOTs Monthly MAPE for 16:00 MTLF Raises questions regarding which model is being reported. Is it the AREVA model alone, or a combination of models, and why it doesn’t include this summer’s bias?

ERCOT MAPES Each model should have its percent error and MAPE measured Actual operational forecasts (the combination of model outcomes) should be also be measured because the select process can increase error and introduce a bias into the process, e.g. selecting the larger of two forecasts. For example: instead of using the current two forecasts methods introduce a third forecast that has a peak of 65,000 MWs. This third forecast method would always be selected. But this would be wrong because it would also introduce bias and have a large error. Forecasts should be used based upon highest reliability, lowest MAPE and lack of bias.

Summary of Current Forecast The forecast is overestimating peak load for the months of September through December 3 by an average of 1,010 MWs (2.12%). Actual error should be much closer to zero. (Overestimation does appear to be declining later in the period) In November the bias appears to be negligible. Suggest a review of this data for accuracy.

Summary MAPE is a measurement tool that improves fundamental market information for market participants. Market participants and the ISO can continually iterate on their forecasts to improve forecast accuracy (i.e., minimize forecast error). This is common practice in other regions, e.g., PJM. Publishing load and weather MAPEs will help to lower forecast error by the natural collaboration between ERCOT and market participants. ERCOT has not isolated the weather forecast error from the overall model. This PRR addition would isolate the causes of errors, and lead to improved forecast performance for each of the modeling method as well as a combination of forecast outcomes.