1 Malaquias Peña and Huug van den Dool Consolidation of Multi Method Forecasts Application to monthly predictions of Pacific SST NCEP Climate Meeting, April 4, 2007 Acknowledgments: Suru Saha retrieved and organized the data, Dave Unger and Peitao Peng provided discussion to the subject

2 DATA Forecasting tools: 8 CGCMs, 1 Statistical model –NCEP CFS: , 15 membs, 9 leads –DEMETER : , 9 membs, 6 leads ECMWF MPI MF UKMO INGV LODYC CERFAX –CPC’ Constructed Analog (CA) : , 12 membs,12 leads This is what all have in common: Monthly Forecasts, leads 0 to 5 Initial months: Feb, May, Aug, Nov Length of retrospective forecasts: 21 years, FOCUS: TROPICAL PACIFIC SST: 12.5 S TO 12.5 N

3 Consolidation: Making the best single forecast out of a number of forecast inputs. Objective consolidation necessary as large supply of forecasts are available. If K is the number of participant forecast systems, ζ, predicting a particular target month with a given lead time, the consolidation is the following linear combination: DEFINITIONS For convenience, systematic errors and observed climatology are removed in ζ. The regression coefficients (weights), α, are based on past performance of the forecast system. o is the verifying field (e.g. observation; climatology removed). Suppose there are N cases of retrospective forecasts, then one can train a consolidation method by comparing:

4 OPTIMIZING WEIGHTS Find weights, α i,for each forecasting tool, ζ i, that minimizes the (sum of square of) errors ε j in Where Z is a matrix whose columns are the forecasting tools and rows are the data points in the training period, o is the column vector containing the verifying field, and ε is a vector of errors. Least square method (unconstrained regression):

5 eigenvaluesNino 3.4PNANAO Corresponding weights for UR for lead 1, im 1 ILL-POSED MATRIX PROBLEM too large

6 RIDGE REGRESSION Constrained to: Minimize: leads to Ridge Regression (DelSole, 2007) (ad hoc) where and Van den Dool estimates such that the weights are small and stable Many more ways to find it Depends on characteristics of covariance matrix Z T Z

7 RIDGE REGRESSION Model weights ( α i, i=1..9 ) as a function of λ for three ridge consolidation methods. Figure illustrates asymptotic values. Our methods stop at λ=0.5. Unconstrained regression ( λ=0 ) results in a wide range (including negative values) of weights. RIDRIM RIW λλλ

8 CONSOLIDATION METHODS ASSESED

9 CROSS-VALIDATION Anomaly Pattern correlation over the tropical Pacific. Average for all leads and initial months. Empty bar: Full (dependent), filled bar: 3-yr out cross-validated.

10 GRIDPOINT BY GRIDPOINT PERFORMANCE

11 EQUATORIAL PACIFIC

12 WESTERN TROPICAL PACIFIC Trust in good models when performed well in a gridpoint. It goes to the opposite direction of the bad models

13 WESTERN TROPICAL PACIFIC MIXES CLOSEST NEIGHBORING GRIDPOINT Trust in good models when performed well in a 3x3 box of gridpoints. It goes to the opposite direction of the bad models

14 WESTERN TROPICAL PACIFIC Trust less good models, damps towards climatology as negative weights are set to zero DOUBLE PASS AND MIXES CLOSEST NEIGHBORING GRIDPOINT

15 INCREASING EFFECTIVE SAMPLE 1 GRIDPOINT BY GRIPOINT 23X3 BOXES 35X5 BOXES 4ALL GRIDPOINTS IN THE DOMAIN 5GRIDPOINTS IN AND OUT DOMAIN Multi- methods average AC Skill of most consolidation methods improve when effective sampling size increases Tropical Pacific SST. AC average for all leads and initial months

16 INCREASING EFFECTIVE SAMPLE Consistency: Percentage cases (leads and initial months) outperforming MM

17 RELATIVE OPERATIONING CURVES Assess the ability to anticipate correctly the occurrence or non occurrence that SST anomalies will fall in the upper, middle and lower terciles. Class limits defined by the observed SST during the training period Probability information from the ensemble: counting the fraction of ensemble members that falls into the “above-normal”, “near-normal”, and “below-normal” categories, and interpreting this fraction as the probability that forecasts will fall in such categories. Approach for the optimized weights: each ensemble member forecast is multiplied by normalized weights. Lead 3 Upper tercile Lower tercile

18 UPPER TERCILE

19 LOWER TERCILE

20 SUMMARY All the points below are for the particular case of SST anomalies in the tropical Pacific. Forecasts arising from a combination of multiple models of similar skill generally outperform those from individual models but not UR after CV-3. Even the simple average of multi-methods shows consistent improvement over individual participant models. Over all and after cross-validation, sophisticated consolidation methods marginally improve over the simple average. Increasing the effective sampling size increases the skill and consistency of consolidation methods. Consolidation methods improve significantly over the multi- methods average in the western Pacific. Probabilistic assessment, as measured by ROC shows some improvement of consolidation methods over MM. Construction of the probability density function of the consolidation requires optimization.