1. Multi-Model Ensembling for Seasonal-to-Interannual Prediction: From Simple to Complex Lisa Goddard and Simon Mason International Research Institute for Climate & Society The Earth Institute of Columbia University

2. Benefit of Combining Models Benefit of Using Multiple Models RPSS for 2m Temperature (JFM 1950-1995) Combining models reduces deficiencies of individual models

3. Refining: (1) RAW MODEL PROBABILITIES (simple) Tercile thresholds determined by model history -- Counting (2) RECALIBRATED PDF PROBABILITIES (less simple) - Contingency table recalibration (CT): categorical probabilities determined by category of ensemble mean - Uncertainty in forecast PDFs based on ensemble mean MSE Varying Complexity in Building MM

4. Combining: (1) POOLED MM ENSEMBLE (simple) Each model weighted equally (2) PERFORMANCE-BASED MM ENSEMBLE (less simple) - Bayesian: determine optimal weights for AGCMs & climatology by maximizing likelihood - Multiple linear regression (MLR): obtain probabilities from prediction error variance using first few moments of ensemble distributions - Canonical Variate (CV): maximize discrimination between categories using first few moments of ensemble distributions Varying Complexity in Building MM

5. Bayesian Model Combination Climo Fcst “Prior” GCM Fcst “Evidence” t=1234 … Combine “prior” and “evidence” to produce weighted “posterior” forecast probabilities, by maximizing the likelihood. … … Combining: Based on model performance

6. The canonical variates are defined to maximize the ratio of the between-category to within-category variance. Canonical Variate Analysis Combining: Based on obs. relative to model

7. NINO3.4

8. DEMETER Data

9. Equal Weighting Probabilistic forecasts were obtained by counting the number of ensemble members beyond the outer quartiles, and then averaging across the three models. The pooled ensemble is thus an equally-weighted combination of predictions uncorrected for model skill (although corrected for model drift). Reliability is good for all three categories.

10. Canonical Variate Analysis

11. Multiple Linear Regression

12. Conclusions I

13. Terrestrial Climate

14. Data AGCMs: Simulations for 1950-2000 * CCM3 (NCAR) – 24 runs * ECHAM4.5 (MPI) – 24 runs * ECPC (Scripps) – 10 runs * GFDL AM2p12b – 10 runs * NCEP-MRF9 (NCEP/QDNR) – 10 runs * NSIPP1 (NASA-GSFC) – 9 runs Observations: 2m Air Temperature and Precipitation from CRU-UEA (v2.0)

15. Effect of Probability Treatment JFM 2m air temperature over land

16. Effect of Probability Treatment

17. Effect of Combination Method

18. Effect of Combination Method RAW Probabilities

19. Effect of Combination Method PDF Probabilities

20. Conclusions II Reliability of N models pooled together, with uncalibrated PDFs, is better than any individual AGCM. Gaussian (PDF) recalibration gives some improvement, but Bayesian recalibration gives the greatest benefit. Reliability is typically gained at the expense of resolution.

21. ISSUES Number of Models Length of training period When simple is complex enough?

22. Effect of # of Models 3 vs 6 AGCMS; 45-year training period (Robertson et al., 2004, MWR) Different approaches are more similar with more models.

23. RPSS for 2m Temperature Bayesian MM from Raw Probs. – 6 models, 45-yr training Jan-Feb-MarJul-Aug-Sep

24. RPSS for Precipitation Bayesian & Pooled MM from Raw Probs. – 6 models, 45-yr training Jan-Feb-MarJul-Aug-Sep

25. Reliability Diagrams * several methods yield similar results over the United States. * MMs are remarkably reliable over the US, even though the accuracy is not high.

26. CONCLUSIONS III MM simulations over the US are remarkably reliable, even if their not terribly accurate. Simple pooling of the AGCMs, with uncalibrated probabilities, is equivalent to any of our techniques over the U.S. Doesn’t require long history, but large number of models (>5?) is desirable.

27. GRAND CONCLUSIONS Overall, we find that recalibrating individual models gives better results than putting models together in complex combination alorithm. In comparing different recalibration/combination methods, we find that generally a gain in reliability is countered with a loss in resolution. More complicated approaches are not necessarily better. This needs to investigated for different forecast situations (i.e. variables, region, season).

29. Ranked Probability Skill Scores Temperature Jan-Feb-Mar (1950-1995)

30. Ranked Probability Skill Scores Precipitation Jul-Aug-Sep (1950-1999) Comparing treatment of probability - Even with 6 models, have regions of large negative RPSS  Suggests common model errors - Recalibration reduces, but does not eliminate, large errors - Some improvement of positive skill Recal-Raw

31. Ranked Probability Skill Scores Precipitation Jul-Aug-Sep (1950-1999) Comparing combination methods - Performance-based combination eliminates large errors - More improvement of positive skill - More cases of negative skill turned to positive skill

32. Canonical Variate Analysis A number of statistical techniques involve calculating linear combinations (weighted sums) of variables. The weights are defined to achieve specific objectives: PCA – weighted sums maximize variance CCA – weighted sums maximize correlation CVA – weighted sums maximize discrimination

33. Canonical Variate Analysis

34. Effect of Probability Treatment