1. Caio A. S. Coelho Department of Meteorology University of Reading c.a.d.s.coelho@reading.ac.uk Met Office, Exeter (U.K.), 20 February 2006 PLAN OF TALK Calibration and combination issues Conceptual framework for forecasting Forecast Assimilation: Example 1: Nino-3.4 index forecasts Example 2: Equatorial Pacific SST forecasts Example 3: S. American rainfall forecasts Example 4: Regional rainfall downscaling EUROBRISA project Forecast calibration and combination: Bayesian assimilation of seasonal climate predictions Thanks to: David B. Stephenson, Magdalena Balmaseda, Francisco J. Doblas-Reyes and Sergio Pezzulli

2. This talk is based on the following work: Coelho C.A.S. 2005: “Forecast Calibration and Combination: Bayesian Assimilation of Seasonal Climate Predictions”. PhD Thesis. University of Reading. 178 pp. Coelho C.A.S., D. B. Stephenson, M. Balmaseda, F. J. Doblas-Reyes and G. J. van Oldenborgh, 2005: Towards an integrated seasonal forecasting system for South America. ECMWF Technical Memorandum No. 461, 26pp. Also in press in the J. Climate. Coelho C.A.S., D. B. Stephenson, F. J. Doblas-Reyes, M. Balmaseda, A. Guetter and G. J. van Oldenborgh, 2006: A Bayesian approach for multi-model downscaling: Seasonal forecasting of regional rainfall and river flows in South America. Meteorological Applications, 13, 1-10. Stephenson, D. B., Coelho, C. A. S., Doblas-Reyes, F.J. and Balmaseda, M., 2005: “Forecast Assimilation: A Unified Framework for the Combination of Multi-Model Weather and Climate Predictions.” Tellus A, Vol. 57, 253-264. Coelho C.A.S., S. Pezzulli, M. Balmaseda, F. J. Doblas-Reyes and D. B. Stephenson, 2004: “Forecast Calibration and Combination: A Simple Bayesian Approach for ENSO”. Journal of Climate. Vol. 17, No. 7, 1504-1516. Coelho C.A.S., S. Pezzulli, M. Balmaseda, F. J. Doblas-Reyes and D. B. Stephenson, 2003: “Skill of Coupled Model Seasonal Forecasts: A Bayesian Assessment of ECMWF ENSO Forecasts”. ECMWF Technical Memorandum No. 426, 16pp. Available from: http://www.met.rdg.ac.uk/~swr01cac

3. Calibration and combination issues Why do forecasts need it? Which are the best ways to calibrate? How to get good probability estimates? Who should do it? Calibration Combination Why combine forecasts? Should model predictions be weighted or selected? How best to combine? Who should do it?

4. Conceptual framework Data Assimilation “Forecast Assimilation”

5. Multi-model ensemble approach DEMETER Development of a European Multi-Model Ensemble System for Seasonal to Interannual Prediction Solution: Multi-model Ensemble Errors: Model formulation Initial conditions http://www.ecmwf.int/research/demeter

6. DEMETER Multi-model ensemble system 7 coupled global circulation models Hindcast period: 1980-2001 (1959-2001) 9 member ensembles ERA-40 initial conditions SST and wind perturbations 4 start dates per year (Feb, May, Aug and Nov) 6 month hindcasts ModelCountry ECMWFInternational LODYCFrance CNRMFrance CERFACSFrance INGVItaly MPIGermany UKMOU.K.....

7. Examples of application 0-d: Niño-3.4 index 1-d: Equatorial Pacific SST 2-d: South American rainfall

8. Example 1: Empirical Niño-3.4 forecasts Well-calibrated: Most observations in the 95% prediction interval (P.I.) 95% P.I.

9. ECMWF coupled model ensemble forecasts  Observations not within the 95% prediction interval!  Coupled model forecasts need calibration m=9 DEMETER: 5-month lead

10. Prior: Univariate X and Y Posterior: Likelihood: Bayes’ theorem:

11. Likelihood modelling: y

12. Combined forecasts  Note: most observations within the 95% prediction interval!

13. Comparison of the forecasts Empirical Coupled Combined SUMMARY Combined forecasts: are better calibrated than coupled have less spread than empirical match obs better than either Blue dots = observations Red dots = mean forecast Grey shade = 95% prediction interval

14. Mean Absolute Error (MAE) defined as: The Brier score (BS) is a simple quadratic score for probability forecasts of binary events (e.g. whether SST anomaly < 0). It is defined as: Some verification statistics ForecastMAE (  C) Brier score Spread (  C) Climatology1.160.251.19 Empirical0.530.050.61 Coupled0.570.180.33 Combined0.310.040.32  Combined forecasts have smallest MAE, BS, and spread

15. Prior: Likelihood: Posterior: Multivariate X and Y: More than one Normal variable Matrices

16. Example 2: Equatorial Pacific SST ForecastBrier Score Climatol p=0.50.25 Multi-model0.19 FA 58-010.17 SST anomalies: Y (°C) Forecast probabilities: p DEMETER: 7 coupled models; 6-month lead

17. Forecast assimilation reduces (i.e. improves) the Brier score in the eastern and western equatorial Pacific Brier Score as a function of longitude Brier Score=0.25 for p=0.5 climatology Brier Score<0.25  more skilful than climatology

18. Brier Score decomposition reliability resolution uncertainty

19. Forecast assimilation improves reliability in the western Pacific Reliability as a function of longitude

20. Resolution as a function of longitude Forecast assimilation improves resolution in the eastern Pacific

21. Why South America? El Niño (DJF) La Niña (DJF) Source: Climate Prediction Center (http://www.cpc.ncep.noaa.gov)  Seasonal climate potentially predictable DEMETER Multi-model Correlation of ensemble mean DJF rainfall forecasts with PREC/L observations

22. Why South American rainfall?  Agriculture  Electricity: More than 90% produced by hydropower stations e.g. Itaipu (Brazil/Paraguay): World largest hydropower plant I nstalled power: 12600 MW 18 generation units (700 MW each) ~25% electricity consumed in Brazil ~95% electricity consumed in Paraguay

23. Itaipu

24. Example 3: S. American rainfall anomaly composites Obs Multi-model Forecast Assimilation (mm/day) DEMETER: 3 coupled models (ECMWF, CNRM, UKMO) 1-month lead Start: Nov DJF ENSO composites: 1959-2001 16 El Nino years 13 La Nina years ACC=0.51 ACC=0.28 ACC=0.97 ACC=0.82 ACC=1.00 ACC=Anomaly Correlation Coefficient Spatial correlation of map with obs map

25. DJF rainfall anomalies for 1975/76 and 1982/83 ObsMulti-model Forecast Assimilation (mm/day) ACC=-0.09 ACC=0.32 ACC=0.59 ACC=0.56 La Nina 1975/76 El Nino 1982/83

26. DJF rainfall anomalies for 1991/92 and 1998/99 ObsMulti-model Forecast Assimilation (mm/day) ACC=0.04 ACC=0.08 ACC=0.32 ACC=0.38

27. Brier Skill Score for S. American rainfall Forecast assimilation improves the Brier Skill Score (BSS) in the tropics

28. Reliability component of the BSS Forecast assimilation improves reliability over many regions

29. Resolution component of the BSS Forecast assimilation improves resolution in the tropics

30. Empirical model for South American rainfall Matrices Z: ASO SST Y: DJF rainfall

31. Empirical Multi-model Integrated Correlation maps: DJF rainfall anomalies  Comparable level of determinist skill  Better skill in tropical and southeastern South America

32. Mean Anomaly Correlation Coefficient Most skill in ENSO years and forecast assimilation can improve skill Multi-model Integrated Empirical

33. ENS Forecast assimilation improved Brier Skill Score (BSS) in the tropics Brier Skill Score for S. American rainfall Empirical Multi-model Integrated

34. Forecast assimilation improved reliability in many regions Reliability component of the BSS Empirical Multi-model Integrated

35. Forecast assimilation improved resolution in the tropics Resolution component of the BSS Empirical Multi-model Integrated

36. Example 4: regional rainfall downscaling Multi-model ensemble 3 DEMETER coupled models ECMWF, CNRM, UKMO 3-month lead Start: Aug NDJ Period: 1959-2001

37. ForecastCorrelationBrier Score Multi-model0.570.22 FA0.740.17 South box: NDJ rainfall anomaly Multi-model Forecast assimilation  Forecast assimilation improves skill substantially - - - Observation Forecast

38. CorrelationBrier Score Multi-model0.620.21 FA0.630.18 - - - Observation  Forecast assimilation improved skill marginally North box: NDJ rainfall anomaly Multi-model Forecast assimilation

39. Forecasts can be improved both by calibration and by combination Statistical calibration and combination is analogous to data assimilation and is a fundamental and essential part of the forecasting process (forecast assimilation) Forecast assimilation is easy to do for normally distributed predictands such as monthly mean temperatures and seasonal rainfall: Nino-3 probability forecasts improved – less biased and smaller spread Equatorial SST forecasts improved in eastern and western Pacific S. American rainfall forecasts improved in Equatorial and Southern regions Combination can improve the resolution of the forecasts (the ability to discriminate between different observed situations) whereas calibration can improve the reliability of the forecasts First steps towards an integrated seasonal forecasting system for South America including both empirical and coupled model predictions EUROBRISA project will implement this system at CPTEC - Brazil Summary

40. The EUROBRISA Project Lead Investigator: Caio A.S. Coelho Key Idea: To improve seasonal forecasts in S. America: a region where there is seasonal forecast skill and useful value. Aims Strengthen collaboration and promote exchange of expertise and information between European and S. American seasonal forecasters Produce improved well-calibrated real-time probabilistic seasonal forecasts for South America Develop real-time forecast products for non-profitable governmental use (e.g. reservoir management, hydropower production, and agriculture) EUROBRISA was approved by ECMWF council in June 2005 http://www.met.rdg.ac.uk/~swr01cac/EUROBRISA InstitutionsCountryPartners CPTECBrazilCoelho, Cavalcanti, Silva Dias, Pezzi ECMWFEUAnderson, Balmaseda, Doblas-Reyes, Stockdale INMETBrazilMoura, Silveira Met OfficeUKGraham, Davey, Colman Météo FranceFranceDéqué SIMEPARBrazilGuetter Uni. of ReadingUKStephenson Uni. of Sao PauloBrazilAmbrizzi, Silva Dias CIIFENEcuadorCamacho, Santos

41. Reliability diagram (Multi-model) (p i ) (o i ) o

42. Direct and inverse regression y Regression of obs on forecastsRegression of forecasts on obs More natural to model uncertainty in forecasts for a given observation (ensemble spread of dots) than to model uncertainty in observations for a given ensemble forecast.  so we model the likelihood on right rather than the more common forecast calibration (MOS) approach on the left.

43. Reliability diagram (FA 58-01) o (p i ) (o i )

44. Moment measure of skewness Measure of asymmetry of the distribution