1. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Model uncertainty in climate forecasting from seasons to decades: the ENSEMBLES project Francisco J Doblas-Reyes ECMWF, Reading, UK

2. First EUROBRISA Workshop17 March 2008The ENSEMBLES project ●Description of the ENSEMBLES project. ●Hindcast production in ENSEMBLES: Streams 1 and 2. ●Diagnosis of the ENSEMBLES simulations and forecast quality assessment. ●Applications: the actual value of forecast systems. ●Data quality control and dissemination. ●Inclusion of NCEP CFS in a European multi-model. Outline

3. First EUROBRISA Workshop17 March 2008The ENSEMBLES project ●Integrated Project funded by the EC within the VIth FP, Sep 2004-Aug 2009, 69 partners. ●Main goal: development of an integrated probabilistic prediction system for time scales from seasons to decades, and beyond. Therefore, research on both seasonal-to- decadal (s2d) and anthropogenic climate change (ACC). ●S2d to be used to assess the reliability of forecast systems used for scenario runs, both from a climate modelling and user perspective, in a seamless framework. ●Forecast uncertainty: comparison of the merits of multi- model, perturbed parameters and stochastic physics approaches to deal with model error. ●Diversity of applications: health, crop yield, energy… The ENSEMBLES project

4. First EUROBRISA Workshop17 March 2008The ENSEMBLES project ●Model error is a major source of forecast uncertainty. Three approaches to deal with model error are being investigated in ENSEMBLES: multi-model, stochastic physics and perturbed parameters. ●The multi-model is built from ECMWF, Met Office, Météo- France operational activities and the DEMETER experience at IfM-Kiel, CERFACS and INGV. ●Perturbed parameter system stems from the decadal prediction system (DePreSys) created at the Met Office. ●Stochastic physics system uses the ECMWF stochastic backscatter system developed for medium-range forecasts. ●A major s2d experiment to determine the benefits of each approach is ongoing (~20,000 years of integrations). Dealing with forecast uncertainty

5. First EUROBRISA Workshop17 March 2008The ENSEMBLES project ●Three systems: multi-model (ECMWF, GloSea, DePreSys, Météo-France, IfM-Kiel, CERFACS, INGV), stochastic physics (ECMWF) and perturbed parameters (DePreSys). ●Hindcasts in two streams: o Stream 1: hindcast period 1991-2001, seasonal (7 months, May and November start date), annual (14 months, November start date) and 2 decadal (1965 and 1994), 9 member ensembles, ERA40 initialization in most cases. o Stream 2: As in Stream 1 but over 1960-2005, with 4 start dates for seasonal hindcasts, at least 1 for annual and at least one 3- member decadal hindcast every 5 years. oAdditional simulations: DePreSys_PP carries out 14-month hindcasts, a 10-year hindcast every year and a 30-year hindcast every 5 years + lots of sensitivity experiments from the other contributors. S2d ENSEMBLES global experiment

6. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Sea surface temperature RMSE (solid) and spread (dashed) averaged over the Niño3.4 region for the 1 st November (bottom row) start dates over the period 1991-2001. RMSE of simple persistence in dashed black. All forecast systems beat persistence. Multi-model is the most skilful system, with highest reliability (RMSE~spread), in the first 6 months, while the perturbed parameter system is as good for longer lead times. Multi-model (5 models, 45 members) ECMWF Stochastic Physics (9 members) Perturbed Parameters (9 members) Stream 1 annual hindcasts

7. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Reliability diagrams for 1-month lead seasonal (JJA) precipitation above the upper tercile over the tropical band for the ENSEMBLES Stream 1 multi-model (left, 45 members), stochastic physics (centre, 9 members) and perturbed parameters (right, 9 members) hindcasts started in May over the period 1991-2001 verified against GPCP. The Brier and ROC skill scores, along with 95% confidence intervals (in brackets) computed using a bootstrap method, are shown on top of each panel. Multi-model 0.110 (0.060,0.160) 0.415 (0.348,0.477) Stochastic physics 0.018 (-0.043,0.070) 0.347 (0.272,0.411) Perturbed parameters 0.016 (-0.054,0.083) 0.339 (0.271,0.408) Stream 1 seasonal hindcasts

8. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Scores for southern South America precipitation from Stochastic Physics, Perturbed Parameters (both with 9- member ensembles) and Multi-model (5 models, 45 members). Sample values are shown with black dots along with 95% confidence intervals obtained using a bootstrap method (verified against GPCP over 1991-2001). Anomaly correlation coefficient Ratio between spread and RMSE ROCSS for anomalies above the lower tercile Stream 1 seasonal hindcasts

9. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Stream 1 seasonal hindcasts 200 17 157 78 156 58 Ratio of spread to RMSE of the three different forecast systems for several regions, lead times (1-1, 2-4, 5-7 months), events (anomalies above/below the upper/lower tercile), start dates (May and November) and variables (T2m, precipitation, Z500 and MSLP) computed over the period 1991-2001. The inset numbers indicate the number of cases where a system is superior with 95% confidence.

10. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Stream 1 seasonal hindcasts 115 25 203 9 33 106 ROC skill score of the three different forecast systems for several regions, lead times (1-1, 2-4, 5-7 months), events (anomalies above/below the upper/lower tercile), start dates (May and November) and variables (T2m, precipitation, Z500 and MSLP) computed over the period 1991-2001. The inset numbers indicate the number of cases where a system is superior with 95% confidence.

11. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Stream 2 ROC area for Stream 2 JJA (start date 1 st of May) 2-metre temperature above the upper tercile over the period 1960-2005 (left) and 1981-2005 (right, hindcast period for ECMWF System 3). 1960-2005 1981-2005

12. First EUROBRISA Workshop17 March 2008The ENSEMBLES project From Keenlyside et al. (2008) 10-year mean prediction of meridional overturning circulation and tropical Pacific SST from 3-member ensembles started every five years over 1955-2005. Decadal predictions: IfM

13. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Decadal predictions: IfM From Keenlyside et al. (2008) 10-year mean prediction of global mean surface temperature from 3-member ensembles started every five years.

14. First EUROBRISA Workshop17 March 2008The ENSEMBLES project T2m difference with ERA40/OPS for 10-year ensemble integrations started on 1 st Nov 1994 of different IFS cycles (HOPE ocean model). Global land NH land Global ocean Tropical oceans Decadal predictions: ECMWF

15. First EUROBRISA Workshop17 March 2008The ENSEMBLES project First two years of predictions (anomalies wrt 1979-2001) from June 2005 (Jun 2005-May 2007) for uninitialized (NoAssim), initialized (DePreSys) and AR4 simulations. Comparison to NCEP analyses. Note the improved forecast for initialized predictions, although not everywhere. D. Smith & J. Murphy (Met Office) Decadal forecasts: DePreSys

16. First EUROBRISA Workshop17 March 2008The ENSEMBLES project DePreSys: Stream 1 decadal predictions Anomaly correlation coefficient for DePreSys_PP as a function of lead time (from 1-1 to 27-62). Each dot shows the result for a version of the model, the final dot of each set being for the ensemble mean. The coloured bars indicate the 95% confidence intervals obtained using a bootstrap method with 1,000 samples. Tropical precipitation Northern Hemisphere T850

17. First EUROBRISA Workshop17 March 2008The ENSEMBLES project DePreSys: Stream 1 decadal predictions Scores for the DePreSys versus DePreSys_PP for several regions (tropics, Northern Hemisphere, Europe, North America), lead times (3-14, 15-26, 27-62 months), events (anomalies above/below the upper/lower tercile and above the median), start dates (May and November) and variables (T2m, Z500, T850, precipitation and MSLP) over the period 1991-2001. The inset numbers give the cases when the value for a system is significantly above the corresponding value of the other. BSSIRelSS 6 43 3 ROCSS 9 18

18. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Thirty-year DePreSys forecasts started on March 2007 for uninitialized (NoAssim, 4 members) and initialized (DePreSys, 10 members) ensembles. Surface temperature THC intensity D. Smith & J. Murphy (Met Office) 30-year climate forecasts: DePreSys

19. First EUROBRISA Workshop17 March 2008The ENSEMBLES project A user strategy: the end-to-end approach A broad range of forecast products might be offered, but a specific analysis of the user requirements is necessary. End-to-end is based on collaboration and continuous feedback. End users develop their models taking into account climate prediction limitations. Users should use objective records of performance. The final level of forecast quality that provides added value is defined by the application -> user-oriented verification. End users should assess the final value of the predictions. Forecast reliability becomes a major issue.

20. First EUROBRISA Workshop17 March 2008The ENSEMBLES project ●The possibility of adaptation to climate change via a learning process taking place at the interannual time scale, when actions can be corrected, is an obvious way to achieve a high degree of integration (integration of time scales) between climate information providers and users. ●This implies: oThat similar forecast information is provided in the climate forecasting and climate-change contexts. oInvolvement of both climate scientists and end-users to consider the whole range of time scales. For instance, crop managers consider long-term climate change as a process that takes place on a yearly basis. Adaptation using climate information

21. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Seasonal prediction with dynamical models N ………… N-1 4 3 2 1 Forecast ………… N N-1 4 3 2 1 Downscaling N ………… N-1 4 3 2 1 Application model 0 Forecast probability of PP Forecasts probability of malaria 0 non-linear transformation

22. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Precipitation composites for the five years with the highest (top row) and lowest (bottom row) malaria incidence in Botswana for DJF DEMETER seasonal predictions (left) and CMAP (right). Areas with epidemic malaria in Africa Quartiles define extreme events (outbreaks) for malaria prediction Dynamical predictions for malaria warning

23. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Probabilistic predictions of standardised malaria incidence quartile categories in Botswana with five months lead time. -- high malaria years -- low malaria years ROC ScorePrecipitationIncidence EventDEMETERCMAPDEMETER Very low0.951.00 Very high0.520.940.84 Very low malaria Very high malaria Available in March Available in November Gain in lead time for malaria warning

24. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Botswana malaria index from Thomson et al. (2005) and incidence simulated by the process-based Liverpool malaria model (LMM) using ERA40. There is a disagreement between both models for the year 2000: is it due to the impact of extreme temperature or precipitation? Interaction of variables (which might be triggered by climate change) affects the user From A. Jones (Univ. of Liverpool) Gain in lead time for malaria warning

25. First EUROBRISA Workshop17 March 2008The ENSEMBLES project ENSEMBLES s2d intends to provide the scientific community with full access to the hindcasts and with automatic tools to perform different analysis. A public data dissemination system with innovative features has been developed. We are collaborating with some ENSEMBLES partners on: oExpanding the KNMI Climate Explorer for exploratory analysis. oDeveloping a downscaling web portal at the University of Cantabria. Public services

26. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Diagnostics and downscaling Climate Explorer & Downscaling portal Hindcasts run/archived at ECMWF (access to member state users) common data atmosphere MARS common data ocean additional data ECFS ECMWF firewall Archiving and dissemination strategy MARS client OPeNDAP server ENSEMBLES public data server (5 Tb) common data atmosphere common data ocean

27. First EUROBRISA Workshop17 March 2008The ENSEMBLES project CFS: oMonthly 9-month long, 15-member ensembles hindcasts over the period 1981-2005 with a coupled ocean-atmosphere. oAtmospheric component initialized from NCEP analysis using lagged average method and ocean from an assimilated ocean state. oT62L64 atmospheric resolution, 1°x1° (1/3° in the tropics) 40 levels ocean. EUROSIP: oECMWF (11 members, 1981-2005 hindcast period), GloSea (15 members, 1987-2005) and Météo-France (5 members, 1993-2005, although now updated to 11 members and 1981-2005) integrated monthly following the EUROSIP strategy. oCommon period 1993-2005. EUROSIP and NCEP CFS

28. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Precipitation biases against GPCP for summer (JJA, May start date) over the period 1993-2005 ECMWF Met Office Météo-France CFS EUROSIP and NCEP CFS: biases

29. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Top row: SST RMSE (solid) and spread (dashed) for the Niño3.4, Niño4 and Niño1.2 regions against Reynolds for the May and November start dates over the period 1981-2005. Bottom row: SST ACC Multi-model has a similar skill to System 3; CFS does not seem to add much skill, although individual forecasts are fairly different; CFS has more variability than System 3 Persistence System 3 CFS Multi-model Niño3.4 Niño4 Niño1.2 EUROSIP and NCEP CFS: tropical SSTs

30. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Scatter plots of EUROSIP vs EUROSIP+CFS multi-model forecast quality over 1993-2005 in terms of reliability skill score (RelSS) and ROC skill score (ROCSS). Symbols indicate results for 19 regions, 2 start dates (May and November), 3 variables (T2m, MSLP, PP), 4 lead times (1-3, 2- 4, 3-5, 4-6) and 2 events (lower and upper terciles). The numbers in brackets indicate differences significant with a 95% confidence using a two-sample inference test.. RelSS ROCSS EUROSIP and NCEP CFS: Forecast quality 101 3 77 22

31. First EUROBRISA Workshop17 March 2008The ENSEMBLES project Summary ●Stream 1 and 2 ensemble simulations are available from ECMWF. In the seasonal/annual time scale, the multi- model seems to provide better results overall. ●Decadal forecasting has made significant progress, forecast systems have been developed and preliminary attempts show that useful climate information is available. ●Progress has been made to increase the public access to the data. ●The inclusion of NCEP CFS in the EUROSIP multi-model increases the reliability of the ensemble (mainly because of the increase in the ensemble size) and the accuracy.

32. First EUROBRISA Workshop17 March 2008The ENSEMBLES project