1. Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Extended range forecasts at MeteoSwiss: User experience and probabilistic verification ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel, Mark Liniger, Paul Della-Marta, Christof Appenzeller

2. 2 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Overview Monthly forecasts WWW: Seasonal forecasts Verification: The RPSS D Comparison: ECMWF vs. other prediction strategies

3. 3 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Overview Monthly forecasts WWW: Seasonal forecasts Verification: The RPSS D Comparison: ECMWF vs. other prediction strategies

4. 4 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Monthly forecasts 100 % 0 Probability of T 2m to be in lowest tercile Forecast of week 1 Start: 20-04-2006

5. 5 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Monthly forecasts 100 % 0 Probability of T 2m to be in lowest tercile Forecast of week 1 Start: 04-05-2006

6. 6 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Monthly forecasts Probability of T 2m to be in lowest tercile 100 % 0 Forecast of week 1 Start: 11-05-2006

7. 7 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Monthly forecasts Observed anomalies for May What is wrong? Problems to deal with enhanced snow cover?...

8. 8 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Overview Monthly forecasts WWW: Seasonal forecasts Verification: The RPSS D Comparison: ECMWF vs. other prediction strategies

9. 9 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel WWW: Seasonal forecasts Since winter 2005/06 MeteoSwiss issues an internet bulletin (»Climate Outlook«) on the upcoming season for Switzerland. Designed to... provide seasonal forecast give background information on methodology point out uncertainties provide climatologic background information Provide common reference for public and media, and avoid dissemination of semi-true information Use seasonal forecasts to promote public interest in other aspects of climate analysis

10. 10 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Facts and figures on the summer in Switzerland What do the records show? WWW: Seasonal forecasts

11. 11 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Maximum temperature in °C Precipitation in mm Average sun shine duration in % WWW: Seasonal forecasts

12. 12 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Seasonal forecast current model run. WWW: Seasonal forecasts

13. 13 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel WWW: Seasonal forecasts Terciles from station data

14. 14 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel What is a seasonal forecast? Methodology WWW: Seasonal forecasts

15. 15 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Past seasonal forecasts for Switzerland Verification WWW: Seasonal forecasts

16. 16 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel WWW: Seasonal forecasts observation

17. 17 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Overview Monthly forecasts WWW: Seasonal forecasts Verification: The RPSS D Comparison: ECMWF vs. other prediction strategies

18. 18 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Verification of probabilistic forecasts Real-valued observations Probabilistic forecasts Common approach: Convert observation into probability distribution Ranked Probability Score (RPS)

19. 19 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Verification of probabilistic forecasts Real-valued observations Probabilistic forecasts Ensemble predictions But: Ensemble predictions are not truly probabilistic !! Ranked Probability Score (RPS)

20. 20 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel 0 100% CDF C N W Example: Three equiprobable categories (e.g. cold, normal, warm) Let the verifying observation fall into the second category Convert real-valued observation into CDF The Ranked Probability Score (RPS)

21. 21 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel C N W d 1,EPS d 2,EPS Ensemble prediction system: RPS = d 2 1,EPS + d 2 2,EPS Example: Three equiprobable categories (e.g. cold, normal, warm) Let the verifying observation fall into the second category Compare with CDF of ensemble forecast The Ranked Probability Score (RPS)

22. 22 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel C N W 1/3 2/3 1 d 1,Cl d 2,Cl Ensemble prediction system: RPS = d 2 1,EPS + d 2 2,EPS Climatologic forecast: RPS Cl = d 2 1,Cl + d 2 2,Cl Example: Three equiprobable categories (e.g. cold, normal, warm) Let the verifying observation fall into the second category... or with CDF of climatologic forecast The Ranked Probability Score (RPS)

23. 23 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel The Ranked Probability Skill Score (RPSS) is defined by relating the RPS of a forecast system with the corresponding RPS of the climatologic reference: The RPSS is negatively biased for small ensemble size ! The RPSS

24. 24 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Synthetic random white noise forecasts, verified against random white noise observations. Skill of this forecast system should be zero by definition ! Three equiprobable categories The RPSS

25. 25 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Negative bias consequence of inconsistent definition of climatologic reference forecast. Müller et al. 2005, J.Clim. Weigel et al. 2006, Mon. Wea. Rev. The RPSS 1/3

26. 26 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Solution K:Number of forecast categories p i :Prob. of i-th forecast category M:Ensemble size General case Weigel et al. 2006, Mon. Wea. Rev. The RPSS D

27. 27 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Special case 1: K equiprobable forecast categories M: Ensemble size Solution The RPSS D Weigel et al. 2006, Mon. Wea. Rev.

28. 28 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Special case 2: Brier score, i.e. two categories with prob p and (1-p) M: Ensemble size Solution Weigel et al. 2006, Mon. Wea. Rev. The RPSS D

29. 29 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel The RPSS D Synthetic random white noise forecasts, verified against random white noise observations. Skill of this forecast system should be zero by definition ! Three equiprobable categories

30. 30 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel ECMWF System 2 forecasts (1988-2002), verified against ERA40 T2m predictions for March, lead time 4 months 2 equiprobable forecast categories (i.e. Brier Score situation) Southern Africa The RPSS D

31. 31 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Large ensembles still useful! RPSS D determines the true skill of the EPS It measures model quality, not forecast quality Particularly useful for model assessment studies: multi-model studies, when models of different ensemble size are to be compared comparison of deterministic and probabilistic forecasts The RPSS D

32. 32 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Overview Monthly forecasts WWW: Seasonal forecasts Verification: The RPSS D Comparison: ECMWF vs. other prediction strategies

33. 33 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Example 1: Statistical model Model:CCA statistical model Training:1880-1960 Verification:1960-2001 Predictors: DJF North Atlantic SST JFMA total precipitation (north. Mediterranian) Predictand: JJA daily homogenized T max station series Reference:Della-Marta et al. (2006), Clim. Dyn.

34. 34 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel ECMWF (DEMETER) CCA model Example 1: Statistical model Three equiprobable forecast categories JJA forecasts of T2m, initialized in May Verification period: 1960-2001 RPSS D

35. 35 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Example 2: The Böögg Böögg: RPSS D = -0.15 ECMWF: RPSS D = 0.19

36. 36 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Bööggs Prognosis for summer 2006: Time until head exploded: 10 minutes 28 seconds warm summer

37. 37 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel The Ranked Probability Score (RPS) Example: Three equiprobable categories (e.g. cold, normal, warm) Let the verifying observation fall into the second category C N W Observation

38. 38 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel The Ranked Probability Score (RPS) Example: Three equiprobable categories (e.g. cold, normal, warm) Let the verifying observation fall into the second category 0 100% PDF C N W Convert real-valued observation into PDF

39. 39 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Böögg Time until head explodes (min) Mean JJA temperature R 2 = 0.0071 p= 0.599 RPSS D = -0.15 ECMWF (DEMETER) Ensemble mean for JJA T2m R 2 = 0.2497 p= 0.0016 RPSS D = 0.19 heat summer 2003 Example 2: The Böögg

40. 40 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006 Andreas Weigel Central Europe The RPSS D ECMWF System 2 forecasts (1988-2002), verified against ERA40 T2m predictions for March, lead time 4 months 2 equiprobable forecast categories (i.e. Brier Score situation)