1. Deutscher Wetterdienst Preliminary evaluation and verification of the pre-operational COSMO-DE Ensemble Prediction System Susanne Theis Christoph Gebhardt, Zied Ben Bouallègue, Carlos Peralta, Michael Buchhold

2. EMS – September 2011 Presentation Overview  Setup of COSMO-DE-EPS  First results of pre-operational phase  Objective verification  Forecasters‘ feedback; evaluation of case studies

3. Deutscher Wetterdienst Setup of COSMO-DE-EPS

4. EMS – September 2011 COSMO-DE-EPS status  pre-operational phase has started: Dec 9 th, 2010  pre-operational setup:  20 members (upgrade to 40 member in 2012)  grid size: 2.8 km convection-permitting  lead time: 0-21 hours, 8 starts per day (00, 03, 06,... UTC)  variations in physics, initial conditions, lateral boundaries model domain

5. EMS – September 2011 Generation of Ensemble Members “multi-configuration” different configurations of COSMO-DE model Variations in Forecast System for the Representation of Forecast Uncertainty Initial Conditions BoundariesModel Physics “multi-model” driven by different global models “multi-model” COSMO-DE initial conditions modified by different global models

6. EMS – September 2011 Generation of Ensemble Members COSMO-DE-EPS 2.8km COSMO 7km plus variations of initial conditions model physics BC-EPS GME, IFS, GFS, GSM Ensemble Chain

7. Deutscher Wetterdienst First Results of Pre-operational Phase - Objective verification - Forecasters‘ feedback; Evaluation of case studies

8. EMS – September 2011 Focus on precipitation  Ensemble Members  Probabilities of Precipitation SYNOP RADAR Verification

9. EMS – September 2011 PREC 1h accumulation Do the ensemble members have different long-term statistics? (multi-model / multi- configuration) Are there many cases with the same „best member“ or „wettest member“? Only small differences in long-term statistics  Members may be treated as equally probable Only small differences in long-term statistics  Members may be treated as equally probable DETERMINISTIC SCORES for Individual Members } 20 members 0 5 10 15 20 Forecast Time [h] IFS GME GFS GSM Equitable Threat Score 0.5 0.4 0.3 0.2 0.1 0.0 JUNE 2011 FBI threshold 1 mm threshold 0.1 mm

10. 0.05 0.00 EMS – September 2011 PREC 1h accumulation How well does the ensemble spread of the forecast represent the true variability of the observation? RANK HISTOGRAM a) Underdispersiveness relatively small b) Four groups  Many cases with large influence by global models a) Underdispersiveness relatively small b) Four groups  Many cases with large influence by global models Frequency JANUARY 2011 Rank 1 6 11 16 21 JUNE 2011 Rank 1 6 11 16 21 0.05 0.00 Frequency

11. EMS – September 2011 PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) BRIER SKILL SCORE Always positive!  Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability) Always positive!  Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability) JANUARY 2011 0 5 10 15 20 Forecast Time [h] > 0.1 mm > 1 mm > 2 mm

12. EMS – September 2011 PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) BRIER SKILL SCORE Always positive!  Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability) Always positive!  Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability) JUNE 2011 0 5 10 15 20 Forecast Time [h] > 0.1 mm > 1 mm > 2 mm

13. EMS – September 2011 PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (x-axis) (probabilites of exceeding a certain threshold) - for all foreast lead times BRIER SKILL SCORE For larger precipitation amounts (summer): even more additional value For larger precipitation amounts (summer): even more additional value MAY - JULY 2011 0.1 1 2 5 10 20 Threshold [mm/h]

14. EMS – September 2011 PREC 1h accumulation RELIABILITY DIAGRAM Reliability diagram shows some bias and underdispersiveness Lines are sloped  additional calibration has good potential Reliability diagram shows some bias and underdispersiveness Lines are sloped  additional calibration has good potential log (# fcst) > 0.1 mm > 1 mm > 2 mm JUNE 2011 Are the probabilities already well calibrated? (without extra calibration) If we isolate all cases with a forecast probability of -say- 75-85% … did the event occur in 80% of these cases? diagonal line: optimal - for different prec thresholds (colors) (probs of exceeding a threshold)

15. EMS – September 2011  Ensemble provides additional value to COSMO-DE (for all accumulations, lead times, precipitation thresholds,…)  Ensemble underdispersiveness is relatively small  Ensemble members may be treated as equally probable  Additional calibration has good potential Pre-operational COSMO-DE ensemble prediction system already meets fundamental quality requirements for precipitation Pre-operational COSMO-DE ensemble prediction system already meets fundamental quality requirements for precipitation Summary of Verification (Precipitation)

16. EMS – September 2011 Other Variables  T_2M and VMAX have been verified  ensemble spread is far too small COSMO-DE ensemble prediction system has been developed with focus on precipitation COSMO-DE ensemble prediction system has been developed with focus on precipitation Upgrade to 40 members will also look at other variables V_MAX 10m Jan./Feb./ Mar.  nevertheless, ensemble provides additional value to COSMO-DE

17. First Results of Pre-operational Phase - verification - forecasters‘ feedback; evaluation of case studies

18. Observation: 12h-precip, 18 UTC, Synop and Radar 22.07.2011 Severe weather Rostock

19. EMS – September 2011 22.07.2011 Rostock (2) COSMO-DE: 12h-precip Synop: 18 UTC, 12h precip COSMO-DE 00 UTC + 18 h Synop: 18 UTC, 12h precip COSMO-DE 06 UTC + 12 h

20. EMS – September 2011 21 UTC-run (previous day) +21h 00 UTC-run +18 03 UTC-run +15 COSMO-DE-EPS: Prob > 25 mm, 12h-RR 06-18 UTC Good consistency of successive model runs 22.07.2011 Rostock (3)

21. EMS – September 2011 21 UTC-Lauf (Vortag) +21h 00 UTC-Lauf +18 03 UTC-Lauf +15 COSMO-DE-EPS: Prob > 40 mm, 12h-RR 06-18 UTC Probabilities provide good information on location and intensity of the event 22.07.2011 Rostock (4)

22. EMS – September 2011 12h-prec, SRNWP PEPS 00+18h 12h-prec, COSMO-DE-EPS 00+18h 22.07.2011 Rostock (2) Prob > 40 mm, 12h-prec 06-18 UTC COSMO-DE-EPS is able to capture heavy convective precip events provides optimal guidance for forecasters (in this case) COSMO-DE-EPS is able to capture heavy convective precip events provides optimal guidance for forecasters (in this case)

23. EMS – September 2011 4.8.2011, 06 UTC 1h precip [mm/h] Synop Radar 4.8.2011: Severe storm Bremen 4.8.2011, 06 UTC 1h precip [mm/h] Synop COSMO-DE deterministic 21 UTC + 09 h Forecaster: Where to expect the main shower activity and what will be maximum intensity?

24. EMS – September 2011 Synop 06 UTC, 1h precip C-EPS upscaled prob >15mm 18 UTC + 12 h Synop 06 UTC, 1h precip C-EPS Max of all members 18 UTC + 12 h Probability defines area of occurrence Maxmem (or 90% quantile) provides estimation of possible intensities 4.8.2011: Bremen (2)

25. EMS – September 2011 18.8.2011: cold front NRW Radar and Synop: 19 UTC, 1h precip COSMO-DE deterministic 12 UTC + 07 h Large location error in C-DE, horizontal extent of severe precip. too small

26. EMS – September 2011 18.8.2011: cold front NRW (2) Radar, Synop: 19 UTC, 1h precip C-DE-EPS upscaled Prob >15mm/h 12 UTC + 7 h Radar, Synop: 19 UTC, 1h precip C-DE-EPS Perc90% 12 UTC + 7 h C-DE-EPS: better localization and intensity estimation But: BC-EPS does not adequately capture uncertainty on synoptic scale C-DE-EPS: better localization and intensity estimation But: BC-EPS does not adequately capture uncertainty on synoptic scale

27. EMS – September 2011 24.8.2011: severe storm Frankfurt Radar 19 UTC, 1h precip Synop 19 UTC, max wind (1h) COSMO-DE max wind (1h), 09 UTC + 08 h Synop 19 UTC, max wind (1h) C-DE determ. run: no indication of gale-force winds (fx>29 m/s)

28. EMS – September 2011 25.8.2011: Frankfurt (2) C-EPS 90%quant max wind (1h), 06 UTC + 11 h Synop 19 UTC, max wind (1h) C-EPS maxwind prob > 29m/s 06 UTC + 11 h Synop 19 UTC, max wind (1h) C-DE -EPS: clear signals for gale-force winds above 29 m/s. Useful guidance for forecasters despite of timing error

29. EMS – September 2011 Forecasters‘ Feedback  what they prefer to use:  90%-quantile or maximum of all members of precipitation  precipitation probabilities for an area (10x10 grid points)  what they appreciate:  early signals for heavy precipitation  indication that deterministic run may be wrong  what they criticize:  jumpiness between subsequent runs  lack of spread in T_2M and VMAX  what they are learning:  dealing with low probabilities (10% probability for extreme weather  issue a warning?)

30. EMS – September 2011

31. Generation of Ensemble Members Perturbation Methods Peralta, C., Ben Bouallègue, Z., Theis, S.E., Gebhardt, C. and M. Buchhold, 2011: Accounting for initial condition uncertainties in COSMO-DE-EPS. Submitted to Journal of Geophysical Research. Peralta, C. and M. Buchhold, 2011: Initial condition perturbations for the COSMO-DE-EPS, COSMO Newsletter 11, 115–123. Gebhardt, C., Theis, S.E., Paulat, M. and Z. Ben Bouallègue, 2011: Uncertainties in COSMO-DE precipitation forecasts introduced by model perturbations and variation of lateral boundaries. Atmospheric Research 100, 168-177. (contains status of 2009)

32. EMS – September 2011 08.07.2011: Thunderstorm München C-EPS upscal prob > 10mm, 00 UTC + 21 h Synop 21 UTC, 1h precip C-DE -EPS: 21 and 12h probability forecasts provide useful information on the storm C-EPS upscal prob > 10mm, 00 UTC + 12 h

33. EMS – September 2011 C-EPS upscal prob > 10mm, 12 UTC + 09 h Synop 21 UTC, 1h precip C-EPS upscal prob > 10mm, 15 UTC + 06 h C-DE -EPS: Event lost in forecast 12 UTC+9h. Jumpiness of successive runs is a problem! C-DE -EPS: Event lost in forecast 12 UTC+9h. Jumpiness of successive runs is a problem!

34. EMS – September 2011 COSMO-DE-EPS plans (2011-2014)  upgrade to 40 members, redesign  statistical postprocessing  initial conditions by LETKF  lateral boundary conditions by ICON EPS 2011 2012 reach operational status 2013