Christoph Gebhardt, Zied Ben Bouallègue, Michael Buchhold COSMO-DE-EPS Susanne Theis Christoph Gebhardt, Zied Ben Bouallègue, Michael Buchhold

Presentation Overview setup of COSMO-DE-EPS first results of pre-operational phase verification forecasters‘ feedback enlarging the sample at low cost looking into past production cycles COSMO GM – September 2011

Presentation Overview setup of COSMO-DE-EPS first results of pre-operational phase verification forecasters‘ feedback enlarging the sample at low cost looking into past production cycles new new COSMO GM – September 2011

Setup of COSMO-DE-EPS

COSMO-DE-EPS status pre-operational phase has started: Dec 9th, 2010 pre-operational setup: 20 members grid size: 2.8 km convection-permitting lead time: 0-21 hours, 8 starts per day (00, 03, 06,... UTC) variations in physics, initial conditions, lateral boundaries model domain COSMO GM – September 2011

Generation of Ensemble Members plus variations of initial conditions model physics Generation of Ensemble Members Ensemble Chain COSMO-DE-EPS 2.8km COSMO 7km BC-EPS GME, IFS, GFS, GSM COSMO GM – September 2011

Generation of Ensemble Members plus variations of initial conditions model physics Generation of Ensemble Members Ensemble Chain COSMO-DE-EPS 2.8km COSMO 7km BC-EPS BC-EPS is running as a time-critical application at ECMWF GME, IFS, GFS, GSM COSMO GM – September 2011

Generation of Ensemble Members Perturbation Methods Peralta, C., Ben Bouallègue, Z., Theis, S.E., Gebhardt, C. and M. Buchhold, 2011: Accounting for initial condition uncertainties in COSMO-DE-EPS. Submitted to Journal of Geophysical Research. Peralta, C. and M. Buchhold, 2011: Initial condition perturbations for the COSMO-DE-EPS, COSMO Newsletter 11, 115–123. Gebhardt, C., Theis, S.E., Paulat, M. and Z. Ben Bouallègue, 2011: Uncertainties in COSMO-DE precipitation forecasts introduced by model perturbations and variation of lateral boundaries. Atmospheric Research 100, 168-177. (contains status of 2009) COSMO GM – September 2011

COSMO-DE-EPS plans (2011-2014) upgrade to 40 members, redesign statistical postprocessing initial conditions by LETKF lateral boundary conditions by ICON EPS 2011 reach operational status 2012 2013 COSMO GM – September 2011

First Results of Pre-operational Phase. - verification First Results of Pre-operational Phase - verification - forecasters‘ feedback new

First Results of Pre-operational Phase. - verification First Results of Pre-operational Phase - verification - forecasters‘ feedback new

Verification Method SYNOP RADAR Ensemble Members Probabilities of Precipitation COSMO GM – September 2011

for Individual Members PREC 1h accumulation, threshold: 0.1 mm DETERMINISTIC SCORES for Individual Members Do the ensemble members have different long-term statistics? (multi-model / multi-configuration) Are there many cases with the same „best member“ or „wettest member“? look at Equitable Threat Score - look at Frequency Bias Index (results similar, not shown) 0.5 0.4 0.3 0.2 0.1 0.0 Equitable Threat Score JUNE 2011 IFS GME GFS GSM } 20 members 0 5 10 15 20 Forecast Time [h] COSMO GM – September 2011

} 20 members PREC 1h accumulation, threshold: 0.1 mm DETERMINISTIC SCORES for Individual Members Do the ensemble members have different long-term statistics? (multi-model / multi-configuration) Are there many cases with the same „best member“ or „wettest member“? look at Equitable Threat Score - look at Frequency Bias Index (results similar, not shown) 0.5 0.4 0.3 0.2 0.1 0.0 Equitable Threat Score JUNE 2011 IFS GME GFS GSM } 20 members 0 5 10 15 20 Forecast Time [h] Only small differences in long-term statistics  Members may be treated as equally probable COSMO GM – September 2011

PREC 1h accumulation RANK HISTOGRAM observation... …treated as „Ensemble Member“ …ranked according to prec amount at each grid point and forecast hour How frequent is each rank? If ensemble underdispersive  U-shaped rank histogram RANK HISTOGRAM 0.05 0.00 JUNE 2011 Frequency 1 6 11 16 21 Observation Rank COSMO GM – September 2011

PREC 1h accumulation RANK HISTOGRAM observation... …treated as „Ensemble Member“ …ranked according to prec amount at each grid point and forecast hour How frequent is each rank? If ensemble underdispersive  U-shaped rank histogram RANK HISTOGRAM 0.05 0.00 JANUARY 2011 Frequency 1 6 11 16 21 Observation Rank COSMO GM – September 2011

PREC 1h accumulation RANK HISTOGRAM observation... …treated as „Ensemble Member“ …ranked according to prec amount at each grid point and forecast hour How frequent is each rank? If ensemble underdispersive  U-shaped rank histogram RANK HISTOGRAM 0.05 0.00 JANUARY 2011 Frequency 1 6 11 16 21 Observation Rank a) Underdispersiveness relatively small b) Four groups  Many cases with large influence by global models COSMO GM – September 2011

PREC 1h accumulation BRIER SKILL SCORE How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) BRIER SKILL SCORE JANUARY 2011 > 0.1 mm > 1 mm > 2 mm 0 5 10 15 20 Forecast Time [h] COSMO GM – September 2011

Always positive!  Ensemble provides additional value to COSMO-DE PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) BRIER SKILL SCORE JANUARY 2011 > 0.1 mm > 1 mm > 2 mm 0 5 10 15 20 Forecast Time [h] Always positive!  Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability) COSMO GM – September 2011

Always positive!  Ensemble provides additional value to COSMO-DE PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) BRIER SKILL SCORE JUNE 2011 > 0.1 mm > 1 mm > 2 mm 0 5 10 15 20 Forecast Time [h] Always positive!  Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability) COSMO GM – September 2011

For larger precipitation amounts (summer): even more additional value PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (x-axis) (probabilites of exceeding a certain threshold) - for all foreast lead times BRIER SKILL SCORE MAY - JULY 2011 0.1 1 2 5 10 20 Threshold [mm/h] For larger precipitation amounts (summer): even more additional value COSMO GM – September 2011

PREC 1h accumulation RELIABILITY DIAGRAM Are the probabilities already well calibrated? (without extra calibration) If we isolate all cases with a forecast probability of -say- 75-85% … did the event occur in 80% of these cases? diagonal line: optimal - for different prec thresholds (colors) (probs of exceeding a threshold) RELIABILITY DIAGRAM JUNE 2011 log (# fcst) > 0.1 mm > 1 mm > 2 mm COSMO GM – September 2011

PREC 1h accumulation RELIABILITY DIAGRAM Are the probabilities already well calibrated? (without extra calibration) If we isolate all cases with a forecast probability of -say- 75-85% … did the event occur in 80% of these cases? diagonal line: optimal - for different prec thresholds (colors) (probs of exceeding a threshold) RELIABILITY DIAGRAM JUNE 2011 log (# fcst) > 0.1 mm > 1 mm > 2 mm Reliability diagram shows some bias and underdispersiveness Lines are not flat  additional calibration has good potential COSMO GM – September 2011

Summary of Verification (Precipitation) Ensemble provides additional value to COSMO-DE (for all accumulations, lead times, precipitation thresholds,…) Ensemble underdispersiveness is relatively small Ensemble members may be treated as equally probable Additional calibration has good potential COSMO GM – September 2011

Summary of Verification (Precipitation) Ensemble provides additional value to COSMO-DE (for all accumulations, lead times, precipitation thresholds,…) Ensemble underdispersiveness is relatively small Ensemble members may be treated as equally probable Additional calibration has good potential Pre-operational COSMO-DE ensemble prediction system already meets fundamental quality requirements for precipitation COSMO GM – September 2011

Other Variables T_2M and VMAX have been verified ensemble spread is far too small nevertheless, ensemble provides additional value to COSMO-DE COSMO GM – September 2011

Other Variables T_2M and VMAX have been verified ensemble spread is far too small nevertheless, ensemble provides additional value to COSMO-DE COSMO-DE ensemble prediction system has been developed with focus on precipitation Upgrade to 40 members will also look at other variables COSMO GM – September 2011

First Results of Pre-operational Phase. - verification First Results of Pre-operational Phase - verification - forecasters‘ feedback new

Forecasters‘ Feedback available products: see figure precipitation, snow, wind gusts, T_2m probability thresholds: warning criteria all products on grid-scale (2.8km) in addition: precipitation probabilities for larger areas (10x10 grid boxes) „probability that the precipitation event will occur anywhere within the region“ probabilities, quantiles, ensemble mean, spread, min, max, … COSMO GM – September 2011

Forecasters‘ Feedback evaluate „full package“ including the visualization tool consistency of products select interesting cases consider forecasters‘ interpretation perception as intended? is there any value in the forecast, additional to forecasters‘ knowledge? COSMO GM – September 2011

Forecasters‘ Feedback what they prefer to use: 90%-quantile of precipitation precipitation probabilities for an area (10x10 grid points) COSMO GM – September 2011

Forecasters‘ Feedback what they prefer to use: 90%-quantile of precipitation precipitation probabilities for an area (10x10 grid points) what they appreciate: early signals for heavy precipitation indication that deterministic run may be wrong COSMO GM – September 2011

Forecasters‘ Feedback what they prefer to use: 90%-quantile of precipitation precipitation probabilities for an area (10x10 grid points) what they appreciate: early signals for heavy precipitation indication that deterministic run may be wrong what they criticize: jumpiness between subsequent runs lack of spread in T_2M and VMAX COSMO GM – September 2011

Forecasters‘ Feedback what they prefer to use: 90%-quantile of precipitation precipitation probabilities for an area (10x10 grid points) what they appreciate: early signals for heavy precipitation indication that deterministic run may be wrong what they criticize: jumpiness between subsequent runs lack of spread in T_2M and VMAX what they are learning: dealing with low probabilities (10% probability for extreme weather  issue a warning?) COSMO GM – September 2011

Enlarging the Sample at Low Cost - looking into past production cycles new

Enlarging the Sample at Low Cost - looking into past production cycles - Introduction - Verification Study - Recommendation new

COSMO-DE-EPS Overview December 2010 start of pre-operational phase / evaluation 20 members  probabilities, quantiles, etc runs at 00 UTC, 03 UTC, 06 UTC,… COSMO GM – September 2011

COSMO-DE-EPS Overview December 2010 start of pre-operational phase / evaluation 20 members  probabilities, quantiles, etc runs at 00 UTC, 03 UTC, 06 UTC,… start of pre-operational phase with 40 ensemble members reach operational status statistical postprocessing of probabilities of precipitation Q1 2012 End of 2012 2013 COSMO GM – September 2011

COSMO-DE-EPS Overview December 2010 start of pre-operational phase / evaluation 20 members  probabilities, quantiles, etc runs at 00 UTC, 03 UTC, 06 UTC,… start of pre-operational phase with 40 ensemble members reach operational status statistical postprocessing of probabilities of precipitation further improvements ? Q1 2012 End of 2012 2013 COSMO GM – September 2011

Enlarging the Sample at Low Cost looking into past production cycles 0 UTC 15 UTC 18UTC } 20 members 21UTC } 20 members 00UTC } 20 members COSMO GM – September 2011

Enlarging the Sample at Low Cost looking into past production cycles 0 UTC 15 UTC 18UTC } 20 members 21UTC } 20 members 00UTC } 20 members } 60 members COSMO GM – September 2011

Important Check How good are forecasts from past production cycles? COSMO GM – September 2011

Important Check How good are forecasts from past production cycles? June 2011 precipitation 1h accumulations threshold: 1 mm/h Forecast Start: 00UTC 21UTC 18UTC FBI ETS similar results for different months different starting times different thresholds 6h accumulations 0 2 4 6 8 10 12 14 0 2 4 6 8 10 12 14 time of day [UTC] time of day [UTC] COSMO GM – September 2011

Important Check How good are forecasts from past production cycles? June 2011 precipitation 1h accumulations threshold: 1 mm/h Forecast Start: 00UTC 21UTC 18UTC FBI ETS similar results for different months different starting times different thresholds 6h accumulations 0 2 4 6 8 10 12 14 0 2 4 6 8 10 12 14 time of day [UTC] time of day [UTC] The quality is similar. Exception: Most recent run with forecast lead time 0-3 hours is the best. COSMO GM – September 2011

Resulting Probabilities Probability of Precipitation > 10 mm/h Resulting Probabilities 20 members 20 + 20 + 20 members % 2011-05-22 15UTC Forecast Start: 09 UTC Forecast Start: 09 UTC, 06 UTC, 03 UTC COSMO GM – September 2011

Look into past production cycles Summary of Study looking into past production cycles (20+20+20) quality gain for precipitation (except first 2 forecast hours) does not harm quality of T_2M and VMAX_10M also applicable to „area probabilities“ Recommendation: Look into past production cycles COSMO GM – September 2011

Technical aspect: how to derive probabilities from 20+20+20 members 0 UTC 15 UTC 21UTC 18UTC 21UTC 00UTC 21h forecast in (pre-)operational mode + 6h additional lead time required COSMO GM – September 2011

COSMO GM – September 2011

Extra Slides

Example

Resulting Probabilities P(TOT_PREC) > 10mm/h 2011-05-22 : 00UTC 14-15h 20 members 20 + 20 + 20 members COSMO GM – September 2011

Resulting Probabilities P(TOT_PREC) > 10mm/h 2011-05-22 : 03UTC 11-12h 20 members 20 + 20 + 20 members COSMO GM – September 2011

Resulting Probabilities P(TOT_PREC) > 10mm/h 2011-05-22 : 06UTC 08-09h 20 members 20 + 20 + 20 members COSMO GM – September 2011

Resulting Probabilities P(TOT_PREC) > 10mm/h 2011-05-22 : 09UTC 05-06h 20 members 20 + 20 + 20 members COSMO GM – September 2011

Resulting Probabilities P(TOT_PREC) > 10mm/h 2011-05-22 : 12UTC 02-03h 20 members 20 + 20 + 20 members COSMO GM – September 2011

Resulting Probabilities P(TOT_PREC) > 10mm/h 2011-05-22 : 12UTC 02-03h 20 members 20 + 20 + 20 members very nice side effect: less “jumpiness” COSMO GM – September 2011

Enlarging the Sample at Low Cost (2) looking into a spatial neighbourhood at each grid point Schwartz et al., 2010 Wea. Forecasting only applicable to probabilites at a grid point not applicable to „area probabilities“ COSMO GM – September 2011

% Example Probability of Precipitation > 5 mm/h 2011-06-16 20UTC 20 members + neighbourhood (radius 10 Δx) 20 members % Forecast Start: 12 UTC Forecast Start: 12 UTC COSMO GM – September 2011

Quality Gain in Resulting Probabilities Probability of Precipitation > 1 mm/h Quality Gain in Resulting Probabilities 20 + 20 + 20 members Resolution Gain Reliability Gain Sharpness Loss most important 20 members + neighbourh. reference: 20 members 0 5 10 15 0 5 10 15 0 5 10 15 June 2011 time of day [UTC] time of day [UTC] time of day [UTC] Definition of scores: Ben Bouallègue, Z., 2011: Upscaled and fuzzy probabilistic foreasts: verification results. COSMO Newsletter 11, 124-132. COSMO GM – September 2011

Quality Gain in Resulting Probabilities Probability of Precipitation > 1 mm/h Quality Gain in Resulting Probabilities 20 + 20 + 20 members Resolution Gain Reliability Gain Sharpness Loss most important 20 members + neighbourh. reference: 20 members 0 5 10 15 0 5 10 15 0 5 10 15 June 2011 time of day [UTC] time of day [UTC] time of day [UTC] Precipitation: Both methods achieve clear quality gain (except first 2 hours). COSMO GM – September 2011

Quality Gain in Resulting Probabilities Probability of Precipitation Quality Gain in Resulting Probabilities 20 + 20 + 20 members Resolution Gain Reliability Gain Sharpness Loss most important 20 members + neighbourh. reference: 20 members 0.1 1. 2. 5. 0.1 1. 2. 5. 0.1 1. 2. 5. June 2011 threshold [mm/h] threshold [mm/h] threshold [mm/h] Same conclusion for different precipitation thresholds COSMO GM – September 2011

20+20+20 does not harm quality Probability of 2m-Temperature 2m-Temperature (T_2M) 20 + 20 + 20 members Resolution Gain Reliability Gain Sharpness Loss most important 20 members + neighbourh. reference: 20 members 10 20 30 10 20 30 10 20 30 June 2011 threshold [°C] threshold [°C] threshold [°C] For 2m-temperature: 20+20+20 does not harm quality COSMO GM – September 2011

20+20+20 does not harm quality Probability of Wind Gusts Wind Gusts (VMAX_10M) 20 + 20 + 20 members Resolution Gain Reliability Gain Sharpness Loss most important 20 members + neighbourh. reference: 20 members 14 18 14 18 14 18 June 2011 threshold [m/s] threshold [m/s] threshold [m/s] For wind gusts: 20+20+20 does not harm quality COSMO GM – September 2011

Weights? F = (1- W) L1 + W L2 0<W<1 Gain compared to L1 COSMO GM – September 2011

Technical Setup of the Ensemble Chain 12 18 00 06 12 18 4 global models IFS start BC-EPS at ECMWF 12 18 00 06 12 18 arrival BC-EPS at DWD 12 15 18 21 00 03 06 09 12 15 18 start COSMO-DE-EPS +06 +09 +06 +09 +06 +09 +06 +09 +06 +09 +06 12 15 18 21 00 03 06 09 12 15 18 COSMO GM – September 2011

Looking back how far? 00UTC 09UTC 2runs 3runs 4runs 2runs 3runs 4runs BSS raw ensemble (1run) as reference COSMO GM – September 2011