Download presentation
Presentation is loading. Please wait.
1
Christoph Gebhardt, Zied Ben Bouallègue, Michael Buchhold
COSMO-DE-EPS Susanne Theis Christoph Gebhardt, Zied Ben Bouallègue, Michael Buchhold
2
Presentation Overview
setup of COSMO-DE-EPS first results of pre-operational phase verification forecasters‘ feedback enlarging the sample at low cost looking into past production cycles COSMO GM – September 2011
3
Presentation Overview
setup of COSMO-DE-EPS first results of pre-operational phase verification forecasters‘ feedback enlarging the sample at low cost looking into past production cycles new new COSMO GM – September 2011
4
Setup of COSMO-DE-EPS
5
COSMO-DE-EPS status pre-operational phase has started: Dec 9th, 2010
pre-operational setup: 20 members grid size: 2.8 km convection-permitting lead time: 0-21 hours, 8 starts per day (00, 03, 06,... UTC) variations in physics, initial conditions, lateral boundaries model domain COSMO GM – September 2011
6
Generation of Ensemble Members
plus variations of initial conditions model physics Generation of Ensemble Members Ensemble Chain COSMO-DE-EPS 2.8km COSMO 7km BC-EPS GME, IFS, GFS, GSM COSMO GM – September 2011
7
Generation of Ensemble Members
plus variations of initial conditions model physics Generation of Ensemble Members Ensemble Chain COSMO-DE-EPS 2.8km COSMO 7km BC-EPS BC-EPS is running as a time-critical application at ECMWF GME, IFS, GFS, GSM COSMO GM – September 2011
8
Generation of Ensemble Members
Perturbation Methods Peralta, C., Ben Bouallègue, Z., Theis, S.E., Gebhardt, C. and M. Buchhold, 2011: Accounting for initial condition uncertainties in COSMO-DE-EPS. Submitted to Journal of Geophysical Research. Peralta, C. and M. Buchhold, 2011: Initial condition perturbations for the COSMO-DE-EPS, COSMO Newsletter 11, 115–123. Gebhardt, C., Theis, S.E., Paulat, M. and Z. Ben Bouallègue, 2011: Uncertainties in COSMO-DE precipitation forecasts introduced by model perturbations and variation of lateral boundaries. Atmospheric Research 100, (contains status of 2009) COSMO GM – September 2011
9
COSMO-DE-EPS plans (2011-2014)
upgrade to 40 members, redesign statistical postprocessing initial conditions by LETKF lateral boundary conditions by ICON EPS 2011 reach operational status 2012 2013 COSMO GM – September 2011
10
First Results of Pre-operational Phase. - verification
First Results of Pre-operational Phase - verification - forecasters‘ feedback new
11
First Results of Pre-operational Phase. - verification
First Results of Pre-operational Phase - verification - forecasters‘ feedback new
12
Verification Method SYNOP RADAR Ensemble Members
Probabilities of Precipitation COSMO GM – September 2011
13
for Individual Members
PREC 1h accumulation, threshold: 0.1 mm DETERMINISTIC SCORES for Individual Members Do the ensemble members have different long-term statistics? (multi-model / multi-configuration) Are there many cases with the same „best member“ or „wettest member“? look at Equitable Threat Score - look at Frequency Bias Index (results similar, not shown) 0.5 0.4 0.3 0.2 0.1 0.0 Equitable Threat Score JUNE 2011 IFS GME GFS GSM } 20 members Forecast Time [h] COSMO GM – September 2011
14
} 20 members PREC 1h accumulation, threshold: 0.1 mm
DETERMINISTIC SCORES for Individual Members Do the ensemble members have different long-term statistics? (multi-model / multi-configuration) Are there many cases with the same „best member“ or „wettest member“? look at Equitable Threat Score - look at Frequency Bias Index (results similar, not shown) 0.5 0.4 0.3 0.2 0.1 0.0 Equitable Threat Score JUNE 2011 IFS GME GFS GSM } 20 members Forecast Time [h] Only small differences in long-term statistics Members may be treated as equally probable COSMO GM – September 2011
15
PREC 1h accumulation RANK HISTOGRAM observation...
…treated as „Ensemble Member“ …ranked according to prec amount at each grid point and forecast hour How frequent is each rank? If ensemble underdispersive U-shaped rank histogram RANK HISTOGRAM 0.05 0.00 JUNE 2011 Frequency Observation Rank COSMO GM – September 2011
16
PREC 1h accumulation RANK HISTOGRAM observation...
…treated as „Ensemble Member“ …ranked according to prec amount at each grid point and forecast hour How frequent is each rank? If ensemble underdispersive U-shaped rank histogram RANK HISTOGRAM 0.05 0.00 JANUARY 2011 Frequency Observation Rank COSMO GM – September 2011
17
PREC 1h accumulation RANK HISTOGRAM observation...
…treated as „Ensemble Member“ …ranked according to prec amount at each grid point and forecast hour How frequent is each rank? If ensemble underdispersive U-shaped rank histogram RANK HISTOGRAM 0.05 0.00 JANUARY 2011 Frequency Observation Rank a) Underdispersiveness relatively small b) Four groups Many cases with large influence by global models COSMO GM – September 2011
18
PREC 1h accumulation BRIER SKILL SCORE How good are the probabilities
derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) BRIER SKILL SCORE JANUARY 2011 > 0.1 mm > 1 mm > 2 mm Forecast Time [h] COSMO GM – September 2011
19
Always positive! Ensemble provides additional value to COSMO-DE
PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) BRIER SKILL SCORE JANUARY 2011 > 0.1 mm > 1 mm > 2 mm Forecast Time [h] Always positive! Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability) COSMO GM – September 2011
20
Always positive! Ensemble provides additional value to COSMO-DE
PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) BRIER SKILL SCORE JUNE 2011 > 0.1 mm > 1 mm > 2 mm Forecast Time [h] Always positive! Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability) COSMO GM – September 2011
21
For larger precipitation amounts (summer): even more additional value
PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (x-axis) (probabilites of exceeding a certain threshold) - for all foreast lead times BRIER SKILL SCORE MAY - JULY 2011 Threshold [mm/h] For larger precipitation amounts (summer): even more additional value COSMO GM – September 2011
22
PREC 1h accumulation RELIABILITY DIAGRAM Are the probabilities already
well calibrated? (without extra calibration) If we isolate all cases with a forecast probability of -say % … did the event occur in 80% of these cases? diagonal line: optimal - for different prec thresholds (colors) (probs of exceeding a threshold) RELIABILITY DIAGRAM JUNE 2011 log (# fcst) > 0.1 mm > 1 mm > 2 mm COSMO GM – September 2011
23
PREC 1h accumulation RELIABILITY DIAGRAM Are the probabilities already
well calibrated? (without extra calibration) If we isolate all cases with a forecast probability of -say % … did the event occur in 80% of these cases? diagonal line: optimal - for different prec thresholds (colors) (probs of exceeding a threshold) RELIABILITY DIAGRAM JUNE 2011 log (# fcst) > 0.1 mm > 1 mm > 2 mm Reliability diagram shows some bias and underdispersiveness Lines are not flat additional calibration has good potential COSMO GM – September 2011
24
Summary of Verification (Precipitation)
Ensemble provides additional value to COSMO-DE (for all accumulations, lead times, precipitation thresholds,…) Ensemble underdispersiveness is relatively small Ensemble members may be treated as equally probable Additional calibration has good potential COSMO GM – September 2011
25
Summary of Verification (Precipitation)
Ensemble provides additional value to COSMO-DE (for all accumulations, lead times, precipitation thresholds,…) Ensemble underdispersiveness is relatively small Ensemble members may be treated as equally probable Additional calibration has good potential Pre-operational COSMO-DE ensemble prediction system already meets fundamental quality requirements for precipitation COSMO GM – September 2011
26
Other Variables T_2M and VMAX have been verified
ensemble spread is far too small nevertheless, ensemble provides additional value to COSMO-DE COSMO GM – September 2011
27
Other Variables T_2M and VMAX have been verified
ensemble spread is far too small nevertheless, ensemble provides additional value to COSMO-DE COSMO-DE ensemble prediction system has been developed with focus on precipitation Upgrade to 40 members will also look at other variables COSMO GM – September 2011
28
First Results of Pre-operational Phase. - verification
First Results of Pre-operational Phase - verification - forecasters‘ feedback new
29
Forecasters‘ Feedback
available products: see figure precipitation, snow, wind gusts, T_2m probability thresholds: warning criteria all products on grid-scale (2.8km) in addition: precipitation probabilities for larger areas (10x10 grid boxes) „probability that the precipitation event will occur anywhere within the region“ probabilities, quantiles, ensemble mean, spread, min, max, … COSMO GM – September 2011
30
Forecasters‘ Feedback
evaluate „full package“ including the visualization tool consistency of products select interesting cases consider forecasters‘ interpretation perception as intended? is there any value in the forecast, additional to forecasters‘ knowledge? COSMO GM – September 2011
31
Forecasters‘ Feedback
what they prefer to use: 90%-quantile of precipitation precipitation probabilities for an area (10x10 grid points) COSMO GM – September 2011
32
Forecasters‘ Feedback
what they prefer to use: 90%-quantile of precipitation precipitation probabilities for an area (10x10 grid points) what they appreciate: early signals for heavy precipitation indication that deterministic run may be wrong COSMO GM – September 2011
33
Forecasters‘ Feedback
what they prefer to use: 90%-quantile of precipitation precipitation probabilities for an area (10x10 grid points) what they appreciate: early signals for heavy precipitation indication that deterministic run may be wrong what they criticize: jumpiness between subsequent runs lack of spread in T_2M and VMAX COSMO GM – September 2011
34
Forecasters‘ Feedback
what they prefer to use: 90%-quantile of precipitation precipitation probabilities for an area (10x10 grid points) what they appreciate: early signals for heavy precipitation indication that deterministic run may be wrong what they criticize: jumpiness between subsequent runs lack of spread in T_2M and VMAX what they are learning: dealing with low probabilities (10% probability for extreme weather issue a warning?) COSMO GM – September 2011
35
Enlarging the Sample at Low Cost - looking into past production cycles
new
36
Enlarging the Sample at Low Cost - looking into past production cycles Introduction - Verification Study - Recommendation new
37
COSMO-DE-EPS Overview
December 2010 start of pre-operational phase / evaluation 20 members probabilities, quantiles, etc runs at 00 UTC, 03 UTC, 06 UTC,… COSMO GM – September 2011
38
COSMO-DE-EPS Overview
December 2010 start of pre-operational phase / evaluation 20 members probabilities, quantiles, etc runs at 00 UTC, 03 UTC, 06 UTC,… start of pre-operational phase with 40 ensemble members reach operational status statistical postprocessing of probabilities of precipitation Q1 2012 End of 2012 2013 COSMO GM – September 2011
39
COSMO-DE-EPS Overview
December 2010 start of pre-operational phase / evaluation 20 members probabilities, quantiles, etc runs at 00 UTC, 03 UTC, 06 UTC,… start of pre-operational phase with 40 ensemble members reach operational status statistical postprocessing of probabilities of precipitation further improvements ? Q1 2012 End of 2012 2013 COSMO GM – September 2011
40
Enlarging the Sample at Low Cost
looking into past production cycles 0 UTC 15 UTC 18UTC } 20 members 21UTC } 20 members 00UTC } 20 members COSMO GM – September 2011
41
Enlarging the Sample at Low Cost
looking into past production cycles 0 UTC 15 UTC 18UTC } 20 members 21UTC } 20 members 00UTC } 20 members } 60 members COSMO GM – September 2011
42
Important Check How good are forecasts from past production cycles?
COSMO GM – September 2011
43
Important Check How good are forecasts from past production cycles?
June 2011 precipitation 1h accumulations threshold: 1 mm/h Forecast Start: 00UTC 21UTC 18UTC FBI ETS similar results for different months different starting times different thresholds 6h accumulations time of day [UTC] time of day [UTC] COSMO GM – September 2011
44
Important Check How good are forecasts from past production cycles?
June 2011 precipitation 1h accumulations threshold: 1 mm/h Forecast Start: 00UTC 21UTC 18UTC FBI ETS similar results for different months different starting times different thresholds 6h accumulations time of day [UTC] time of day [UTC] The quality is similar. Exception: Most recent run with forecast lead time 0-3 hours is the best. COSMO GM – September 2011
45
Resulting Probabilities
Probability of Precipitation > 10 mm/h Resulting Probabilities 20 members members % UTC Forecast Start: 09 UTC Forecast Start: 09 UTC, 06 UTC, 03 UTC COSMO GM – September 2011
46
Look into past production cycles
Summary of Study looking into past production cycles ( ) quality gain for precipitation (except first 2 forecast hours) does not harm quality of T_2M and VMAX_10M also applicable to „area probabilities“ Recommendation: Look into past production cycles COSMO GM – September 2011
47
Technical aspect: how to derive probabilities from 20+20+20 members
0 UTC 15 UTC 21UTC 18UTC 21UTC 00UTC 21h forecast in (pre-)operational mode + 6h additional lead time required COSMO GM – September 2011
48
COSMO GM – September 2011
49
Extra Slides
50
Example
51
Resulting Probabilities
P(TOT_PREC) > 10mm/h : 00UTC h 20 members members COSMO GM – September 2011
52
Resulting Probabilities
P(TOT_PREC) > 10mm/h : 03UTC h 20 members members COSMO GM – September 2011
53
Resulting Probabilities
P(TOT_PREC) > 10mm/h : 06UTC h 20 members members COSMO GM – September 2011
54
Resulting Probabilities
P(TOT_PREC) > 10mm/h : 09UTC h 20 members members COSMO GM – September 2011
55
Resulting Probabilities
P(TOT_PREC) > 10mm/h : 12UTC h 20 members members COSMO GM – September 2011
56
Resulting Probabilities
P(TOT_PREC) > 10mm/h : 12UTC h 20 members members very nice side effect: less “jumpiness” COSMO GM – September 2011
57
Enlarging the Sample at Low Cost (2)
looking into a spatial neighbourhood at each grid point Schwartz et al., 2010 Wea. Forecasting only applicable to probabilites at a grid point not applicable to „area probabilities“ COSMO GM – September 2011
58
% Example Probability of Precipitation > 5 mm/h 2011-06-16 20UTC
20 members + neighbourhood (radius 10 Δx) 20 members % Forecast Start: 12 UTC Forecast Start: 12 UTC COSMO GM – September 2011
59
Quality Gain in Resulting Probabilities
Probability of Precipitation > 1 mm/h Quality Gain in Resulting Probabilities members Resolution Gain Reliability Gain Sharpness Loss most important 20 members + neighbourh. reference: 20 members June 2011 time of day [UTC] time of day [UTC] time of day [UTC] Definition of scores: Ben Bouallègue, Z., 2011: Upscaled and fuzzy probabilistic foreasts: verification results. COSMO Newsletter 11, COSMO GM – September 2011
60
Quality Gain in Resulting Probabilities
Probability of Precipitation > 1 mm/h Quality Gain in Resulting Probabilities members Resolution Gain Reliability Gain Sharpness Loss most important 20 members + neighbourh. reference: 20 members June 2011 time of day [UTC] time of day [UTC] time of day [UTC] Precipitation: Both methods achieve clear quality gain (except first 2 hours). COSMO GM – September 2011
61
Quality Gain in Resulting Probabilities
Probability of Precipitation Quality Gain in Resulting Probabilities members Resolution Gain Reliability Gain Sharpness Loss most important 20 members + neighbourh. reference: 20 members June 2011 threshold [mm/h] threshold [mm/h] threshold [mm/h] Same conclusion for different precipitation thresholds COSMO GM – September 2011
62
20+20+20 does not harm quality
Probability of 2m-Temperature 2m-Temperature (T_2M) members Resolution Gain Reliability Gain Sharpness Loss most important 20 members + neighbourh. reference: 20 members June 2011 threshold [°C] threshold [°C] threshold [°C] For 2m-temperature: does not harm quality COSMO GM – September 2011
63
20+20+20 does not harm quality
Probability of Wind Gusts Wind Gusts (VMAX_10M) members Resolution Gain Reliability Gain Sharpness Loss most important 20 members + neighbourh. reference: 20 members June 2011 threshold [m/s] threshold [m/s] threshold [m/s] For wind gusts: does not harm quality COSMO GM – September 2011
64
Weights? F = (1- W) L1 + W L2 0<W<1 Gain compared to L1
COSMO GM – September 2011
65
Technical Setup of the Ensemble Chain
12 18 00 06 12 18 4 global models IFS start BC-EPS at ECMWF 12 18 00 06 12 18 arrival BC-EPS at DWD 12 15 18 21 00 03 06 09 12 15 18 start COSMO-DE-EPS +06 +09 +06 +09 +06 +09 +06 +09 +06 +09 +06 12 15 18 21 00 03 06 09 12 15 18 COSMO GM – September 2011
66
Looking back how far? 00UTC 09UTC 2runs 3runs 4runs 2runs 3runs 4runs
BSS raw ensemble (1run) as reference COSMO GM – September 2011
Similar presentations
