1. Verification of multi-model ensemble forecasts using the TIGGE dataset
Verification of multi-model ensemble forecasts using the TIGGE dataset
Laurence J. Wilson
Environment Canada
Anna Ghelli
ECMWF
With thanks to Marcel Vallée

2. Outline Introduction – TIGGE goals and verification
Status of verification of TIGGE ensembles
Standard methods
Spatial methods
Precipitation verification project: plan and early results
Summary
SEE the extended abstract also
November 20, 2018

3. Verification and the goals of TIGGE
Enhance collaborative research
Enable evolution towards GIFS
Develop ensemble combination methods; bias removal
Essential question: If we are going to move towards a GIFS, then we must demonstrate that the benefits of combined ensembles are worth the effort with respect to single-center ensembles. OR: Do we get a “better” pdf by merging ensembles?
Verification – Relevant, user-oriented
November 20, 2018

4. Status of Verification of TIGGE ensembles
Mostly model-oriented verification so far
Upper air data
Against analyses
Standard scoring and case studies
Studies on the TIGGE website
Park et al, 2008
First study involving several months of data
Found modest improvement with combined ensembles, greatest benefits in tropics and lower atmosphere
“Advantage” of using one’s own analysis as truth
Pappenberger et al.
Case study of flooding event in Romania
User-oriented, Q-Q plots, RPS and RMSE main scores used
Multimodel ensemble has best average properties, ECMWF next.
November 20, 2018

5. Studies using TIGGE data (cont’d)
Johnson and Swinbank, 2008
Study of calibration/combination methods
Used only 3 ensembles
Mslp and 2m temperature, but from analyses
Multimodel ensemble improves on individual ensembles, but not by much in general. More at 2m than 500mb
Matsueda 2008
Comparison of 5 combined ensembles vs ECMWF alone
RMSE skill and RPSS with ECMWF as standard forecast
Multimodel eps outperforms ECMWF at medium and longer ranges.
November 20, 2018

6. Status of TIGGE – related verification
Current efforts – Verification of surface variables?
This conference ---?
Studies using spatial methods
Ebert – application of CRA technique to ensemble forecasts. So far, only ECMWF.
Application of Wilks minimum spanning tree or T. Gneiting’s multi-dimensional rank histogram for TC centers. (idea stage)
Precipitation verification project:
November 20, 2018

7. Precipitation verification project
Goal: to verify global 24h precipitation forecasts from all the ensembles in the TIGGE archive and combinations
One region at a time, using highest density observations
Canada and Europe so far
Methodology
Cherubini et al upscaling, verify only where data available
Single station, nearest gridpoint where data is sparser
Kernel density fitting following Peel and Wilson to look at extremes of distributions.
November 20, 2018

8. Precipitation verification project : methodology - Europe
Upscaling:
1x1 gridboxes, limit of model resolution
Average obs over grid boxes, at least 9 stns per grid box (Europe data)
Verify only where enough data
Matches obs and model resolution locally
Answers questions about the quality of the forecasts within the capabilities of the model
Most likely users are modelers.
November 20, 2018

9. European Verification
-Upscaled observations according to Cherubini et al (2002)
-OBS from gauges in Spain, Portugal, France, Italy, Switzerland, Netherlands, Romania, Czech Republic, Croatia, Austria, Denmark, UK, Ireland, Finland and Slovenia
-At least 9 stns needed per grid box to estimate average
-24h precip totals, thresholds 1,3,5,10,15,20,25,30 mm
-one year (oct 07 to oct 08
November 20, 2018

10. Reliability – Summer 08 – Europe – 42h
November 20, 2018

11. Reliability – Summer 08- Europe 114 h
November 20, 2018

12. Reliability – Winter 07-08 – Europe – 114h
November 20, 2018

13. ROC – Summer 08 – Europe – 42h
November 20, 2018

14. ROC – Summer 08 – Europe – 114 h
November 20, 2018

15. Precipitation verification project: methodology - Canada
Single station verification
Canadian verification over 20 widely-spaced stations, only one station per gridbox; comparison of nearest gridpoint fcst to obs
Pointwise verification, does not (we cannot) upscale properly because don’t have the data density necessary.
Valid nevertheless as absolute verification of model predictions
November 20, 2018

16. Results – Canada – ROC curves – 24h
November 20, 2018

17. Results – Canada – ROC Curves – 144h
November 20, 2018

18. RMSE of pcpn probability – Canada – Oct 07 to Oct 08 – 20 stns
2.0 mm
10 mm
BOM in blue (darker blue); ECMWF in red; UKMET in green
CMC in gray; NCEP in cyan (lighter blue)
November 20, 2018

19. Combined ensemble verification
Verification of TIGGE forecasts with respect to surface observations – next steps
Combined ensemble verification
Other regions – Southern Africa should be next. – Non-GTS data is available
Evaluation of extreme events – kernel density fitting to ensembles.
Other high-density observation datasets such as SHEF in the US
Other variables: TC tracks and related surface weather
Use of spatial verification methods
THEN maybe we will know the answer to the TIGGE question.
November 20, 2018

20.
November 20, 2018

21. Issues for TIGGE verification
Use of analyses as truth – advantage of one’s own model. Alternatives:
Each own analysis
Analyses as ensemble (weighted or not)
Random selection from all analyses
Use “best” analysis; eliminate the related model from comparison
Average analysis (may have different statistical characteristics)
Model-independent analysis (restricted to data – rich areas, but that is where verification might be most important for most users
Problem goes away for verification against observations (as long as they are not qc’d with respect to any model)
November 20, 2018

22. Park et al study – impact of analysis used as truth in verification
November 20, 2018

23. Issues for TIGGE Verification (cont’d)
Bias adjustment/calibration
Reason: to eliminate “artifical” spread in combined ensemble arising from systematic differences in component models
First (mean) and second (spread) moments
Several studies have/are being undertaken
Results on benefits not conclusive so far
Due to too small sample for bias estimation?
Alternative: Rather than correcting bias, eliminate inter-ensemble component of bias and spread variation.
November 20, 2018

24. ROC – Winter 07-08 – Europe – 42h
November 20, 2018

25. ROC – Winter 07-08 – Europe – 114h
November 20, 2018

26. Reliability – Winter 07-08 – Europe – 42h
November 20, 2018

27. Results – Canada – Brier Skill, Resolution and Reliability
November 20, 2018