1. Model verification and it’s relevance for forecasters
Dublin

2. Goals Models are becoming more accurate, however;
The forecasters have access to many models without knowing their weaknesses and limitations..
The roles of the forecaster and the modeller are still very divided.
The actual model verification is performed over long periods or averaged over large area. This only provides general global information, but doesn’t meet the specific needs of the forecaster.
The forecaster would like to know specifically when to trust the accuracy of the model.
In COST733, the evaluation of the different weather types was done by looking at the ability to represent different precipitation patterns in the Alpine domain. For verification purposes we are more interested in differentiating weather classes where the models have difficulties from those where it performs well.

3. History Alps orography T213 and T106
In 1987 only 1 model was available in Switzerland (ECMWF)
The orography was coarse, a good knowledge of the local climate was crucial to provide a forecast.
The forecaster knew the weakness of the model.

4. Which model to trust?

5. COSMO2 Design IFS/ECMWF, 16 km, synoptic scale
4 daily updates
COSMO Design
COSMO 7km, 385x325x45, regional scale
Own assimilation cycle (nudging)
2 daily 72h forecast
COSMO 2.2km, 480x350x60, local scale
Own assimilation cycle (nudging)
8 daily 18h nested forecast
forecast range
clock time
+18h
00 UTC
01 UTC 3h
03 UTC
04 UTC
06 UTC
07 UTC

6. Verification available (SYNOP, spring)

7. Verification available (Upper air)

8. Stratification by weather type

9. New weathertype classification for verification
OLD: Manual classification by E. Zala based on 500 hPa and surface charts
NEW: automatic classification based on IFS analyses (Internship D. Jean-Mairet)
 11 classes: 8 wind directions + flat, high, low pressure
 63 % agreement with Zala classification for the years

10. automatic classifikation gives more N, less H
Weather types for the years (Zala classification and new automatic classification in comparison)
automatic classifikation gives more N, less H

11. Neighborhood Verification
„multi-scale, multi-intensity approach“
calculation of score for growing windows and various intensity thresholds
suitable for precipitation verification
„Fuzzy Verification Toolbox“ of Beth Ebert (
fcst
obs

12. Fuzzy Verification - Methods
Upscaling - ETS
Fraction Skill Score - FSS
Roberts & Lean (2005)
Score: FSS for fractions / probabilities
(0 = mismatch, 1 = perfect match)
Score: ETS (Equitable Threat Score)
(-1/3 = mismatch, 1 = perfect match)
„Good verification approach to evaluate how well the forecast mean value agrees with the mean value of the observations.“
„Spatial averaging damps out areas with high rain rates and spreads areas of lower rain rates.“ (B.Ebert,2009)
„FSS is ideally suited for considering the overall performance and aggregating results over many forecasts” (Mittermaier,2009)
„The FSS is a good measure of the spatial accuracy of precipitation forecasts.“ (Mittermaier,2009)

13. How to read Neighborhood plots…
best skill for low thresholds and large spatial scales
numbers / colors
 Value of the score, hier: FSS
good
bad
increasing spatial scale
bold numbers
 useful scales (only FSS)
low skill for high thresholds and small spatial scales
increasing threshold

14. Results for 2009 Model Observations
3 h acc.: leadtimes 04 – 06 h for each model run of both models (00,03,06,12,15,18,21 UTC), „fairer comparison“
Observations
3 h acc.: Swiss radar composit (NASS product); in the case of missing data, the whole day is omitted (26 days)

15. Skill of COSMO models, 2009 Fractions Skill Score (FSS)
Spatial scale = 100 km
(COSMO-7: 15 grdpts, COSMO-2: 45 grdpts)
COSMO-2
thresholds

16. Differences of FSS, 2009 COSMO-2 minus COSMO-7 NE S N SW F H NW E SE W
COSMO-7 better
COSMO-2 better N SW F H NW E SE W L

17. Example NW, 2009 COSMO-2 better especially for high thresholds…
COSMO-2 - COSMO-7
COSMO-7 better
COSMO-2 better
good
bad

18. Other application of weathertypes
Radar verification (only COSMO-7 at the moment)

19. Guideline

20. Conclusions of the COST meeting in June: Recommendations for guidelines
The guidelines should be self-contained (without links).
They could look like a cookbook, for example for the use of a parameter or for the treatment of a specific situation.
A light version can be at the disposal of the forecaster on duty (usually under time stress) when a longer version can be studied offline.
This longer version can also be used as an education tool for newcomers.
The shorter version can also be implemented as a seasonal factsheet.
The seasonal factsheets should include (if possible) the expected changes of the current model version with respect to the version which was running the previous season.
Generally speaking the guidelines should be short, attractive and meaningful.

21. Forecasters feedback
Forecaster feedback should be organized either by mailbox, a forum or regular discussions.
At the end of each season a debriefing could be organized and a synthesis written. This could form a good base for the following corresponding season.

22. Suggestions for verification

23. Day to day verification (for rainfall and sunshine duration)

24. Synthetic map

25. Advice
Northerly situation:
Precipitation generally
overestimated.

26. Conclusions The verification should be simple and easy to access.
It is important to have a targeted verification of models, not averaged over long periods and large areas such as:
Verification by weather situation.
Verification by season.
Verification by climatic region.
A day to day verification could allow analysis a posteriori of a forecast considering the elements available at the time.
The forecaster would like to know for which situation one model is better than another; which model is better to predict cut-off or ending an Omega block (IFS, GFS)?
With the stratify verification the forecaster can judge the capability of the model according to the weather situation.
A feedback from the forecasters plays a crucial role.
The model physics is frequently upgraded or modified; making it difficult for the forecaster to keep pace with the model’s bias.

27. Thank you

29. … Upscaling (UP) 1. Principle:
(Zepeda-Arce et al., 2000)
1. Principle:
Define box around region of interest and calculate the average of observation and forecast data within this box.
2. Contingency Table
Rave
observation
3. Equitable Threat Score (ETS)
yes no
Hit
False Alarm
Miss
Correct negative
forecast
Q: Which fraction of observed yes - events was correctly forecast?

30. … Fraction Skill Score (FSS)
(Roberts and Lean, 2005)
1. Principle:
Define box around region of interest and determine the fraction pj and oj of grid points with rain rates above a given threshold.
2. Probabilities X x
3. Skill Score for Probabilities
Q: On which spatial scales does the forecast resemble the observation?

31. … Fraction Skill Score (FSS)
(Roberts and Lean, 2005)
4. Useful Scales
useful scales are marked in bold in the graphics

32. Zusammenfassung Was lernen wir aus den Fuzzy-Ergebnissen?
beide COSMO - Modelle haben Skill
gute Vorhersage der räumlichen Struktur auf grösseren Skalen (hohe FSS-Werte)
der Skill der Modelle variiert stark für verschiedene Wetterlagen und auch der Unterschied zwischen COSMO-2 und COSMO-7 ist unterschiedlich stark ausgeprägt
Bester Skill: Frühsommer und Herbst bzw. Süd- und Westlagen
Grösste Differenz COSMO-2 minus COSMO-7: Sommer (Mai bis September) bzw. Nord- und Westlagen sowie konvektive Lagen
Was bleibt unklar?
Interpretation der Egebnisse des Intensity Scale: Gibt es einen räumlichen Shift im COSMO-2?

33. Example Case , UTC

34. Example Case , UTC

35. 3. Useful Scales, 2009 (FSS)
COSMO-2, gridpoints* 2.2 km
COSMO-7, gridpoints* 6.6 km
# cases dx
0.1
0.2
0.5
1.0
2.0
5.0 10 20 NE 9 15 27 / N 63 NW 3 SE 45 S 1 SW E W F H L dx
0.1
0.2
0.5
1.0
2.0
5.0 10 20 NE 5 9 15 / N NW 3 SE 21 S SW E W F H L 10
Determination of minimum spatial scale for each threshold und weathertype, where the forecast can be considered „useful“ according to the definition of the FSS; i.e. the forecast is better than a constant forecast of the wet area ratio. The lower limit of the useful scales is defined by L(FSS lt 0.5+f/2) where f is the observed fractional rainfall coverage over the domain (wet area ratio). 41 41 4 11 59 4 74
Bestimmung der kleinsten Skala [gridpoints] für die die Vorhersage nach der Definition der FSS „useful scales“ nützlich gewesen ist… 59 31 31 17 14 10 7 4 1
0.4
<0.1 17 14 10 7 4 1
0.4
<0.1
% obs gridpts >= thresh (whole period)

36. Bias correction map against rain gauges (long term corrections)
Data sets and methodology RADAR:
24h precipitations sums from 06 to 06 UTC
MeteoSwiss network composit "rain" product
horizontal resolution 2x2km visibility-, clutter- and bias- corrected
                      !!! Note that this bias correction is currently based on precipitation data of 2004! An updated verification with corrections valid for 2008 will follow soon. !!!
Bias correction map against rain gauges (long term corrections)

37. COSMO-7 model:
+06h to +30h  precipitations sums from 00UTC runs
horizontal resolution 7x7km
Weather Type classification:
ele classification
subjective
mainly based on 500hPa geopotential/winds
classes: north, northeast, east, southeast, south, southwest, west, northwest, high, low, flat
new, automatic weather type classification (mo classification)
based on IFS analyses valid at 12 UTC and with a resolution of 1°
first step: distinction between advective and convective classes based on 500hPa wind speed averaged over the alpine domain. > 11 m/s : advective, < 11m/s convective
the advective classes are further divided into the main 8 wind directions based on the GWT (GrossWetterTypes)-Method developed in the COST-Action733. To this end the 500hPa geopotential is correlated with three prototype patterns (Figure below). Based on the correlation coefficients of the zonal and meridional patterns the wind direction is determined. The vorticity pattern is not used.
the convective classes are divided into flat, high and low pressure situations according to the following rules:
average pressure over Alpine domain >= 1019 hPa : high
average pressure over Alpine domain < 1012 hPa: low
average pressure over Alpine domain between those thresholds and all types with 500hPa wind speeds lower than 5 m/s: flat.