Model verification and it’s relevance for forecasters Dublin 1.10.2010

Goals Models are becoming more accurate, however; The forecasters have access to many models without knowing their weaknesses and limitations.. The roles of the forecaster and the modeller are still very divided. The actual model verification is performed over long periods or averaged over large area. This only provides general global information, but doesn’t meet the specific needs of the forecaster. The forecaster would like to know specifically when to trust the accuracy of the model. In COST733, the evaluation of the different weather types was done by looking at the ability to represent different precipitation patterns in the Alpine domain. For verification purposes we are more interested in differentiating weather classes where the models have difficulties from those where it performs well.

History Alps orography T213 and T106 In 1987 only 1 model was available in Switzerland (ECMWF) The orography was coarse, a good knowledge of the local climate was crucial to provide a forecast. The forecaster knew the weakness of the model.

Which model to trust?

COSMO2 Design IFS/ECMWF, 16 km, synoptic scale 4 daily updates COSMO2 Design COSMO 7km, 385x325x45, regional scale Own assimilation cycle (nudging) 2 daily 72h forecast COSMO 2.2km, 480x350x60, local scale Own assimilation cycle (nudging) 8 daily 18h nested forecast forecast range clock time +18h 00 UTC 01 UTC 3h 03 UTC 04 UTC 06 UTC 07 UTC

Verification available (SYNOP, spring)

Verification available (Upper air)

Stratification by weather type

New weathertype classification for verification OLD: Manual classification by E. Zala based on 500 hPa and surface charts NEW: automatic classification based on IFS analyses (Internship D. Jean-Mairet)  11 classes: 8 wind directions + flat, high, low pressure  63 % agreement with Zala classification for the years 2002-2008

automatic classifikation gives more N, less H Weather types for the years 2002-2009 (Zala classification and new automatic classification in comparison) automatic classifikation gives more N, less H

Neighborhood Verification „multi-scale, multi-intensity approach“ calculation of score for growing windows and various intensity thresholds suitable for precipitation verification „Fuzzy Verification Toolbox“ of Beth Ebert (http://www.bom.gov.au/bmrc/wefor/staff/eee/beth_ebert.htm) fcst obs

Fuzzy Verification - Methods Upscaling - ETS Fraction Skill Score - FSS Roberts & Lean (2005) Score: FSS for fractions / probabilities (0 = mismatch, 1 = perfect match) Score: ETS (Equitable Threat Score) (-1/3 = mismatch, 1 = perfect match) „Good verification approach to evaluate how well the forecast mean value agrees with the mean value of the observations.“ „Spatial averaging damps out areas with high rain rates and spreads areas of lower rain rates.“ (B.Ebert,2009) „FSS is ideally suited for considering the overall performance and aggregating results over many forecasts” (Mittermaier,2009) „The FSS is a good measure of the spatial accuracy of precipitation forecasts.“ (Mittermaier,2009)

How to read Neighborhood plots… best skill for low thresholds and large spatial scales numbers / colors  Value of the score, hier: FSS good bad increasing spatial scale bold numbers  useful scales (only FSS) low skill for high thresholds and small spatial scales increasing threshold

Results for 2009 Model Observations 3 h acc.: leadtimes 04 – 06 h for each model run of both models (00,03,06,12,15,18,21 UTC), „fairer comparison“ Observations 3 h acc.: Swiss radar composit (NASS product); in the case of missing data, the whole day is omitted (26 days)

Skill of COSMO models, 2009 Fractions Skill Score (FSS) Spatial scale = 100 km (COSMO-7: 15 grdpts, COSMO-2: 45 grdpts) COSMO-2 thresholds

Differences of FSS, 2009 COSMO-2 minus COSMO-7 NE S N SW F H NW E SE W COSMO-7 better COSMO-2 better N SW F H NW E SE W L

Example NW, 2009 COSMO-2 better especially for high thresholds… COSMO-2 - COSMO-7 COSMO-7 better COSMO-2 better good bad

Other application of weathertypes Radar verification (only COSMO-7 at the moment)

Guideline

Conclusions of the COST meeting in June: Recommendations for guidelines The guidelines should be self-contained (without links). They could look like a cookbook, for example for the use of a parameter or for the treatment of a specific situation. A light version can be at the disposal of the forecaster on duty (usually under time stress) when a longer version can be studied offline. This longer version can also be used as an education tool for newcomers. The shorter version can also be implemented as a seasonal factsheet. The seasonal factsheets should include (if possible) the expected changes of the current model version with respect to the version which was running the previous season. Generally speaking the guidelines should be short, attractive and meaningful.

Forecasters feedback Forecaster feedback should be organized either by mailbox, a forum or regular discussions. At the end of each season a debriefing could be organized and a synthesis written. This could form a good base for the following corresponding season.

Suggestions for verification

Day to day verification (for rainfall and sunshine duration)

Synthetic map

Advice Northerly situation: Precipitation generally overestimated.

Conclusions The verification should be simple and easy to access. It is important to have a targeted verification of models, not averaged over long periods and large areas such as: Verification by weather situation. Verification by season. Verification by climatic region. A day to day verification could allow analysis a posteriori of a forecast considering the elements available at the time. The forecaster would like to know for which situation one model is better than another; which model is better to predict cut-off or ending an Omega block (IFS, GFS)? With the stratify verification the forecaster can judge the capability of the model according to the weather situation. A feedback from the forecasters plays a crucial role. The model physics is frequently upgraded or modified; making it difficult for the forecaster to keep pace with the model’s bias.

Thank you

… Upscaling (UP) 1. Principle: (Zepeda-Arce et al., 2000) 1. Principle: Define box around region of interest and calculate the average of observation and forecast data within this box. 2. Contingency Table Rave observation 3. Equitable Threat Score (ETS) yes no Hit False Alarm Miss Correct negative forecast Q: Which fraction of observed yes - events was correctly forecast?

… Fraction Skill Score (FSS) (Roberts and Lean, 2005) 1. Principle: Define box around region of interest and determine the fraction pj and oj of grid points with rain rates above a given threshold. 2. Probabilities X x 3. Skill Score for Probabilities Q: On which spatial scales does the forecast resemble the observation?

… Fraction Skill Score (FSS) (Roberts and Lean, 2005) 4. Useful Scales useful scales are marked in bold in the graphics

Zusammenfassung Was lernen wir aus den Fuzzy-Ergebnissen? beide COSMO - Modelle haben Skill gute Vorhersage der räumlichen Struktur auf grösseren Skalen (hohe FSS-Werte) der Skill der Modelle variiert stark für verschiedene Wetterlagen und auch der Unterschied zwischen COSMO-2 und COSMO-7 ist unterschiedlich stark ausgeprägt Bester Skill: Frühsommer und Herbst bzw. Süd- und Westlagen Grösste Differenz COSMO-2 minus COSMO-7: Sommer (Mai bis September) bzw. Nord- und Westlagen sowie konvektive Lagen Was bleibt unklar? Interpretation der Egebnisse des Intensity Scale: Gibt es einen räumlichen Shift im COSMO-2?

Example Case 06.06.2009, 03-06 UTC

Example Case 01.07.2009, 12-15 UTC

3. Useful Scales, 2009 (FSS) COSMO-2, gridpoints* 2.2 km COSMO-7, gridpoints* 6.6 km # cases dx 0.1 0.2 0.5 1.0 2.0 5.0 10 20 NE 9 15 27 / N 63 NW 3 SE 45 S 1 SW E W F H L dx 0.1 0.2 0.5 1.0 2.0 5.0 10 20 NE 5 9 15 / N NW 3 SE 21 S SW E W F H L 10 Determination of minimum spatial scale for each threshold und weathertype, where the forecast can be considered „useful“ according to the definition of the FSS; i.e. the forecast is better than a constant forecast of the wet area ratio. The lower limit of the useful scales is defined by L(FSS lt 0.5+f/2) where f is the observed fractional rainfall coverage over the domain (wet area ratio). 41 41 4 11 59 4 74 Bestimmung der kleinsten Skala [gridpoints] für die die Vorhersage nach der Definition der FSS „useful scales“ nützlich gewesen ist… 59 31 31 17 14 10 7 4 1 0.4 <0.1 17 14 10 7 4 1 0.4 <0.1 % obs gridpts >= thresh (whole period)

Bias correction map against rain gauges (long term corrections) Data sets and methodology RADAR: 24h precipitations sums from 06 to 06 UTC MeteoSwiss network composit "rain" product horizontal resolution 2x2km visibility-, clutter- and bias- corrected                       !!! Note that this bias correction is currently based on precipitation data of 2004! An updated verification with corrections valid for 2008 will follow soon. !!! Bias correction map against rain gauges (long term corrections)

COSMO-7 model: +06h to +30h  precipitations sums from 00UTC runs horizontal resolution 7x7km Weather Type classification: ele classification subjective mainly based on 500hPa geopotential/winds classes: north, northeast, east, southeast, south, southwest, west, northwest, high, low, flat new, automatic weather type classification (mo classification) based on IFS analyses valid at 12 UTC and with a resolution of 1° first step: distinction between advective and convective classes based on 500hPa wind speed averaged over the alpine domain. > 11 m/s : advective, < 11m/s convective the advective classes are further divided into the main 8 wind directions based on the GWT (GrossWetterTypes)-Method developed in the COST-Action733. To this end the 500hPa geopotential is correlated with three prototype patterns (Figure below). Based on the correlation coefficients of the zonal and meridional patterns the wind direction is determined. The vorticity pattern is not used. the convective classes are divided into flat, high and low pressure situations according to the following rules: average pressure over Alpine domain >= 1019 hPa : high average pressure over Alpine domain < 1012 hPa: low average pressure over Alpine domain between those thresholds and all types with 500hPa wind speeds lower than 5 m/s: flat.