Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz Weather type dependant fuzzy verification of precipitation COSMO General Meeting, Offenbach, Tanja Weusthoff

2 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Fuzzy Verification „multi-scale, multi-intensity approach“ „Fuzzy verification toolbox“ of B. Ebert Two methods Upscaling (UP) Fraction Skill Score (FSS) present output scale dependent standard setting: 3h accumulations, Jun-Nov 2007, vs Swiss radar data COSMO-2 (2.2km): leadtimes COSMO-7 (6.6km): leadtimes 03-06,06-09,09-12,12-15 fcstobs increasing box size increasing threshold

3 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff … Upscaling (UP) 1. Principle: Define box around region of interest and calculate the average of observation and forecast data within this box. 3. Equitable Threat Score (ETS) R ave Event if R ave ≥ threshold No-Event if R ave < threshold yesno yesHit False Alarm noMiss Correct negative observation forecast 2. Contingency Table Q: Which fraction of observed yes - events was correctly forecast? (Atger, 2001)

4 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff … Fraction Skill Score (FSS) (Roberts and Lean, 2005) XX XX XX xX X X x 1. Principle: Define box around region of interest and determine the fraction p j and o j of grid points with rain rates above a given threshold. 3. Skill Score for Probabilities 2. Probabilities Q: On which spatial scales does the forecast resemble the observation? FBS worst  no colocation of non-zero fractions 0 threshold

5 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff … Fraction Skill Score (FSS) (Roberts and Lean, 2005) 4. Useful Scales useful scales are marked in bold in the graphics

6 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Fuzzy Verification - Methods Upscaling - ETSFraction Skill Score - FSS Roberts & Lean (2005) Score: FSS for fractions / probabilities (0 = mismatch, 1 = perfect match) Score: ETS (Equitable Threat Score) (-1/3 = mismatch, 1 = perfect match) „Spatial averaging damps out areas with high rain rates and spreads areas of lower rain rates.“ (B.Ebert,2009) „ETS tends to increase with white space (rare events).“ (B.Ebert,2009) „The absolute value of the FSS is less useful than the scale where an acceptable level of skill is reached.“ (Mittermaier,2008) “The score is most sensitive to rare events.” (Roberts, 2008)

7 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff good bad COSMO-7 better COSMO-2 better COSMO-2COSMO-7Difference - = - = Upscaling and Fractions Skill Score Fractions skill score Upscaling

8 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff COSMO-2 vs. COSMO-7, bootstrapping DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better abs(median) / 0.5(q95-q05) [ 10, Inf ] [ 2, 5 [ [ 5, 10 [ [ 1, 2 [ [ 0, 1 [ Values = Score of COSMO-2 Size of numbers = abs(Median) / 0.5(q95-q05)  measure for significance of differences ¦ COSMO-2 – COSMO-7 ¦ q(50%) q(95%) q(5%) 5% 95%

9 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Sensitivities Only 00 and 12 UTC model runs COSMO-2 & COSMO-7: leadtimes 3-6,6-9,9-12,12-15  absolute values of COSMO-2 slightly lower, but still the same pattern of differences DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better

10 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Weather type verification: COSMO # cases Threshold [mm/3h] Upscaling Fractions Skill Score Window size: 3gp, 6.6 kmWindow size: 27gp, 60 km Window size: 3gp, 6.6 km Window size: 27gp, 60 km

dx NE45 ///// N ////// NW // SE11119/// S111119// SW // E91527////45 W113915/// F 27 45/// H27 45//// L /// dx NE15/////// N//////// NW // SE11139/// S111339// SW33599/// E15/////// W33359/// F99 //// H/ ////// L3599 /// COSMO-2, gridpoints* 2.2 kmCOSMO-7, gridpoints* 6.6 km Smallest spatial scale [gridpoints] where the forecast has been useful regarding to FSS „useful scales“ definition # cases % obs gridpts >= thresh (whole period) < <0.1

12 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff % NE <0.1 N <0.1 0 NW <0.1 SE <0.1 S SW E <0.1 W <0.1 F H <0.1 L ALL <0.1 Fraction of observation gridpoints >= threshold  climatology

13 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Frequency of Weather Classes, June – November

14 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Northerly Winds (NE,N) 11 days COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all)  COSMO-2 in northerly wind situations clearly worse than over whole period. DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better DIFFERENCESDIFFERENCES COSMO-2 (all) better COSMO-2 (wc) better

15 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Southerly Winds (SE,S) 12 days COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all)  COSMO-2 in southerly wind situations clearly better than over whole period. DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better DIFFERENCESDIFFERENCES COSMO-2 (all) better COSMO-2 (wc) better

16 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Northwesterly winds (NW) 23 days COSMO-2 vs.COSMO-7 COSMO-2 (wc) vs.COSMO-2 (all)  COSMO-2 in nothwesterly wind situations at large thresholds clearly better than over whole period. DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better DIFFERENCESDIFFERENCES COSMO-2 (all) better COSMO-2 (wc) better

17 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Flat (F) 15 days COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all) DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better DIFFERENCESDIFFERENCES COSMO-2 (all) better COSMO-2 (wc) better  COSMO-2 in flat pressure situations clearly worse than over whole period.

18 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Summary & Outlook Fuzzy verification of the 6 months period in 2007 has shown that COSMO-2 generally performed better than COSMO-7 on nearly all scales part of this superiority is caused by the higher update frequency, but using same model runs still shows the same pattern of differences the results for different weather types show large variations best results were found for southerly winds and winds from Northwest and West, Northeasterly winds as well as Flat pressure situations lead to worse perfomance of both models operational Fuzzy verification is about to start, including Upscaling and Fraction Skill Score