1. Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz Weather type dependant fuzzy verification of precipitation COSMO General Meeting, Offenbach, 07.-11.09.2009 Tanja Weusthoff

2. 2 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Fuzzy Verification „multi-scale, multi-intensity approach“ „Fuzzy verification toolbox“ of B. Ebert Two methods Upscaling (UP) Fraction Skill Score (FSS) present output scale dependent standard setting: 3h accumulations, Jun-Nov 2007, vs Swiss radar data COSMO-2 (2.2km): leadtimes 03-06 COSMO-7 (6.6km): leadtimes 03-06,06-09,09-12,12-15 fcstobs increasing box size increasing threshold

3. 3 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff … Upscaling (UP) 1. Principle: Define box around region of interest and calculate the average of observation and forecast data within this box. 3. Equitable Threat Score (ETS) R ave Event if R ave ≥ threshold No-Event if R ave < threshold yesno yesHit False Alarm noMiss Correct negative observation forecast 2. Contingency Table Q: Which fraction of observed yes - events was correctly forecast? (Atger, 2001)

4. 4 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff … Fraction Skill Score (FSS) (Roberts and Lean, 2005) XX XX XX xX X X x 1. Principle: Define box around region of interest and determine the fraction p j and o j of grid points with rain rates above a given threshold. 3. Skill Score for Probabilities 2. Probabilities Q: On which spatial scales does the forecast resemble the observation? FBS worst  no colocation of non-zero fractions 0 threshold

5. 5 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff … Fraction Skill Score (FSS) (Roberts and Lean, 2005) 4. Useful Scales useful scales are marked in bold in the graphics

6. 6 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Fuzzy Verification - Methods Upscaling - ETSFraction Skill Score - FSS Roberts & Lean (2005) Score: FSS for fractions / probabilities (0 = mismatch, 1 = perfect match) Score: ETS (Equitable Threat Score) (-1/3 = mismatch, 1 = perfect match) „Spatial averaging damps out areas with high rain rates and spreads areas of lower rain rates.“ (B.Ebert,2009) „ETS tends to increase with white space (rare events).“ (B.Ebert,2009) „The absolute value of the FSS is less useful than the scale where an acceptable level of skill is reached.“ (Mittermaier,2008) “The score is most sensitive to rare events.” (Roberts, 2008)

7. 7 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff good bad COSMO-7 better COSMO-2 better COSMO-2COSMO-7Difference - = - = Upscaling and Fractions Skill Score Fractions skill score Upscaling

8. 8 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff COSMO-2 vs. COSMO-7, bootstrapping DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better abs(median) / 0.5(q95-q05) [ 10, Inf ] [ 2, 5 [ [ 5, 10 [ [ 1, 2 [ [ 0, 1 [ Values = Score of COSMO-2 Size of numbers = abs(Median) / 0.5(q95-q05)  measure for significance of differences ¦ COSMO-2 – COSMO-7 ¦ q(50%) q(95%) q(5%) 5% 95%

9. 9 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Sensitivities Only 00 and 12 UTC model runs COSMO-2 & COSMO-7: leadtimes 3-6,6-9,9-12,12-15  absolute values of COSMO-2 slightly lower, but still the same pattern of differences DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better

10. 10 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Weather type verification: COSMO-2 4755745312315 347 # cases Threshold [mm/3h] Upscaling Fractions Skill Score Window size: 3gp, 6.6 kmWindow size: 27gp, 60 km Window size: 3gp, 6.6 km Window size: 27gp, 60 km

11. dx0.10.20.51.02.05.01020 NE45 ///// N ////// NW339927 // SE11119/// S111119// SW1139945// E91527////45 W113915/// F 27 45/// H27 45//// L39152745/// dx0.10.20.51.02.05.01020 NE15/////// N//////// NW3355915// SE11139/// S111339// SW33599/// E15/////// W33359/// F99 //// H/ ////// L3599 /// COSMO-2, gridpoints* 2.2 kmCOSMO-7, gridpoints* 6.6 km Smallest spatial scale [gridpoints] where the forecast has been useful regarding to FSS „useful scales“ definition. + - + - 7 4 23 5 7 45 5 31 15 34 7 # cases % obs gridpts >= thresh (whole period) 1614107520.5<0.11614107520.4<0.1

12. 12 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff %0.10.20.51.02.05.01020 NE764210.3<0.1 N210.80.40.1<0.1 0 NW13118520.70.1<0.1 SE1715129620.4<0.1 S21191511830.70.1 SW252218139310.2 E3210.70.2<0.1 W1815118410.2<0.1 F1715118520.80.2 H220.80.50.20.1<0.1 L423627191130.80.1 ALL1614107520.5<0.1 Fraction of observation gridpoints >= threshold  climatology

13. 13 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Frequency of Weather Classes, June – November 2007 4 7 55 7 7 45 31 23 15 34

14. 14 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Northerly Winds (NE,N) 11 days COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all)  COSMO-2 in northerly wind situations clearly worse than over whole period. DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better DIFFERENCESDIFFERENCES COSMO-2 (all) better COSMO-2 (wc) better

15. 15 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Southerly Winds (SE,S) 12 days COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all)  COSMO-2 in southerly wind situations clearly better than over whole period. DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better DIFFERENCESDIFFERENCES COSMO-2 (all) better COSMO-2 (wc) better

16. 16 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Northwesterly winds (NW) 23 days COSMO-2 vs.COSMO-7 COSMO-2 (wc) vs.COSMO-2 (all)  COSMO-2 in nothwesterly wind situations at large thresholds clearly better than over whole period. DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better DIFFERENCESDIFFERENCES COSMO-2 (all) better COSMO-2 (wc) better

17. 17 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Flat (F) 15 days COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all) DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better DIFFERENCESDIFFERENCES COSMO-2 (all) better COSMO-2 (wc) better  COSMO-2 in flat pressure situations clearly worse than over whole period.

18. 18 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Summary & Outlook Fuzzy verification of the 6 months period in 2007 has shown that COSMO-2 generally performed better than COSMO-7 on nearly all scales part of this superiority is caused by the higher update frequency, but using same model runs still shows the same pattern of differences the results for different weather types show large variations best results were found for southerly winds and winds from Northwest and West, Northeasterly winds as well as Flat pressure situations lead to worse perfomance of both models operational Fuzzy verification is about to start, including Upscaling and Fraction Skill Score