COSMO General Meeting 2011 WG5 Parallel Session 5 September 2011 Verification results at MeteoSwiss in the year 2011 Francis Schubiger, Vanessa Stauch, Tanja Weusthoff, Pirmin Kaufmann MeteoSwiss COSMO General Meeting 2011 WG5 Parallel Session 5 September 2011

Content results of the operational verification in 2011 separate verification of all eight daily COSMO-2 forecasts neighborhood verification (also weather type dependant) COSMO-2 03 UTC forecasts up to +48h extension of Beth Ebert‘s package: -> „fuzzy in time“

Verification of COSMO-2, COSMO-7, COSMO-LEPS and IFS Operational (COSMO-2, COSMO-7, COSMO-LEPS [mean], IFS): Surface (3h-steps SYNOP and 1h-steps SMN [Swiss Met Net]) psred, ps, T2m, Td2m, 10m-wind, cloud cover, gusts, precipitation (1h, 12h), global radiation Upper-air (TEMP) T, RH, wind, F Quasi-operational (COSMO-2, COSMO-7, IFS): precipitation with RADAR (neighborhood/fuzzy verification, Ebert’s package) also weather-type dependant Monitoring: SMN (swiss SYNOPs) , RADAR, windprofilers Ongoing in experimental modus: windprofiler over Switzerland fluxes at Payerne New / Next developments: VAD (locations of swiss radars) => All these packages should be replaced by VERSUS 2 in red: new developments in 2010

T2m: mean diurnal cycle (first 24h forecasts) domain Switzerland (hourly SYNOP‘s) Autumn 2010 Spring 2011 OBS COSMO-7 COSMO-2 Winter 2010/2011 Summer 2011 V. Stauch, P. Kaufmann

T2m: mean diurnal cycle Spring 2011 COSMO-7 vs IFS SYNOP‘s over COSMO-7 domain OBS COSMO-7 IFS V. Stauch, P. Kaufmann

T2m: mean diurnal cycle Spring 2011 COSMO-7 vs COSMO-LEPS SYNOP‘s over COSMO-7 domain OBS COSMO-7 COSMO-LEPS V. Stauch, P. Kaufmann

Global radiation (inclined): Spring 2011 COSMO-7 mean diurnal cycle SYNOP‘s over Switzerland The mean temperature of the COSMO-LEPS mean is generally too low resulting in negative biases of between 0 and -1.5K for most European stations, and ~ -2K for Alpine stations. This results from the underestimated daily cycle with a too low temperature maximum. Therefore, the standard deviation of the error for COSMO-LEPS mean is higher when compared to COSMO-7, in particular during the daytime. V. Stauch, P. Kaufmann

Global radiation (horizontal): Spring 2011 COSMO-7 mean diurnal cycle SYNOP‘s over Switzerland The mean temperature of the COSMO-LEPS mean is generally too low resulting in negative biases of between 0 and -1.5K for most European stations, and ~ -2K for Alpine stations. This results from the underestimated daily cycle with a too low temperature maximum. Therefore, the standard deviation of the error for COSMO-LEPS mean is higher when compared to COSMO-7, in particular during the daytime. V. Stauch, P. Kaufmann

Precipitation: Spring 2011 geographical distribution precipitation 12h sums frequency bias (1 mm/12h) Spring 2011 COSMO-7 +24 & +48h V. Stauch, P. Kaufmann

Precipitation (12h-sums +12 to +24h): Spring 2011 over Switzerland (SYNOP‘s) frequency bias: COSMO-7 & COSMO-2 observed frequency Comparison of the frequency bias (FBI) for COSMO-2 and COSMO-7 over Switzerland as a function of thresholds for the lead time range 13-24h. The observed frequency (OF) is plotted underneath. (lines between the data points are added for better overview and are not meant to be meaningful interpolations) V. Stauch

Precipitation (12h-sums +36 to +48h): Spring 2011 over Switzerland (SYNOP‘s) frequency bias: COSMO-7 & IFS observed frequency Comparison of the frequency bias (FBI) for COSMO-2 and COSMO-7 over Switzerland as a function of thresholds for the lead time range 13-24h. The observed frequency (OF) is plotted underneath. (lines between the data points are added for better overview and are not meant to be meaningful interpolations) V. Stauch

Precipitation (12h-sums +12 to +24h): Spring 2011 over Switzerland (SYNOP‘s) COSMO-7 & COSMO-2 for both models mean over 9 gridpoints for each station V. Stauch

Precipitation (12h-sums): Spring 2011 over Switzerland (SYNOP‘s) COSMO-7 & IFS V. Stauch

Hourly precipitation: diurnal cycle Swiss SYNOPs (1h acc) CH-radarcomposit (3h acc) SMN COSMO-7 COSMO-7 July 2010 COSMO-2 COSMO-2 RADAR fcst: +12..+24h fcst: +3..+6h 03 09 15 21 00 August 2010 RADAR COSMO-7 COSMO-2 SMN 03 09 15 21 00 fcst: +12..+24h fcst: +3..+6h V. Stauch

Verification of surface weather parameters 2m-temperature: cold bias in winter and warm bias in summer positive bias more pronounced during nighttime COSMO-2 ~0.5 K warmer (higher values) than COSMO-7 2m-dewpoint: negative bias (~1 K) in the summer and autumn period 10m-windspeed: negative bias along the coast and on mountains, positive bias inland total cloudiness: mean daily cycle not well represented (mainly overestimation during night and underestimation during day) precipitation: overestimation (except in Summer ~ no bias) most pronounced in Winter (over COSMO-2 domain: 30-35% ; over Swizerland ~45%) higher amounts in COSMO-2 / higher amounts over Alpine area low amounts (0.1 mm/12h): overestimated in Winter (over the Alps all seasons) high amounts (10 mm/12h): underestimated in COSMO-7 (over full domain by ~30% in summer, 20% in winter)

TEMPS verification: Windspeed +24h Winter 10/11 all TEMPs COSMO-2 domain COSMO-7 COSMO-2

Verification with vertical profiles: main results temperature: cold bias (~ 0.5 K) in winter and warm bias (~ 0.5 K) in summer from ground up to 300 hPa windspeed: positive bias in PBL (up to +1 m/s in winter) and slight negative bias above 400 hPa

Separate verification of the eight daily COSMO-2 forecasts questions of interest: which is the best forecast for the afternoon/evening convection ? – is it the latest forecast or perhaps a forecast with a longer „spin up“ time ? are there significant differences between the eight forecasts ? period: COSMO-2 forecasts from 1 May to 31 July 2011 Verification with hourly obs over Switzerland

COSMO-2: precipitation diurnal cycle forecasts of 00, 03, 06 and 09 UTC Vanessa Stauch

COSMO-2: precipitation diurnal cycle forecasts of 12, 15, 18 and 21 UTC Vanessa Stauch

COSMO-2: precipitation diurnal cycle all eight daily forecasts Vanessa Stauch

COSMO-2: 2m-temperature diurnal cycle all eight daily forecasts Vanessa Stauch

COSMO-2: 2m-temperature bias all eight daily forecasts Vanessa Stauch

COSMO-2: 10m-wind speed diurnal cycle all eight daily forecasts Vanessa Stauch

Neighborhood (fuzzy) verification for precipitation verification on lower resolution: avoid the „double penalty problem“ method Original data “fuzzyfication” score example Upscaling Average Equitable threat score Fractions Skill Score (Roberts and Lean, 2005) Fractional coverage Skill score with reference to worst forecast X x X x T. Weusthoff

How to read „fuzzy grafics“ best skill for low precipitation on large scales numbers / colours  value of the score, here: FSS bad good increasing window size numbers in bold  useful scales (only for FSS) low skill for high precipitation on small scales Increasing threshold T. Weusthoff, MeteoSwiss

Neighborhood verification for precipitation results for 2010 3h accumulated precipitation sums over the domain of the swiss radar composit models: COSMO-2 and COSMO-7 for all 8 daily forecast runs, precipitation sums from +3 to +6h observation precipitation estimates of the swiss radar composit in case of a missing value, the full date will not be evaluated

- = - = Fractions Skill Score Upscaling Neighbourhood Verification, January–December 2010 3h sums (+3 ..+6h) Fractions Skill Score (top) and Upscaling (bottom) Fractions Skill Score - = COSMO-2 COSMO-7 COSMO-2 - COSMO-7 Upscaling - = Zunehmender Skill mit räumlicher Skala, abnehmender Skill mit zunehmendem Threshold… COSMO-2 besser als COSMO-7 auf nahezu allen Skalen T. Weusthoff good bad COSMO-7 better COSMO-2 better

FSS, COSMO-2 minus COSMO-7, 2010 Winter Spring COSMO-2 better COSMO-7 better numbers = FSS-Score of COSMO-2 colours = differences COSMO-2 minus COSMO-7 COSMO-2: similar skill in all seasons COSMO-2 better than COSMO-7 on almost all scales COSMO-2 better than COSMO-7 especially in Winter and Summer Summer Autumn Tanja Weusthoff

Achtung: FSS, COSMO-2 minus COSMO-7, 2009 Winter (DJF) Frühling (MAM) COSMO-2 better COSMO-7 better Zahlen = FSS-Score von COSMO-2 Farben = Differenzen COSMO-2 minus COSMO-7 COSMO-2: vergleichbaren Skill in allen Saisons für kleine Schwellenwerte hohe Schwellenwerte (ge 5mm/3h) besserer Skill in Frühling / Sommer Frühling und Sommer: COSMO-2 grösster Vorteil im Vergleich zu COSMO-7 Herbst und Winter: COSMO-2 leicht besser auf nahezu allen Skalen Sommer (JJA) Herbst (SON) Tanja Weusthoff

Dependency of the lead time +3 bis +6h +12 bis +15h +21 bis +24h Fractions Skill Score Upscaling - ETS Tanja Weusthoff COSMO-7 better COSMO-2 better

YEAR 2010 NE (11x) S (10x) COSMO-7 better COSMO-2 better differences in Fractions Skill Score for weather-type dependant verif COSMO-2 minus COSMO-7 F (78x) N (18x) SW (49x) H (73x) E (4x) NW (38x) W (56x) SE (4x) L (25x)

Summary neighbourhood Verifikation precipitation in 2010 COSMO-2 better than COSMO-7 on all scales, differences become less with increasing leadtime good forecast of the spatial structure on higher scales The skill of the models varies for different weather types and the differences between COSMO-2 and COSMO-7 varies also: - best skill: Autumn and Spring, south to northwest weather types - greatest difference COSMO-2 minus COSMO-7: Summer and Winter, north- and east types, convective cases Tanja Weusthoff

COSMO-2 03 UTC up to +48h: Comparison COSMO-2 to COSMO-7 period (full year): 1 June 2010 – 31 May 2011 Main interest: advance of COSMO-2 as compared to COSMO-7 question: “are the COSMO-2 forecasts at +48h still better than those of COSMO-7” ? Verification with: SYNOP over Switzerland (hourly): comparison +0 to +24h vs +24 to +48h Swiss RadarComposit: neighborhood Verifikation for four 12h-Summen (0..+12, +12..+24, +24..+36, +36..+48h) TEMPs for +21h and +45h

Verification SYNOP-CH: 2m-temperature COSMO-2 vs COSMO-7 BIAS VAL advance of COSMO-2 also in the second day MAE STDE V. Stauch

Verification SYNOP-CH: precipitation COSMO-2 vs COSMO-7 [+0 to +24h] 0.1 mm/h 1.0 mm/h 10 mm/h Frequency bias (mm/h) COSMO-2 COSMO-7 V. Stauch

Verification SYNOP-CH: precipitation COSMO-2 vs COSMO-7 [+24 to +48h] 0.1 mm/h 1.0 mm/h 10 mm/h Frequency bias (mm/h) COSMO-2 COSMO-7 V. Stauch

Verification with swiss radarcomposit daytime: 03-15 UTC nighttime: 15-03 UTC +0.. +12h first forecast day +12..+24h COSMO-2 COSMO-7 COSMO-2 COSMO-7 +24..+36h second forecast day +36..+48h Overerstimation of COSMO-2 COSMO-2 COSMO-7 COSMO-2 COSMO-7 T. Weusthoff

FRACTIONS SKILL SCORE: COSMO-2 – COSMO-7 precipitation sums of 12h COSMO-2 better COSMO-7 better + 24 .. + 36h + 36 .. + 48h T. Weusthoff

UPSCALING: COSMO-2 – COSMO-7 precipitation sums of 12h; score: ETS (equitable threat score) COSMO-2 better COSMO-7 better + 24 .. + 36h + 36 .. + 48h T. Weusthoff

UPSCALING: COSMO-2 – COSMO-7 precipitation sums of 12h; score: frequency bias (absolute) T. Weusthoff

UPSCALING: COSMO-2 – COSMO-7 precipitation sums of 12h; score: frequency bias (relative) T. Weusthoff

3. „Fuzzy in Time“ Extension of the spatial window with a window in time  volume (dx * dy * dt) Evaluation of the forecasts in this volume Time-window ntm = [1h,3h,5h,7h,9h] dx dy dt T. Weusthoff

model dx dy observation dx dy T. Weusthoff t0+1 t0 e.g. FSS fraction blue pxiels model = 12/75 (dt=3), 5/25 (dt=1) fraction blue pxiels obs = 12/75 (dt=3), 3/25 (dt=1) t0-1 observation dx dy t0+1 t0 T. Weusthoff t0-1

hourly accumulated precipitation 20 21 22 23 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 … 00-02 01-03 3 hourly accumulated precipitation 02-04 20-22 03-05 21-23 04-06 22-00 05-07 23-01 Evaluation of 3h accumulations, time window ntm = 1 09 UTC 00 UTC 03 UTC 06 UTC 23 UTC 01 UTC 02 UTC 04 UTC Evaluation of 3h accumulations, time window ntm = 5 (t0 +-2h) 00 UTC 03 UTC 06 UTC 08 UTC 22 UTC 02 UTC 01 UTC 05 UTC T. Weusthoff

FSS for different time-windows COSMO-2, July 2010 T. Weusthoff

FSS for different time-windows COSMO-7, July 2010 T. Weusthoff

FSS for different time-windows COSMO-2 minus COSMO-7, July 2010 T. Weusthoff

Summary „fuzzy in time“ FSS increases on all scales with increasing time-window greatest effect for small spatical scales lowest effect for high threshods Both models show a similar increase difference COSMO-2 minus COSMO-7 stays equal, resp. becomes littler for high time-windows For Upscaling the influence of a time tolerance is relatively low und restricted on low thresholds ( effect of the avergaing)  Application of time-windows on the gridsclae would make sense; simultaneous application with space tolerance brings no great change T. Weusthoff

Precipitation [mm/12h]: frequency bias [in %] (1) threshold mod/domain 10S3 10S4 11S1 11S2 11S3 7_all 98 106 128 113 90 7_alps 112 125 157 143 0.1 mm 7_CH 124 145 168 169 2_alps 107 119 135 2_CH 120 136 148 150 85 102 93 87 126 149 122 1.0 mm 137 100 144 130 111 142 156 140

Precipitation [mm/12h]: frequency bias [in %] (2) threshold mod/domain 10S3 10S4 11S1 11S2 11S3 7_all 67 85 95 70 63 7_alps 115 122 100 5 mm 7_CH 91 138 141 2_alps 94 118 127 114 2_CH 103 143 135 123 62 80 72 55 57 93 10 mm 86 142 87 119 110 102 147 144 105 Compared to the previous year: differences > 15% only during Winter 09/10 for threshold 5 and 10 mm/12h over CH-domain for COSMO-7 and alps + CH-domain for COSMO-2 with > 15% less precipitation (especially over CH-domain for 10 mm/12h)

Precipitation [mm/12h]: frequency bias [in %] (3) threshold mod/domain 10S3 10S4 11S1 11S2 11S3 7_all 43 55 31 30 28 7_alps 54 67 34 38 30 mm 7_CH 78 86 (7) 35 2_alps 118 91 74 95 2_CH 130 108 (35) 85 87 113 131 107 97 104 120 136 124 146 147 126 Summary: mean bias

4. Verifikation anderer Modellparameter  VERA verfügbare VERA Parameter (nur für 2007): 10 m wind (u,v), pot. temperature, equiv. pot. temperature, mixing ratio, moisture flux convergence

„Mixing ratio“ im Vergleich zu VERA Analyse konsequent unterschätzt „Neighbourhood“ Verifikation, COSMO-2, stündlich ab Leadtime +3h, Jun-Aug 2007 (oben) und Sep – Nov 2007 (unten) mixing ratio [g/kg] „Mixing ratio“ im Vergleich zu VERA Analyse konsequent unterschätzt zeitliche und räumliche Struktur recht gut erfasst

4. Verifikation anderer Modellparameter  CM SAF (Globalstrahlung) Globalstrahlung [W/m^2] Differenz Globalstrahlung [W/m^2] (model-obs) Verifikation der Globalstrahlung aus COSMO-2 gegen CM-SAF (Bsp. 1 Juli 2010 12 UTC)

„Neighbourhood“ Verifikation, COSMO-2, stündlich ab Leadtime +3h Verifikation der Globalstrahlung aus COSMO-2 gegen CM-SAF (Bsp. 1 Juli 2010 12 UTC)

Zusammenfassung (IV) Anwendung der „Neighbourhood“ Verifikation auf andere Parameter prinzipiell möglich vor allem für räumlich stark variable Daten weitere Untersuchungen bzw. Anpassungen nötig, auch bezüglich der Beobachtungsdaten z.B. Wolkenfilter für Strahlungsdaten