U. Damrath, COSMO GM, Athens 2007 Verification of numerical QPF in DWD using radar data - and some traditional verification results for surface weather elements Ulrich Damrath

U. Damrath, COSMO GM, Athens 2007 Outline Basic data Traditional verification  Diurnal cycles (observation, forecasts of GME, COSMO-EU and COSMO-DE) starting at different forecast times  Contingency tables „Fuzzy“-type verification  Application of B. Eberts package  Coupling intensity scale method and fraction skill score Verification against SYNOP-observations

U. Damrath, COSMO GM, Athens 2007 Basic data Data of GME (resoluton i192f ~ 40 km), COSMO-EU on the grid of COSMO-DE Method of „interpolation“: using the data to the nearest gridpoint of COSMO-DE

U. Damrath, COSMO GM, Athens 2007 Basic data for comparison with RADARs (example with some missing observations)

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 00 UTC July 2007

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 03 UTC July 2007

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 06 UTC July 2007

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 09 UTC July 2007

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 15 UTC July 2007

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 18 UTC July 2007

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 21 UTC July 2007

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 00 UTC August 2007

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 03 UTC August 2007

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 06 UTC August 2007

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 09 UTC August 2007

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 12 UTC August 2007

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 15 UTC August 2007

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 18 UTC August 2007

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 21 UTC August 2007

U. Damrath, COSMO GM, Athens 2007 Traditional point to point verification Scores: FBI, POD, FAR, ETS Thresholds: 0.0, 0.1, 0.5, 1.0, 5.0, 10.0 mm/h

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 00 UTC July Scores precip y/n

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 00 UTC August Scores precip y/n

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 00 UTC July Scores precip > 5mm/h

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 00 UTC August Scores precip > 5mm/h

U. Damrath, COSMO GM, Athens 2007 Surprise? COSMO-DE: FBI=2.5 POD=0.55 FAR~0.8 COSMO-EU: FBI=0.6 POD=0.1 FAR~ after some simple arithmetic manipulation FAR=0.78 FAR=0.83

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 03 UTC July Scores precip y/n

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 06 UTC July Scores precip y/n

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 09 UTC July Scores precip y/n

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 15 UTC July Scores precip y/n

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 18 UTC July Scores precip y/n

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 21 UTC July Scores precip y/n

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 06 UTC July Scores precip > 5mm/h

U. Damrath, COSMO GM, Athens 2007 Diurnal cycle of precipitation for forecasts starting at 18 UTC July Scores precip > 5mm/h

U. Damrath, COSMO GM, Athens 2007 The framework for fuzzy verification (I) 1. Select a set of scales with indeces s=1, 2,…, S and event intensity thresholds with indeces k=1, 2,…, K over which to compute the fuzzy verification results 2. For each scale s: a. collect the gridded forecasts within the window of scale s surrounding the observation. b. For each intensity threshold k compute scale-dependent quantities (,,,, ) k according to various decision models c. For each intensity threshold k compute the desired verification scores over the domain. and event intensity thresholds with indeces k=1, 2,…, K over which to compute the fuzzy verification results. From: Fuzzy Verification of High Resolution Gridded Forecasts: A Review and Proposed Framework E. E. Ebert

U. Damrath, COSMO GM, Athens 2007 The framework for fuzzy verification (II) Observation Forecast

U. Damrath, COSMO GM, Athens 2007 Possible properties of a score  Theoretical properties  score should be illustrative and understandable (by each interested person)  score should give a real information of forecast quality on the different scales  monotonic behaviour concerning (see Lorenz 1963, 1969) scale (best values for large scales) frequency of occurence (best values for high frequencies of occurence)  Murphys properties of scores Bias, Accuracy, Skill, Reliability, Resolution, Sharpness, Discrimination, Uncertainty  Practical properties  agreement between subjective and objective judgement  possible help in decision making

U. Damrath, COSMO GM, Athens yet another score N. Roberts (summing up over all scales) B. Casati (real scale separation) Problem: Smooth fields give best results concerning rmse-related scores!

U. Damrath, COSMO GM, Athens yet another score B. Casati energy squared Skill Score x Modification by U. Damrath in order to have the same structure as the FSS

U. Damrath, COSMO GM, Athens yet another score A couple of B.Casatis‘ and N. Roberts‘ approach: Calculation of a wavelet skill score for each scale, where scale can be got by the wavelet or upscaling method

U. Damrath, COSMO GM, Athens yet another score

U. Damrath, COSMO GM, Athens 2007 „Fuzzy“-type verification for 12 h forecasts (vv=06 till vv=18) starting at 00 UTC June 2007 (fraction skill score)

U. Damrath, COSMO GM, Athens 2007 „Fuzzy“-type verification for 12 h forecasts (vv=06 till vv=18) starting at 00 UTC July 2007 (fraction skill score)

U. Damrath, COSMO GM, Athens 2007 „Fuzzy“-type verification for 12 h forecasts (vv=06 till vv=18) starting at 00 UTC August 2007 (fraction skill score)

U. Damrath, COSMO GM, Athens 2007 „Fuzzy“-type verification for 24 h forecasts (vv=06 till vv=30) starting at 00 UTC June 2007 (fraction skill score)

U. Damrath, COSMO GM, Athens 2007 „Fuzzy“-type verification for 24 h forecasts (vv=06 till vv=30) starting at 00 UTC July 2007 (fraction skill score)

U. Damrath, COSMO GM, Athens 2007 „Fuzzy“-type verification for 24 h forecasts (vv=06 till vv=30) starting at 00 UTC August 2007 (fraction skill score)

U. Damrath, COSMO GM, Athens 2007 „Fuzzy“-type verification for 12 h-forecasts (vv=06 till vv=18) starting at 00 UTC June 2007 (intensity scale skill score)

U. Damrath, COSMO GM, Athens 2007 „Fuzzy“-type verification for 12 h-forecasts (vv=06 till vv=18) starting at 00 UTC July 2007 (intensity scale skill score)

U. Damrath, COSMO GM, Athens 2007 „Fuzzy“-type verification for 12 h-forecasts (vv=06 till vv=18) starting at 00 UTC August 2007 (intensity scale skill score)

U. Damrath, COSMO GM, Athens 2007 „Fuzzy“-type verification for 24 h forecasts (vv=06 till vv=30) starting at 00 UTC June 2007 (intensity scale skill score)

U. Damrath, COSMO GM, Athens 2007 „Fuzzy“-type verification for 24 h forecasts (vv=06 till vv=30) starting at 00 UTC July 2007 (intensity scale skill score)

U. Damrath, COSMO GM, Athens 2007 „Fuzzy“-type verification for 24 h forecasts (vv=06 till vv=30) starting at 00 UTC August 2007 (intensity scale skill score)

U. Damrath, COSMO GM, Athens 2007 „Fuzzy“-type verification for COSMO-DE 12 h forecasts (vv=18 -vv=06) starting at 00 UTC June 2007 (scale dependent skill score)

U. Damrath, COSMO GM, Athens 2007 „Fuzzy“-type verification for 12 h forecasts (vv=18 -vv=06) starting at 00 UTC June 2007 (scale dependent skill score) COSMO-DECOSMO-EU

U. Damrath, COSMO GM, Athens 2007 COSMO-DECOSMO-EU „Fuzzy“-type verification for 12 h forecasts (vv=18 -vv=06) starting at 00 UTC July 2007 (scale dependent skill score)

U. Damrath, COSMO GM, Athens 2007 COSMO-DECOSMO-EU „Fuzzy“-type verification for 12 h forecasts (vv=18 -vv=06) starting at 00 UTC August 2007 (scale dependent skill score)

U. Damrath, COSMO GM, Athens 2007 GMECOSMO-EU „Fuzzy“-type verification for 24 h forecasts (vv=30 -vv=06) starting at 00 UTC June 2007 (scale dependent skill score)

U. Damrath, COSMO GM, Athens 2007 GMECOSMO-EU „Fuzzy“-type verification for 24 h forecasts (vv=30 -vv=06) starting at 00 UTC July 2007 (scale dependent skill score)

U. Damrath, COSMO GM, Athens 2007 GMECOSMO-EU „Fuzzy“-type verification for 24 h forecasts (vv=30 -vv=06) starting at 00 UTC August 2007 (scale dependent skill score)

U. Damrath, COSMO GM, Athens 2007 Some traditional verification results for surface weather elements

U. Damrath, COSMO GM, Athens 2007 Some traditional verification results for surface weather elements -Wind speed February 2007 Start 00 UTCStart 06 UTCStart 12 UTC RMSE BIAS COSMO-EUCOSMO-DE

U. Damrath, COSMO GM, Athens 2007 Some traditional verification results for surface weather elements -Temperature July 2007 Start 00 UTCStart 06 UTCStart 12 UTC RMSE BIAS COSMO-EUCOSMO-DE

U. Damrath, COSMO GM, Athens 2007 Summary (1)Traditional verification shows some problems concerning the diurnal cycle forecasts of COSMO-DE  Latent heat nudging overestimates precipitation values during the first forecast hours  Maximum of precipitation occurs relatively late for start hours during morning Verification using contingency tables  Although point 1 is evident, ETS‘s of COSMO-DE forecasts are better than for COSMO-EU and GME  There seem to be problems with some extreme forecasted values „Fuzzy“-type verification  Fraction skill score shows advantages of COSMO-xx against GME  Fraction skill score shows in most cases advantages of COSMO-DE against COSMO-EU  There is no unique signal concerning intensity scale skill score  The new score demonstrates also advantages of the smaller scale models  Range of predictability is only above 0.2 o ! We are really satisfied with Temperature- and Wind forecasts of COSMO-DE compared to COSMO-EU (at least in the average - an afeternoon?) !

U. Damrath, COSMO GM, Athens after some simple arithmetic manipulation