6th WMO tutorial Verification Martin GöberContinuous 1 Good afternoon! नमस्कार नमस्कार Guten Tag! Buenos dias! до́брый день! до́брыйдень Qwertzuiop asdfghjkl! Bom dia ! Bonjour! Please, verify ! Good afternoon! नमस्कार नमस्कार Guten Tag! Buenos dias! до́брый день! до́брыйдень Qwertyuiop asdfghjkl! Bom dia ! Bonjour!

6th WMO tutorial Verification Martin GöberContinuous 2 Verification of continuous variables Martin Göber Deutscher Wetterdienst (DWD) Hans-Ertel-Centre for Weather Research (HErZ) Acknowledgements: Thanks to Barb Brown and Barbara Casatti!

6th WMO tutorial Verification Martin GöberContinuous 3 Types of forecasts, observations  Continuous  Temperature  Rainfall amount  500 hPa geopotential height  Categorical  Dichotomous Rain vs. no rain Thresholding of continuous variables Strong winds vs. no strong wind Often formulated as Yes/No  Multi-category Cloud amount category Precipitation type YY NY YN NN Except when it is meaningful, forecasts should not be degraded to categorical, due to the resulting loss of information.

6th WMO tutorial Verification Martin GöberContinuous 4 observation o forecast f (961 classes)*(100 stations)*(2 days)*(5 kinds of forecasts) = 1 Million numbers to analyse „curse of dimensionality“ Joint frequency distribution, road surface temperature, winter 2011 The joint probability distribution p(f,o) Boil down to a few numbers  (little ?) loss of information

6th WMO tutorial Verification Martin GöberContinuous 5 5 Continuous verification Normally distributed ERRORS

6th WMO tutorial Verification Martin GöberContinuous 6 If errors are normally distributed, then 2 parameters are enough, to answer all questions approximately If systematic error („bias“) small, then Root(MSE )= Standard error Normally distributed errors

6th WMO tutorial Verification Martin GöberContinuous 7 mean error ME, ideally=0 “systemtic error”  “on average, something goes wrong into one direction”, e.g. model physics wrongly tuned, missing processes, wrong interpretation of guidances tells us nothing about the pairwise match of forecasts and observations large in the past, rather small nowadays on average, but maybe large e.g. for certain weather types misleading for multi-modal error distributions  take Mean Absolute Error MAE Bias

6th WMO tutorial Verification Martin GöberContinuous 8 ME and MAE Q: If the ME is similar to the MAE, performing the bias correction is safe, if MAE >> ME performing the bias correction is dangerous: why ? A: if MAE >>ME it means that positive and negative errors cancel out in the bias evaluation …

6th WMO tutorial Verification Martin GöberContinuous 9 mean squared error or root mean square error RMSE accuracy measure: determines the distance between individual forecasts and observations, Ideally RMSE = 0 “It might be useful on average, but when its really important its not good ! ????” NOT necessarily, e.g: 1 five degree error is penalised like 25 one degree error 1 ten degree error is penalised like 100 one degree errors RMSE

6th WMO tutorial Verification Martin GöberContinuous 10 If errors normally distributed, then Interpretation of RMSE

6th WMO tutorial Verification Martin GöberContinuous 11 Decomposition of the MSE Consequence: smooth forecasts verify better Bias can be subtracted !

6th WMO tutorial Verification Martin GöberContinuous 12 Correlation coefficient  Measures the level of “association” between the forecasts and observations  Related to the “phase error” of the harmonic decomposition of the forecast  Is familiar and relatively easy to interpret  Has a nonparametric analog based on ranks

6th WMO tutorial Verification Martin GöberContinuous 13 Correlation coefficient

6th WMO tutorial Verification Martin GöberContinuous 14 Correlation coefficient

6th WMO tutorial Verification Martin GöberContinuous 15 What is wrong with the correlation coefficient as a measure of performance? Doesn’t take into account biases and amplitude – can inflate performance estimate More appropriate as a measure of “potential” performance Correlation coefficient

6th WMO tutorial Verification Martin GöberContinuous 16 Comparative verification  Generic skill score definition: Where M is the verification measure for the forecasts, M ref is the measure for the reference forecasts, and M perf is the measure for perfect forecasts  Measures percent improvement of the forecast over the reference  Positively oriented (larger is better)  Choice of the standard matters (a lot!)

6th WMO tutorial Verification Martin GöberContinuous 17 Comparative verification Skill scores  A skill score is a measure of relative performance Ex: How much more accurate are my temperature predictions than climatology? How much more accurate are they than the model’s temperature predictions? Provides a comparison to a standard  Standard of comparison can be Chance (easy?) Long-term climatology (more difficult) Sample climatology (difficult) Competitor model / forecast (most difficult) Persistence (hard or easy)

6th WMO tutorial Verification Martin GöberContinuous 18 Reduction of error Variance (also often called „skill score“ SS) Skill scores General skill score definition:

6th WMO tutorial Verification Martin GöberContinuous 19 Reduced variance MSE(Persistence) MSE(forecast) 24h mean wind forecast Higher skill Lower accuracy Accuracy vs skill

6th WMO tutorial Verification Martin GöberContinuous 20 “hits” = percentage of “acceptable” forecast errors (e.g. ICAO - dd:+-30°, ff:+-5kt bis 25kt, etc.) „hits“ and RMSE Forecast error in K “hits” in % “hits”

6th WMO tutorial Verification Martin GöberContinuous 21 Reduction of Error “mass“: Through reduction of large errors „hits“ and RMSE Forecast error in K “hits” in % “hits”

6th WMO tutorial Verification Martin GöberContinuous 22 Maximum temperature Potsdam Every 10 years one day better Long term trends “Hit rate” (errors +- 2k) in %

6th WMO tutorial Verification Martin GöberContinuous 23 Linear Error in Probability Space  LEPS is an MAE evaluated by using the cumulative frequencies of the observation  Errors in the tail of the distribution are penalized less than errors in the centre of the distribution q 0.75

6th WMO tutorial Verification Martin GöberContinuous 24  Verification is a high dimensional problem  can be boiled down to a lower dimensional under certain assumptions or interests  If forecast errors are normally distributed, continuous verification allows usage of only a few numbers like bias and RMSE  Accuracy and skill are different things Summary