1. © Crown copyright Met Office Verifying modelled currents using a threshold exceedance approach Dr Ray Mahdon An exploration of the Gerrity Skill Score NPE - Cross-cutting research on verification techniques Presentation Session Code: SCI-PS153.03

2. © Crown copyright Met Office Verifying modelled currents using a threshold exceedance approach An exploration of the Gerrity Skill Score Table of Contents Introduction Data Source & Locations Differing Current Regimes Time Series, Continuous Statistics & Simple Cat. Metrics Neighbourhood Methods Bias Removal Questions Multi-Cat. Metric – Gerrity Skill Score & Ocean Currents Threshold Choices

3. © Crown copyright Met Office Introduction Some questions we are trying to answer….. Does the model capture extreme events or “weather-windows”? In which locations or time of year do the models have the best performance; is there a significant difference in regime, time or area? Surface currents forecasts important for commercial or defence “weather-windows” e.g. Current speed below 1kt for 12 hours. e.g. Does not exceed 1kt more than x times Good for site-specific & threshold based analysis

4. © Crown copyright Met Office 26-56N,19W-5E MyOcean - Puertos Del Estado 61198 61281 62024 620103062025 61430 62083 61417 Matxitxako 61280 Donostia 62085 Data Source & Locations Slope Current General Ocean Circulation Eddies Shelf Circulation Wind & Tidal Currents

5. © Crown copyright Met Office Data, Time Series & Continuous Statistics Hourly frequency, Jan 2012 – Jun 2014 (30 months) Collocated model & In-Situ moored observation surface currents Continuous statistics are helpful to describe overall behaviour e.g. q-q & histogram plots describe climatology Timeseries can show seasonal patterns or significant events Do not quantify the performance of a system when exceeding thresholds is of interest We focus on surface currents validation is relatively sparse for this parameter → Categorical Metric Assessment Simple 2x2 (binary) contingency table per chosen threshold

6. © Crown copyright Met Office Neighbourhoods: 1x1, 3x3, 5x5,..,NxN Neighbourhood Sampling T+0 T+1 T-1 Spatial Neighbourhoods Temporal Neighbourhoods Time averaging & shifting Combinations spatial & temporal neighbourhoods trialled

7. © Crown copyright Met Office Simple Categorical Metrics CSI & ETS require un-biased input data Over what period should a tidally dominated field be normalised:– 1 tidal cycle; spring-neap cycle; astronomical cycle? How to handle –ve currents? MISSES HITS F. ALARMS ETS CSI CORR. REJ. Improvements from temporal averaging hour-hour assessment not good as CSI → ETS says model mostly correct by chance!

8. © Crown copyright Met Office Multi-Categorical Metric Method The Gerrity Skill Score

9. © Crown copyright Met Office Gerrity* Skill Score (GSS) Refinement of binary categorical methods Does not depend on the forecast distribution Rewards/penalises for rare(extreme)/disparate events does not reward conservative forecasting Large choice of threshold divisions Good observation (sample) climatology required Contingency table distribution leads to scoring matrix Equitable (i.e., random & constant forecasts score a value of 0) C≤T1 OBS T1<C≤T2C≥T3 C≤T146214 MODEL T1<C≤T2 26555 C≥T3548 0.56-0.45 -0.450.55-0.01 -0.015.08 × GSS=0.38 * Gerrity, J.P., (1992), Monthly Weather Review, 120, 2709-2712.

10. © Crown copyright Met Office GSS - Threshold Choices Equal Frequency Distribution [20,40,60,80] percentiles Skewed Thresholds [0.10,0.25,0.45,0.7] 1 year rolling data per point, captured from 2 ½ years (365 × 24 = 8760 pts. – a good climatology!) Variability in skill versus thresholds, neighbourhood & time Clues in events from time series & data captured

11. © Crown copyright Met Office Mean error = -0.03 ms -1 RMSE = 0.11 ms -1 GSS - Threshold Choices Cont. Equal Frequency Distribution = [0.07, 0.12, 0.18, 0.25] Skewed Thresholds = [0.10, 0.25, 0.45, 0.70] Daily Max/Min Current Speed - 62024

12. © Crown copyright Met Office Mean error = -0.03 ms -1 RMSE = 0.11 ms -1 GSS - Threshold Choices Cont. Equal Frequency Distribution = [0.05, 0.1, 0.15, 0.2] Skewed Thresholds = [0.1, 0.25, 0.45, 0.7] Daily Max/Min Current Speed - 62024

13. © Crown copyright Met Office GSS - Threshold Choices Cont. Equal Frequency DistributionRegular Thresholds Equal Frequency Distribution = [0.07, 0.12, 0.18, 0.25] Regular Thresholds = [0.25, 0.5, 0.75, 1.0] CHECK YOUR ANALYSIS Multi-Category test reduced to 2x2 in many cases ! 1 year’s data captured from 2 ½ years (365 × 24 = 8760 pts. – a good climatology ) 1916 FC 0.25<C<=0.5 6272 FC C<=0.25 OBS 0.25<C<=0.5 OBS C<=0.25 11.52 0.09 × GSS=0.7

14. © Crown copyright Met Office Other trials & results Various spatial & temporal neighbourhoods Report similar results Preliminary results on other model systems show similar skill scores Met Office FOAM-Shelf system Maximum skill versus neighbourhood size Other binning thresholds No firm a priori binning remains a deficiency Decoupling tidal cycle & residual current from raw signal to highlight skill partitioning Doodson sea surface height decoupler trialled Separation of potentially non-parallel (orthogonal) fields not addressed

15. © Crown copyright Met Office Conclusions

16. © Crown copyright Met Office Conclusions Hourly frequency currents, Jan 2012 – Jun 2014 (30 months) Threshold based assessment Continuous statistics are helpful to describe overall behaviour Timeseries can show seasonal patterns Does not quantify spatial or temporally coordinated model/obs values → Categorical Metric Assessment Gerrity Skill Score – attractive attributes for rewards/penalties

17. © Crown copyright Met Office Conclusions cont. Choice of thresholds important Model CAN CAPTURE EXTREME EVENTS – Threshold dependent ! Equal Frequency Distribution appears to be the fairest a priori Can be personalised to a particular regime or current distribution Timeseries needed alongside Gerrity Missing data can skew results Similar locations/regimes appear to give broadly similar Gerrity Skill Scores Winter months tend to show better skill – more extreme events Multi-category methods on surface ocean current speed are relatively new, so expectation of skill level is unknown

18. © Crown copyright Met Office Future Work Now concept established, apply to forecast data Include other regional models which have long-term observation record Bootstrapping Gerrity Skill Score Error estimation around each score Return to bias removal issue Scaled currents, rather than constant removal? Assess wind speed with Gerrity Skill Score & compare to surface currents Potentially highlights efficiency of wind speed transmission to surface currents in Ocean:Atmosphere boundary

19. © Crown copyright Met Office Acknowledgement Thank you to MyOcean for funding towards this work

20. © Crown copyright Met Office THANK YOU FOR YOUR ATTENTION Any Questions (& answers)?