1. © Crown copyright Met Office Pass the baton: Verification and the NCOF product chain Andy Saulter, Business Support and Waves

2. © Crown copyright Met Office Why verify? Why pass the baton? Two purposes to scientific verification of our product set.. 1. To prove to ourselves and our peers that the research and development that we do is scientifically valid, and an improvement on the status quo 2. To quantify the risk taken by those who use what we create in their work and decision making

3. © Crown copyright Met Office Why verify? Why pass the baton? 2. To quantify the risk taken by those who use what we create in their work and decision making ‘To face the sea is, to be sure, no light matter when the sea is in its grandest mood. You must the know the sea, and know that you know it, and not forget that it was made to be sailed over’ – Joshua Slocum (1 st solo circumnavigator – 1895-1898) This talk will concentrate on this second use of our verification Developing and communicating the necessary metrics to fulfil this function is not always something we do well However, supplying unbiased statements of risk alongside our products should be a key element both within an NCOF product development chain and to the end user – we must clearly communicate risk when we ‘pass the baton’

4. © Crown copyright Met Office Risk in decision making – breakdown of an end-user decision I want to go surfing Apply cost benefit model High Benefit vs Cost Low requirement for risk information Low Benefit vs Cost Nil requirement for risk information Marginal Scenario High requirement in close analysis of forecast + risk Apply if at marginal thresholds for beach / board Need significant confidence in a positive outcome based on my forecast

5. © Crown copyright Met Office Risk in decision making – mitigation in the field, ‘alpha factor’ Alpha is employed by Det Norske Veritas, a major international classification firm – responsible for certifying and assuring offshore operations worldwide Alpha factors are applied to weather forecasts for defined working thresholds (often significant wave height) in order to assure that only a low percentage of forecasts would ‘miss’ an event where the threshold is exceeded

6. © Crown copyright Met Office Risk in decision making – mitigation in the field, ‘alpha factor’

7. © Crown copyright Met Office Risk in decision making – mitigation in the field, ‘alpha factor’

8. © Crown copyright Met Office Risk in decision making – mitigation in the field, ‘alpha factor’ Alpha is employed by Det Norske Veritas, a major international classification firm – responsible for certifying and assuring offshore operations worldwide Alpha factors are applied to weather forecasts for defined working thresholds (often significant wave height) in order to assure that only a low percentage of forecasts would ‘miss’ an event where the threshold is exceeded The ‘cost’ of a conservative alpha are excessive numbers of false alarms and increased deployment times for construction vessels – often these ships and rigs are hired at rates of £10ks per day

9. © Crown copyright Met Office Verification and the NCOF product chain PROCESS MODEL CASE STUDIES OPERATIONAL MODEL (TESTING) OPERATIONAL MODEL PRODUCTS END USER

10. © Crown copyright Met Office Verification and the NCOF product chain PROCESS MODEL CASE STUDIES OPERATIONAL MODEL (TESTING) OPERATIONAL MODEL PRODUCTS END USER Potential knock on effects for wider model Everyday scenarios may not be covered; domain/run time compromises Performance in new scenarios – how does test period relate to wider climatology? Performance changes versus existing system; under forecast forcing? Aligning performance measure(s) to decision risk

11. © Crown copyright Met Office Verification and the NCOF product chain Some ground rules for ‘passing the baton’.. Assume that familiarity with the science will decrease through the product chain – but that an interested end-user will expect a complete ‘audit trail’ Verification metadata must be clear through all stages Know your ‘customer’ (immediate next step in the chain) and provide information on potential negatives as well as positives Geo-referencing verification information is vital Regions for area based verification must be appropriate

12. © Crown copyright Met Office Geo-referencing verification information ‘Whole domain metrics’ are not very often the most appropriate for real world applications

13. © Crown copyright Met Office Verification and the NCOF product chain Questions to ask ourselves.. What happens next with my work? What are the key issues, parameters, time and space scales that interest the next stage? Do my verification metrics assess risk for (any or all of) these? Are the metrics and their outcomes explained clearly? Are gaps identified?

14. © Crown copyright Met Office Theory into practise – proposed verification for Met Office operational wave forecasting Metrics have an identified purpose

15. © Crown copyright Met Office Theory into practise – proposed verification for Met Office operational wave forecasting Metrics have an identified purpose General Model Performance Statistics Key performance indicators that define the model’s overall performance for broadscale domains and provide a simple baseline for measuring improvement. Model Intercomparison Statistics Agreed indicators from data exchanges that allow headline comparison of different centre’s models Process Study Statistics Indicators of specific model performance characteristics (e.g. swell arrival time, wind- sea growth) Model vs Remote Sensing vs In-Situ Statistics 3-way comparison to test a) model and remote sensing errors vs an in-situ benchmark, b) allow estimates or model error with global coverage. These analyses should provide a pre-cursor to the introduction of global post processing or data assimilation methodologies. Forecaster Guidance Statistics Regional analyses aimed at providing forecasters with ‘rules of thumb’ in interpreting wave model performance for given locations. Customer Focused Verification Regional or site based analyses providing customers with an easily interpreted guide to forecast performance and assessment of risk associated with taking forecast data. These statistics should also provide a comparison between model and human intervened forecast performance.

16. © Crown copyright Met Office Theory into practise – proposed verification for Met Office operational wave forecasting Metrics have an identified purpose Customer focused verification should be provided covering a number of key strategic areas for wave forecasting (e.g. 4 or 5 regions covering the North Sea)

17. © Crown copyright Met Office DNV Alpha metrics – study locations

18. © Crown copyright Met Office Theory into practise – proposed verification for Met Office operational wave forecasting Metrics have an identified purpose Customer focused verification should be provided covering a number of key strategic areas for wave forecasting (e.g. 4 or 5 regions covering the North Sea) A regular update cycle and method of publication must exist in order to communicate the results to those needing the information

19. © Crown copyright Met Office Theory into practise – referencing skill assessment to customer questions Output CategoryUsageCustomer Question Qualitative Climatological Indicates climatological process differences in area, could be used to plan a monitoring programme [Planning] Where should I monitor in order to understand / warn for my area of interest? Quantitative Climatological Provides broadscale quantification of climate variations over an area [Planning] How do regions of my area compare in terms of operability or resource? Qualitative ForecastIndicates events in the broadscale; could be used to describe how events relate to a reference climatology [Response] Should I monitor developments in a certain region of my area more closely in the future? Quantitative ForecastWarns of broadscale events and allows quantification of risk relative to an operating threshold [Response] Can I quantify risks / opportunities for my operations in regions of my area in future? Area based information

20. © Crown copyright Met Office Thank you for listening You are here because you are driving NCOF product development Have a beer, have a chat, get someone else’s perspective.. Think about how best to pass the baton…