1. © Crown copyright Met Office Probabilistic turbulence forecasts from ensemble models and verification Philip Gill and Piers Buchanan NCAR Aviation Turbulence Workshop, Boulder, 28 August 2013

2. © Crown copyright Met Office Contents This presentation covers the following areas 1. Introduction 2. Turbulence forecasting 3. Ensemble turbulence trial 4. Current status 5. Summary and future work

3. © Crown copyright Met Office 1. Introduction Turbulence - major cause of aviation incidents & active area of research Forecasts routinely produced by UK Met Office - World Area Forecast Centre (WAFC) service (along with WAFC Washington, USA) Operational forecasts currently derived from deterministic models There is always a degree of uncertainty in deterministic forecasts Ensembles are a way of communicating that uncertainty Photos © P Gill

4. © Crown copyright Met Office At the time of the trial (Nov 2010 – Oct 2011), MOGREPS-G:  60km, 70 Levels T+72h  Run at 00Z, 12Z with 24 members  ETKF for initial condition perturbations  Stochastic physics (SKEB2)  N.B MOGREPS-15 run to 15 days at 00z and 12Z (at ECMWF) MOGREPS-G Upgrade (January 2013)  Same as above but at a resolution of approx. 33 km in mid latitudes and keeping 70 levels  12 member forecasts run every 6 hours i.e. 00Z, 06Z, 12Z and 18Z. MOGREPS-G Met Office Global and Regional Ensemble Prediction System Operational from Sep 2008 after 3 years of trials

5. © Crown copyright Met Office MOGREPS-G “postage stamp” plots

6. © Crown copyright Met Office Ensemble Turbulence Forecasts Deterministic turbulence forecast shows the value of a given turbulence indicator Ensemble turbulence forecast shows the probability of a turbulence indicator exceeding certain chosen values Probabilistic Deterministic

7. © Crown copyright Met Office 2. Turbulence forecasting

8. © Crown copyright Met Office Turbulence predictors Windshear related: Ellrod TI1, Ellrod TI2 Brown Dutton Lunnon Convection related: Convective rainfall rate Convective rainfall accumulation Both wind shear and convection: Richardson number Turbulence climatology Gridded field of observed turbulence frequency produced from aircraft observations from previous year Light or greater and moderate or greater turbulence climatology produced Turbulence can come from different sources – wind shear, convection, (mountain-wave)

9. © Crown copyright Met Office Combining predictors Combining turbulence predictors has been shown to increase forecast skill (Sharman et al, 2006) We use weights derived from verification using ROC area Predictors combined using a weighted sum Combined probabilistic predictors

10. © Crown copyright Met Office 3. Ensemble turbulence trial

11. © Crown copyright Met Office Ensemble turbulence trial Objective verification of deterministic and probabilistic model forecasts Global verification to assess T+24h MOGREPS-G forecasts of turbulence T+24 chosen because this is a typical product time range Verification against automated aircraft observations from the Global Aircraft Data Set (GADS) 12-month trial from November 2010-October 2011 Eight numerical predictors and climatology verified Five thresholds used on each predictor to generate probability forecasts Thresholds designed to cover light to moderate to severe turbulence

12. Turbulence indicator thresholds

13. © Crown copyright Met Office Aircraft observations Global coverage, but flights mainly over northern hemisphere Automated aircraft observations available every 4 seconds 10-19 January 2009 Good coverage of N Atlantic, US and Europe Poor coverage of E Asia/Pacific region Derived Equivalent Vertical Gust (DEVG) – Measurement of observed turbulence derived from vertical acceleration, aircraft mass, altitude and airspeed

14. © Crown copyright Met Office Verification methodology Aircraft track within +/- 1.5h of validity time Turbulent event Turbulence forecast field Ellrod TI1

15. © Crown copyright Met Office Forecast assessment Turbulent/non turbulent event defined on 10min aircraft track ~120km - approx grid size of WAFC grid Deterministic: Forecast turbulent event – CAT potential >= Threshold Ensemble: Probability of exceeding a certain threshold for given turbulence indicator Observed (moderate or greater) turbulent event - DEVG>=4.5m/s Construct 2x2 contingency tables for each threshold Sum entries in contingency tables over the verification period Turbulence observed No turbulence observed Turbulence forecast HitFalse alarm No turbulence forecast MissCorrect rejection 2x2 contingency table

16. © Crown copyright Met Office Verification measures Relative Operating Characteristic (ROC) curve by plotting the hit rate against false alarm rate for each threshold. The area under the ROC curve is a measure of skill. Useful for both deterministic and probabilistic forecasts Reliability Diagram by plotting the forecast probability against the frequency of occurrence Relative economic value (Richardson, 2000) by calculating the value for a range of cost/loss ratios. Useful for both deterministic and probabilistic forecasts.

17. © Crown copyright Met Office Ensemble turbulence verification - ROC Gill PG, Buchanan P. 2013. “An ensemble based turbulence forecasting system”, Meteorological Applications Greater skill combining probabilistic forecasts Perfect forecast No skill

18. © Crown copyright Met Office ROC area for each predictor Gill PG, Buchanan P. 2013. “An ensemble based turbulence forecasting system”, Meteorological Applications Latest combined AUC = 0.77

19. © Crown copyright Met Office Ensemble turbulence verification - Reliability Gill PG, Buchanan P. 2013. “An ensemble based turbulence forecasting system”, Meteorological Applications Low probabilities but significant compared to background frequency Perfect reliability

20. © Crown copyright Met Office Ensemble turbulence verification – Relative economic value Gill PG, Buchanan P. 2013. “An ensemble based turbulence forecasting system”, Meteorological Applications Greater value combining probabilistic forecasts

21. © Crown copyright Met Office 4. Operational production Piers Buchanan and Lisa Murray

22. © Crown copyright Met Office Current status Project underway to produce probabilistic forecasts within an operational setting ready for implementation – completion in Mar 2014 Verification complete for November 2010 to Feb 2013 – analysis ongoing. Studies into using logistic regression to combine predictors using regional and seasonal weightings Work on sourcing additional observations continues.

23. © Crown copyright Met Office Logistic regression Study to compare methods for training algorithms to produce the most skilful forecasts Current method is not scalable to a large number of predictors Logistic regression provides an alternative method and initial studies demonstrate it can produce a forecast with higher value.

24. © Crown copyright Met Office 5. Summary and future work

25. © Crown copyright Met Office Summary Benefits of using Ensemble turbulence forecasts Significant increase in skill Increased economic value of forecast Confidence can be communicated with every forecast Use of verification can help users to maximise the value of the forecast

26. © Crown copyright Met Office Future plans and challenges Create operational probabilistic aviation hazard forecasts for Turbulence, Cb and Icing Include additional predictors (CAPE, Mountain wave) Investigate using a multi-model ensemble for WAFC turbulence forecasts (in collaboration with WAFC Washington) Educating users in interpretation of probabilistic forecasts and verification Photo © P Gill

27. © Crown copyright Met Office Acknowledgements Thanks to the UK Civil Aviation Authority for funding this project The content of this presentation is available as a paper Gill PG, Buchanan P. 2013. “An ensemble based turbulence forecasting system”, Meteorological Applications Other references from this talk: Richardson D.,2000. Q. J. R. Meteorolog. Soc. 126 649-667 Sharman R, Tebaldi C, Wiener G, Wolff J, 2006. Weather Forecast. 21 268-287

28. © Crown copyright Met Office Questions & answers