EMS 2013 (Reading UK) Verification techniques for high resolution NWP precipitation forecasts Emiel van der Plas Kees Kok Maurice.

Slides:

Advertisements

Similar presentations

Slide 1ECMWF forecast User Meeting -- Reading, June 2006 Verification of weather parameters Anna Ghelli, ECMWF.

Advertisements

Slide 1ECMWF forecast products users meeting – Reading, June 2005 Verification of weather parameters Anna Ghelli, ECMWF.

5 th International Conference of Mesoscale Meteor. And Typhoons, Boulder, CO 31 October 2006 National Scale Probabilistic Storm Forecasting for Aviation.

Report of the Q2 Short Range QPF Discussion Group Jon Ahlquist Curtis Marshall John McGinley - lead Dan Petersen D. J. Seo Jean Vieux.

Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Improving COSMO-LEPS forecasts of extreme events with.

Comparison of Several Methods for Probabilistic Forecasting of Locally-Heavy Rainfall in the 0-3 Hour Timeframe Z. Sokol 1, D. Kitzmiller 2, S. Guan 2.

Improving Excessive Rainfall Forecasts at HPC by using the “Neighborhood - Spatial Density“ Approach to High Res Models Michael Eckert, David Novak, and.

The use of the NWCSAF High Resolution Wind product in the mesoscale AROME model at the Hungarian Meteorological Service Máté Mile, Mária Putsay and Márta.

NWP Verification with Shape- matching Algorithms: Hydrologic Applications and Extension to Ensembles Barbara Brown 1, Edward Tollerud 2, Tara Jensen 1,

Institut für Physik der Atmosphäre Predictability of precipitation determined by convection-permitting ensemble modeling Christian Keil and George C.Craig.

EMS ECAM 13 september 2011 GlamEps: Current and future use in operational forecasting at KNMI Adrie Huiskamp.

Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Quantitative precipitation forecasts in the Alps – first.

SREF in Hirlam Sander Tijm (Kees Kok, Ben Wichers Schreur, Jeanette Onvlee, Hilde Haakenstad, Jose Antonio Garcia-Moya)

Exploring the Use of Object- Oriented Verification at the Hydrometeorological Prediction Center Faye E. Barthold 1,2, Keith F. Brill 1, and David R. Novak.

1 Intercomparison of low visibility prediction methods COST-722 (WG-i) Frédéric Atger & Thierry Bergot (Météo-France)

Chapter 13 – Weather Analysis and Forecasting. The National Weather Service The National Weather Service (NWS) is responsible for forecasts several times.

1 Localized Aviation Model Output Statistics Program (LAMP): Improvements to convective forecasts in response to user feedback Judy E. Ghirardelli National.

COSMO General Meeting Zurich, 2005 Institute of Meteorology and Water Management Warsaw, Poland- 1 - Verification of the LM at IMGW Katarzyna Starosta,

Creating Empirical Models Constructing a Simple Correlation and Regression-based Forecast Model Christopher Oludhe, Department of Meteorology, University.

WWOSC 2014 Assimilation of 3D radar reflectivity with an Ensemble Kalman Filter on a convection-permitting scale WWOSC 2014 Theresa Bick 1,2,* Silke Trömel.

Probabilistic forecasts of (severe) thunderstorms for the purpose of issuing a weather alarm Maurice Schmeits, Kees Kok, Daan Vogelezang and Rudolf van.

1 GOES-R AWG Hydrology Algorithm Team: Rainfall Probability June 14, 2011 Presented By: Bob Kuligowski NOAA/NESDIS/STAR.

Page 1© Crown copyright 2007SRNWP 8-11 October 2007, Dubrovnik SRNWP – Revised Verification Proposal Clive Wilson Presented by Terry Davies at SRNWP Meeting.

How can LAMEPS * help you to make a better forecast for extreme weather Henrik Feddersen, DMI * LAMEPS =Limited-Area Model Ensemble Prediction.

“High resolution ensemble analysis: linking correlations and spread to physical processes ” S. Dey, R. Plant, N. Roberts and S. Migliorini Mesoscale group.

Towards an object-oriented assessment of high resolution precipitation forecasts Janice L. Bytheway CIRA Council and Fellows Meeting May 6, 2015.

Guidance on Intensity Guidance Kieran Bhatia, David Nolan, Mark DeMaria, Andrea Schumacher IHC Presentation This project is supported by the.

© Crown copyright Met Office Preliminary results using the Fractions Skill Score: SP2005 and fake cases Marion Mittermaier and Nigel Roberts.

On the spatial verification of FROST-2014 precipitation forecast fields Anatoly Muraviev (1), Anastasia Bundel (1), Dmitry Kiktev (1), Nikolay Bocharnikov.

29th EWGLAM Meeting HIRLAM-A Verification Xiaohua Yang with contributions from Kees Kok, Sami Niemela, Sander Tijm, Bent Sass, Niels W. Nilsen, Flemming.

Improving Ensemble QPF in NMC Dr. Dai Kan National Meteorological Center of China (NMC) International Training Course for Weather Forecasters 11/1, 2012,

“New tools for the evaluation of convective scale ensemble systems” Seonaid Dey Supervisors: Bob Plant, Nigel Roberts and Stefano Migliorini.

Latest results in verification over Poland Katarzyna Starosta, Joanna Linkowska Institute of Meteorology and Water Management, Warsaw 9th COSMO General.

The latest results of verification over Poland Katarzyna Starosta Joanna Linkowska COSMO General Meeting, Cracow September 2008 Institute of Meteorology.

Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Priority project « Advanced interpretation and verification.

Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Accounting for Change: Local wind forecasts from the high-

Short-Range Ensemble Prediction System at INM José A. García-Moya SMNT – INM 27th EWGLAM & 12th SRNWP Meetings Ljubljana, October 2005.

Page 1© Crown copyright Scale selective verification of precipitation forecasts Nigel Roberts and Humphrey Lean.

DIAMET meeting 7 th-8th March 2011 “New tools for the evaluation of convective scale ensemble systems” Seonaid Dey Supervisors: Bob Plant, Nigel Roberts.

WWRP_WSN05, Toulouse, 5-9 September 2005 / 1 Pertti Nurmi Juha Kilpinen Sigbritt Näsman Annakaisa Sarkanen ( Finnish Meteorological.

DIAMET meeting 7 th-8th March 2011 “New tools for the evaluation of convective scale ensemble systems” Seonaid Dey Supervisors: Bob Plant, Nigel Roberts.

Page 1© Crown copyright 2005 Met Office Verification -status Clive Wilson, Presented by Mike Bush at EWGLAM Meeting October 8- 11, 2007.

Deutscher Wetterdienst Preliminary evaluation and verification of the pre-operational COSMO-DE Ensemble Prediction System Susanne Theis Christoph Gebhardt,

NWP models. Strengths and weaknesses. Morten Køltzow, met.no NOMEK

Verification of ensemble precipitation forecasts using the TIGGE dataset Laurence J. Wilson Environment Canada Anna Ghelli ECMWF GIFS-TIGGE Meeting, Feb.

Trials of a 1km Version of the Unified Model for Short Range Forecasting of Convective Events Humphrey Lean, Susan Ballard, Peter Clark, Mark Dixon, Zhihong.

WRF Verification Toolkit Workshop, Boulder, February 2007 Spatial verification of NWP model fields Beth Ebert BMRC, Australia.

NCAR, 15 April Fuzzy verification of fake cases Beth Ebert Center for Australian Weather and Climate Research Bureau of Meteorology.

VERIFICATION Highligths by WG5. 2 Outlook Some focus on Temperature with common plots and Conditional Verification Some Fuzzy verification Long trends.

Extracting probabilistic severe weather guidance from convection-allowing model forecasts Ryan Sobash 4 December 2009 Convection/NWP Seminar Series Ryan.

Case study of a gust front passing the Danish island Funen on 9 November 2008 Majken Demant Meteorologist, aviation forecaster, DMI Eumetcal.

Overview of SPC Efforts in Objective Verification of Convection-Allowing Models and Ensembles Israel Jirak, Chris Melick, Patrick Marsh, Andy Dean and.

OSEs with HIRLAM and HARMONIE for EUCOS Nils Gustafsson, SMHI Sigurdur Thorsteinsson, IMO John de Vries, KNMI Roger Randriamampianina, met.no.

Predicting Intense Precipitation Using Upscaled, High-Resolution Ensemble Forecasts Henrik Feddersen, DMI.

1 Application of MET for the Verification of the NWP Cloud and Precipitation Products using A-Train Satellite Observations Paul A. Kucera, Courtney Weeks,

11 Short-Range QPF for Flash Flood Prediction and Small Basin Forecasts Prediction Forecasts David Kitzmiller, Yu Zhang, Wanru Wu, Shaorong Wu, Feng Ding.

Hirlam-Aladin All Staff Meeting Jose A. Garcia-Moya. AEMET. Spain. On behalf of Carlos Santos, Anna Ghelli (ECMWF) & Imanol Guerrero ECMWF & AEMET.

Characteristics of precipitating convection in the UM at Δx≈200m-2km

I. Sanchez, M. Amodei and J. Stein Météo-France DPREVI/COMPAS

met.no – SMHI experiences with ALARO

Fuzzy verification using the Fractions Skill Score

Systematic timing errors in km-scale NWP precipitation forecasts

Multi-scale validation of high resolution precipitation products

Convective Scale Modelling Humphrey Lean et. al

Daniel Leuenberger1, Christian Keil2 and George Craig2

Finnish Meteorological Institute

Linking operational activities and research

Verification of Tropical Cyclone Forecasts

Some Verification Highlights and Issues in Precipitation Verification

Rapid Adjustment of Forecast Trajectories: Improving short-term forecast skill through statistical post-processing Nina Schuhen, Thordis L. Thorarinsdottir.

Presentation transcript:

EMS 2013 (Reading UK) Verification techniques for high resolution NWP precipitation forecasts Emiel van der Plas Kees Kok Maurice Schmeits

2/15 EMS 2013 (Reading UK) 2 Introduction NWP has come a long way… It was: Then it became Hirlam: Now it is Harmonie It should be GALES (or so) It looks better… But how is it better? Does it perform better? That remains to be seen…

3/15 EMS 2013 (Reading UK) 3 Representation: “double penalty” Forecast localised phenomena: False alarm + Miss = double penalty Station (gauge) data: Forecast vs Radar data: When we take point-by-point errors (ME/RMSE):

4/15 EMS 2013 (Reading UK) 4 This talk HARP: Hirlam Aladin R-based verification Packages Tools for spatial, ensemble verification Based on R FSS, SAL, … Relies on eg SpatialVX package (NCAR) Generalized MOS approach Comparison high vs low resolution Hirlam (11 km, hydrostatic) Harmonie (2.5 km, non-hydrostatic, w/ & w/o Mode-S) ECMWF (T1279, deterministic) Lead times: +003, +006, +009, +012 Accumulated precipitation vs (Dutch) radar, synop

5/15 EMS 2013 (Reading UK) 5 Neo-classical: neighborhood methods, FSS Options: FSS, ISS, SAL, … Fraction Skill Score (fuzzy verification) (Roberts & Lean, 2008) Straightforward interpretation ‘Resolves’ double penalty But ‘smoothes’ away resolution that may contain information! ( V storm t ) == upscaling Baserate , FSS  observation forecast

6/15 EMS 2013 (Reading UK) 6 FSS results: Differences are sometimes subtle:

7/15 EMS 2013 (Reading UK) 7 FSS results: Differences are sometimes subtle: 1x1 3x3

8/15 EMS 2013 (Reading UK) 8 FSS: more results Higher resolutions: higher thresholds? DMO!

9/15 EMS 2013 (Reading UK) 99/15 How would a trained meteorologist look at direct model output? Model Output Statistics Learn for each model, location, … separately!

10/15 EMS 2013 (Reading UK) 10 Model Output Statistics Construct a set of predictors (per model, station, starting and lead time) : For now: use precipitation only Use various ‘areas of influence’: 25,50,75,100 km DMO, coverage, max(DMO) within area, distance to forecasted precipitation, … Apply logistic regression Forward stepwise selection, backward deletion Probability of threshold exceedance! Verify probabilities based on DMO, coefficients of selected predictors Training data: day 1-20, `independent’ data: day 21 – 28/31

11/15 EMS 2013 (Reading UK) 11 Model (predictor) selection Based on AIC (Akaike Information Criterion) Take the predictor with highest AIC in training set (day ) Test on independent set (day 21 – 28/31) Sqrt(tot_100) Sqrt(max)_100 More predictors != more skill distext_100 exp2int_100

12/15 EMS 2013 (Reading UK) 12 Model (predictor) selection Based on AIC (Akaike Information Criterion) Take the predictor with highest AIC in training set (day ) Test on independent set (day 21 – 28/31) Sqrt(tot_75) Max_50 More predictors != more skill

13/15 EMS 2013 (Reading UK) 13 Model comparison (April – October 2012) Hirlam, Harmonie (based on Hirlam) ECMWF 12UTC UTC UTC+009

14/15 EMS 2013 (Reading UK) 14 Summer vs winter Harmonie is expected to perform better in convective situations: What happens during winter? Summer (apr-nov)Winter (nov-apr)

15/15 EMS 2013 (Reading UK) 15/15 Discussion, to do MOS method: Stratification per station, season, … More data necessary, reforecasting under way Representation error: take (small) radar area Use ELR, conditional probabilities for higher thresholds Extend to wind, fog/visibility, MSG/cloud products, etc FSS: Use OPERA data

16/15 EMS 2013 (Reading UK) 16/15 Conclusion/Discussion Comparison between NWP’s of different resolution is, well, fuzzy Realism != Score Fraction Skill Score yields numbers, but sometimes hard to draw conclusions MOS method: Resolution/model independent Takes into account what we know Doubles (potentially) as predictive guide Thank you for your attention!

17/15 EMS 2013 (Reading UK) 17 Binary predictand y i (here: precip > q) Probability: logistic: Joint likelihood: L 2 penalisation (using R: stepPLR by Mee Young Park and Trevor Hastie, 2008) : minimise Use threshold (sqrt(q)) as predictor: complete distribution function (Wilks, 2009) Few cases, many potential predictors: pool stations, max 5 terms 17/15 Extended Logistic Regression (ELR)