1. Ed Tollerud, Tara Jensen, Barb Brown ( also Yuejian Zhu, Zoltan Toth, Tony Eckel, Curtis Alexander, Huiling Yuan,…) Module 6 Objective: Provide a portable, inclusive toolset for Testing and Evaluation applications Verification Module for the DTC Ensemble Testbed DET Ensemble Workshop 2010 Wally Clark http://esrl.noaa.gov/gsd/fab

2. Module 1: Configuration Module 2: Initial Perturbations Module 4: Statistical Post- Processing External Input (HMT, HWT, HFIP, etc) Module 3: Module Perturbations Module 5: Product Generation Module 6: Verification DET MODULES

3. Motivation and Requirements  Provide ensemble and probabilistic verification utilities that are of value to: 1.Model development 2.Research applications 3.Operational feedback for forecasters 4.DET tasks and user-driven projects  Develop readily applicable routines to: 1.Assemble and process ensemble forecasts 2.Apply standard probabilistic verification algorithms 3.Employ newly-developed evaluation methods 4.Display results in thorough and intuitive fashion DET Ensemble Workshop 2010 Questions for breakout group Ensuring applicability/acceptability to user groups?

4. Motivation and Requirements (continued) Module should include verification utilities that are:  Objective but user-driven (‘honest broker’)  Comprehensive but portable  Flexible and extensible  User-friendly and supported (internet-enable)  State-of-the-art (Object-based; spatial ensemble verification) Utilities should be designed for:  Retrospective diagnostic studies and intensive forecast evaluation  Real-time application during quasi-operational field exercises (HWT and HMT experience) DET Ensemble Workshop 2010 Questions for breakout group How to insure portability to different ensemble forecast systems and applications? Critical operational or real-time capabilities? Other usage types?

5. Verification Algorithms, Options, and Related Displays DET Ensemble Workshop 2010 1)Comprehensive set of standard and probabilistic scores 2)Why not all that are feasible? 3)Examples of standard scores: MAE, ME, RMSE, ETS, POD,FAR, Bias, Odds ratio, etc. 4)Probabilistic scores and display types: Brier score and Brier skill scores and decomposition (reliability and resolution), Rank and Continuous Rank Probability Scores, ROC curve, Talagrand Diagram, Rank Histogram Questions for breakout group How to involve user group in contributions to displays and scoring algorithms? DET Visitor Program?

6. Examples: Individual member standard scores from HMT/HWT

7. DET Ensemble Workshop 2010 Examples: Brier Skill Score and Decomposition Products; Talagrand Diagram (Zhu et al.); Dispersion Diagram (Tony Eckel)

8. Examples: New Display Products (RAL and NAEFS) DET Ensemble Workshop 2010 Questions for breakout group How to develop and solicit new display possibilities? (Ensemble Products Workshop II?)

9. Verification Dataset Choices DET Ensemble Workshop 2010 QPF verification example: QPE possibilities include:  Operational Gage datastreams  Other Mesonet gage sets (CoCoRahs; OK mesonet; ….)  Radar products (Stage IV; Q2;…)  Satellite estimates from many sources Start-up default for verification: NCEP/EMC verification datasets; e.g., RTMA, PrepBufr,…. Analysis of Record (AOR) as source of high- resolution data Questions for breakout group Offer data options? Develop standardized datasets? Supply QC applications? Incorporate MADIS? How to build in data uncertainty?

10. Verification Packages and Products to Incorporate MET tools: Advantages include: 1.Strong synergy with other DTC tasks (HMT; HWT; HFIP) 2.In-house DTC product with good support 3.Continual updates, revisions, and novel additions 4.Evolving ensemble and probabilistic capabilities 5.Portable display systems with wide use (r; ncl) 6. ‘Metviewer’ interactive web display 7.Replication of EMC capabilities 8.Emphasis on confidence intervals and spatial methods Other verification products and packages with valuable input: 1.EMC probabilistic products 2.OHD hydrologic verification products 3.RUC and HRRR ensemble display products 4.NEVS procedures for data access 5. AWIPS; ALPS; NPVU; AFWA; others…. DET Ensemble Workshop 2010

11. Tentative Timeline and Milestones DET Ensemble Workshop 2010 June 2010 – Initial probabilistic scoring utilities demonstrated as part of HWT spring exercise October 2010 – Verification module plan completed and reviewed by WRF ensemble working group January 2011 – Expanded probabilistic scoring utilities demonstrated as part of HMT winter exercise June 2011 – Prototype real-time web-based verification site with basic capabilities completed September 2011 – Verification utility installed as full module in end-to-end workflow Questions for breakout group Contribute to other workshops (i.e., RAL Verification Workshop 11/2010; WMO Verification tutorials; others?)

12. Useful Websites and Information Sources DET Ensemble Workshop 2010 NPVU: http://www.hpc.ncep.noaa.gov/npvu/ ESRL/AMB: http://ruc.noaa.gov/hcpf/hcpf_verif.cgi OHD: http://hydis8.eng.uci.edu/hepex/Workshops/postprocesswksp/ POSTERS/Demargne-33.pdf OHD: http://hydis8.eng.uci.edu/hepex/testbeds/Verification.htm EMC/NCEP: https://ams.confex.com/ams/pdfpapers/131645.pdf ‘r’ Stats Pkg: http://cran.r-project.org/web/packages/verification/verification.pdf MET Verification example: http://verif.rap.ucar.edu/eval/hmt/2010/http://verif.rap.ucar.edu/eval/hmt/2010/ Others: Questions for breakout group How to promote portability between user groups?

13. Breakout Group Discussion – User Community Issues DET Ensemble Workshop 2010 1)Prioritized list of ensemble-relevant verification products 2)Dataset choices and quality How to promote portability between user groups? Questions for breakout group Ensuring applicability to user groups? How to insure portability to different ensemble forecast systems and applications? Critical operational or real-time capabilities? How to involve user group in contributions to displays and scoring algorithms? DET Visitor Program? How to develop and solicit new display possibilities? (Ensemble Products Workshop II?) Offer data options? Develop standardized datasets? Supply QC applications? Incorporate MADIS? How to build in data uncertainty? Contribute to other workshops (i.e., RAL Verification Workshop 11/2010; WMO Verification tutorials; others?)