1. Testbeds and Projects with Ongoing Ensemble Research:  Hydrometeorology Testbed (HMT)  Hazardous Weather Testbed (HWT)  Hurricane Forecast Improvement Project (HFIP) DET Collaboration with Other Projects: Past and Future Ed Tollerud 1 and Tara Jensen 2 1 NOAA/GSD, Boulder, CO 2 NCAR/RAL, Boulder, CO Acknowledgements: Tim Schneider, Marty Ralph, Gary Wick, Rob Cifelli, Dave Reynolds, Steve Weiss, Barbara Brown, Jack Kain, Ming Xue, Fanyou Kong, Adam Clark, Patrick Marsh, Mike Coniglio, Russ Schneider, Evan Kucera, Tom Hamill, Tony Eckel, Louisa Nance, Mark DeMaria, James Franklin, Ed Rappaport, Mike Fiorino and many others

2. Motivation Many testbeds now have an ensemble component Robust and objective evaluation of promising techniques needed Determine potential utility for operations Model A Model C Model E Model B Ensemble D Forecast Exercise Testbed Operations General Format of Many Testbeds Research

3. Goals Leverage off work done in other testbeds Extend understanding of their results Help in planning for next year or season Facilitate faster Research to Ops (R2O) Effort Time Forecast Exercise DET Retrospective Runs There is much work to be done in the off-season - DET might be able to help Most effort expended around R2O Forecast Exercise

4. Recent Activities HWT Participants Evaluate Models Traditional Scores for HWT ensemble members and baselines Object-Oriented Evaluation Of HMT members

5. 2010 DTC/HMT Ensemble Activities High-resolution 9 member WRF regional ensemble for HMT-West Winter Exercises ◦ First DTC ensemble evaluation ◦ Individual deterministic models ◦ Simple ensemble mean calculated and evaluated  30-day Summary Scoring Displays  Baseline Comparisons (with GFS)

6. 2010 HMT Ensemble NWP 9 member ensemble ◦ Grid: 9 km outer - 3 km inner ◦ LBC: GFS Ensemble Members  North American domain  80 km resolution ◦ IC: LAPS

7. January Summary, ETS 24h forecast, full domain Verified at 24h gage sites Segregated by threshold Dramatic performance hit at higher thresholds No real ensemble member advantages in statistics but large inter-quartile range Real-time QPF verification for HMT-West

8. 2011 HMT Ensemble NWP DET 1 st Benchmark (NCEP) will hopefully be run for experiment ◦ North American Domain ◦ 20 km resolution ◦ IC “cycling” option 9 member ensemble ◦ Grid: 9 km outer – 3 km inner (or potentially finer resolution) ◦ IC: LAPS and/or Cycling ◦ LBC: DET 1 st Benchmark (NCEP)

9. Future DTC/HMT Activities ◦ Use DET Ensemble Configuration Module to set-up and run ◦ First use of MET v3.0 including Ensemble-Stat Tool ◦ Additional Baselines ◦ Ensemble-based Spatial Verification ◦ Estimate Uncertainty in Verification Statistics

10. What can HMT do for DET HMT used for test domain for DET Module 2 (IC/LBC) Benchmark Workflow Procedures will backbone of DET system Input data options for initialization and verification (eg., QPF) Ensemble Products Algorithms for inclusion in Module 5 (Products/Display) Exploration of Spatial Verification Applications

11. What can DET do for HMT Help determine optimal ensemble configuration Evaluate Ensemble Products ◦ Visualization utilities (Displays) ◦ Probability Products State-of-the-art probabilistic verification measures

12. HWT 2008 Introduce Objective Evaluation HWT 2009 Realtime system Address scientific question HMT 2010 1 st Ensemble evaluation Satellite data into MET HWT 2010 Add Ensemble methods AWC/HPC present HMT 2011 Refine Ensemble methods Data Impact Studies ModelEvaluation Tools (MET) DTCEnsembleTestbed(DET) etc…

13. 2010 DTC/HWT Ensemble Activities Evaluated 26 member CAPS Storm-Scale Ensemble Forecast System during 2010 Spring Experiment ◦ Baselines: HRRR, SREF, NAM ◦ Ensemble Products generated by CAPS – using 15 members with radar assimilation ◦ First plots of Brier Score and Area Under Curve

14. NSSL Q2 QPE CAPS Simple PQPF SREF Simple PQPF CAPS PN PQPF NAM QPF SSEF PM QPF PQPF QPF Evaluation of Simple and Derived Ensemble Products: Probability fields using Neighborhood Method Probability matching method for QPF

15. Future DTC/HWT Activities ◦ DET participates in planning of CAPS ensemble constituents potentially with DET Ensemble Configuration Module, IC/LBC, and Physics Modules ◦ Real-time verification performed using DET Verification Module (HWT-DTC Collaboration) ◦ HWT products evaluated in DET Module 5 (Products/Display) ◦ Feedback to model developers RETOP

16. How DET may help HWT Module 1 - Work with CAPS to define optimal ensemble member configuration Module 2 – Work with DTC GSI task to identify GSI- based IC perturbation capabilities for CAPS ensemble Module 3 – Evaluate individual members in 2010 dataset to provide suggestions for physics improvements Module 4 – Implement bias correction technique that requires limited historical data Module 5 – Evaluate ensemble products in 2010 dataset Module 6 – Used in all evaluation

17. HFIP Ensemble Work Many institutions involved Many approaches being investigated Global Ensembles ◦ Single Models with IC/LBC and physics perturbations (GFS, NOGAPS, and FIM) Regional Ensembles ◦ Multi-model ensemble made up of regional models run by different organizations statistically post-processed (FSU) ◦ Ensembles initialized from EnKF (PSU) Consensus ensembles (NHC)

18. How DET might contribute to HFIP DET could be a testing ground for most promising ensemble techniques Retrospective testing to address specific questions: Module 2 – Modify near storm environment or storm position? Module 3 – Which cumulus schemes provide best spread? Module 4 – Which is best way to bias correct track and intensity? Module 5 – How do we best display of uncertainty in track and intensity? Module 6 – Does spatial (object oriented) verification of ensembles improve understanding of output?

19. SOME IDEAS FOR THE FUTURE…

20. MODULE 1: Ensemble Configuration MODULE 2: IC/LBC Perturbations MODULE 3: Physics Perturbations MODULE 4: Statistical Post-Processing MODULE 5: Products/Display MODULE 6: Verification Mostly Retrospective -------------------------------------- 1) ROBUST Test & Eval of promising techniques uncovered during forecast exercises Use Near Real-time & Retrospectively ---------------------------------------- 1)Immediate Feedback during forecast exercises 2)ROBUST Test & Eval of promising techniques uncovered during forecast exercises Concept of Leveraging DET Modules

21. Questions for Working Group: Should DET consider capability for realtime (or near realtime) demonstrations? Where do we place the “overlap area” between DET and other ensemble-related testbeds? What does that area include? What are the most critical testbed collaboration issues to focus on in the next year or two? What interactions with other testbeds beyond HMT and HWT should be explored in the next few years?

22. HWT 2012 Pre-Experiment Evaluation HWT 2011 Ensemble membership Evaluate New Products HFIP 2011 Ensemble Membership Statistical Post- Processing HMT 2012 Ensemble Methods for Spatial Verification HMT 2011 Refine Ensemble Verification methods Data Impact Studies DTCEnsembleTestbed etc… ModelEvaluation Tools (MET)