1. Bill Kuo 1, Louisa Nance 1, Barb Brown 1 and Zoltan Toth 2 Developmental Testbed Center 1. National Center for Atmospheric Research 2. Earth System Research Laboratory Highlights of DTC Model Testing and Evaluation Results* *Contribution from all DTC Staff

2. Objectives of the DTC* Advance NWP research by providing the research community an environment that is functionally similar to that used in operations to test and evaluate the components of the NWP systems supported by the DTC; Reduce the average time required to implement promising codes emerging from the research community by performing initial extensive testing to demonstrate the potential of new science and technologies for possible use in operations; Sustain scientific interoperability of the community modeling system; Manage and support the common baseline of end-to-end community software to users, including dynamic cores, physics and data assimilation codes, pre- and post-processors and codes that support ensemble forecasting systems; and Establish, maintain and support a community statistical verification system for use by the broad NWP community. *DTC is jointly sponsored by NOAA, Air Force, NSF, & NCAR 2

3. Model Evaluation Tools (MET) State-of-the-art tools Traditional and advanced (e.g., spatial) Database and display system Supported to community Tutorials, email help, etc. Ensemble tools Brier score + decompositions ROC, Reliability Ensemble quantiles Rank histogram CRPS (continuous rank probability score) See Verification Methods session Friday, 10:30 and P57 - Fowler http://www.dtcenter.org/met/users/ 3

4. DTC Test & Evaluation Activities Mesoscale Modeling (WRF) Assess performance of select configurations for new WRF releases Test physics in a functionally similar operational environment P58 - Wolff, P59 - Harrold Test SREF member configurations QPF verification of high resolution models Assess performance of microphysics schemes (HMT) Hurricanes (HWRF) Test HWRF configured from WRF repository for use at EMC 2.2 – Bernardet, P84 - Bao Test HWRF physics options & assess their impact on rapid intensification 10.7 - Biswas Perform diagnostics studies to examine the strengths & weaknesses of HWRF (HFIP) Data Assimilation Test GSI baseline (comparison w/ WRF-Var) 9.6 - Shao Test regional EnKF systems P8 – Newman Ensembles Test bias-correction & down-scaling schemes for SREF Verification of storm-scale ensemble systems for severe weather & QPF (HWT) Demonstration of real-time QPF verification for mesoscale ensemble system (HMT) P55 – Jensen, P56 – Tollerud Data Assimilation/Ensembles/Hurricanes Assess the impact of GSI-hybrid DA on HWRF forecasts (HFIP) P7 - Zhou 4

5. WRF Innovation T&E Inter-comparison T&E allows for a quantitative assessment of forecast performance between an operational baseline and community contributed scheme Inter-comparison T&E allows for a quantitative assessment of forecast performance between an operational baseline and community contributed scheme 5 P59 - Harrold QNSE vs AFWA OCRRTMG vs AFWA OC

6. WRF Innovation T&E Inter-comparison T&E allows for a quantitative assessment of forecast performance between an operational baseline and community contributed scheme two different versions of WRF using the same physics scheme Inter-comparison T&E allows for a quantitative assessment of forecast performance between an operational baseline and community contributed scheme 6 AFWA OC V3.3.1 vs V3.1.1

7. Comparison of V3.1.1 and V3.3.1 From: Wei Wang and Ming Chen

8. WRF Member Testing for NCEP’s SREF New membership: NMMB(7), NMME(7) & ARW(7) Tested performance of 5 WRF configurations for ~50 cases distributed over a year  Candidate configuration NMM-GFS replaced w/ NMM-NCAR Cursory timing tests – ARW adaptive time step  Transition from 32/35 km to 16/17 km 8 Physics Parameterization ARW- NCAR ARW-RR ARW- NAM NMM- NAM NMM- GFS MicrophysicsWSM3ThompsonFerrier Surface Layer M-O Similarity Eta Similarity GFS PBLYSUMYJ GFS Convection Kain- Fritsch Grell-3DBMJ SAS LSMNoahRUCNoah Radiation RRTM/ Dudhia RRTM/ Goddard GFDL/ GFDL GFDL/ GFDL GFDL/ GFDL

9. Mesoscale Model Evaluation Testbed (MMET) – P58 – Jamie Wolff et al. Outcome of NWP Workshop on Model Physics with an Emphasis on Short-Range Weather Prediction, held at EMC 26-28 July 2011 Mechanism to assist research community with initial stage of testing and allow for efficient demonstration of merits of a new development Common framework for testing; allow for direct comparisons between different techniques Model input and observational datasets provided to utilize for testing Baseline results for select operational models established by the DTC Hosted by the DTC; served through Repository for Archiving, Managing and Accessing Diverse DAta (RAMADDA) http://dtcenter.org/repository 9

10. MMET Cases Initial solicitation of cases from DTC Science Advisory Board Members and Physics Workshop Participants – great response and enthusiasm towards endeavor Target cases during initial year 20090228 – Mid-Atlantic snow storm where North American Mesoscale (NAM) model produced high QPF shifted too far north 20090311 – High dew point predictions by NAM over the upper Midwest and in areas of snow 20091007 –High-Resolution Window (HIRESW) runs underperformed compared to coarser NAM model 20091217 – “Snowapocalypse ‘09”: NAM produced high QPF over mid-Atlantic, lack of cessation of precipitation associated with decreasing cloud top over eastern North Carolina 20100428-0504 – Historic Tennessee flooding associated with an atmospheric river event 20110404 – Recording breaking severe report day 20110518-26 – Extended period of severe weather outbreak covering much of the mid- west and into the eastern states later in the period 20111128 – Cutoff low over SW US; NAM had difficulties throughout the winter of breaking down cutoff lows and progressing them eastward 20120203-05 – Snow storm over Colorado, Nebraska, etc.; NAM predicted too little precipitation in the warm sector and too much snow north of front (persistent bias) 10

11. Research System: ESRL/GSD and HMT Ensemble Modeling System  WRF model 9-member ensemble; ARW and NMM cores  Outer domain 9km; Nested domain 3 km  Hybrid members: Multi physics packages, two model cores, and different GFS initial conditions  Outer domain runs to 5 day lead time; Nest to 12 hr; DTC evaluated first 72 hours  Comparisons made with current operational systems (GFS, SREF, NAM, HRRR, etc)  Evaluation focus on QPF with addition of state variables in 2011-2012  HMT-West typically runs December – March; DTC has evaluated approximately 3.5 months of data for past 3 seasons Innovations from HMT-West 11 see P55 Jensen et. al

12. HRRR (3km) HMT-Ens Mean (9km)NMM-B parallel (4km) NAM (12km) GFS (0.5 deg) Model Comparisons from 2010-2011 HMT-West 6 12 18 24 30 36 42 48 54 60 66 72 Gilbert Skill Score (or ETS) 6hr Accum Precip > 1” – Meso- and fine-scale models tended to have higher median Gilbert Skill Scores over GFS for extreme precipitation events. Differences appear statistically significant at hours 18-30 and 66  Including Parallel Runs in Testbed Evaluations: DTC testing of NMM-B parallel runs provided additional confidence (beyond EMC routine pre-implementation testing) and helped push forward an Oct. 2011 implementation of NMM-B core. 12

13.  Beyond higher resolution: Different initialization sources (AFWA) and methods (HMT and AFWA) may prove useful for the next-generation ensemble system. Select innovations from HMT-West will be tested by DTC during the coming year. Model Comparisons from HMT-West 2012 Gilbert Skill Score – Ability to forecast given amount 6hr Accum Precip > 1” - All scores are low – partially due to sample-size but SREF (32km) shows very little skill whereas HMT & AFWA (3 & 4km) ensembles can score as high at 0.3 Area Under ROC – Ability to discriminate between event/non-event Prob(6hr Accum Precip) > 1” - All scores are low at 6hr lead time – There are differences in the median AFWA and SREF values at 12 hr leads that may be significant No Skill Better Optimal (21 member)(10 member)(9 member) Gilbert Skill Score (or ETS) Area Under ROC Curve 13 see P55 Jensen et. al

14. HFIP GSI-Hybrid Data Assimilation Test: P7 Zhou No DA GSI 3DVAR GSI-Hybrid Best Track GSI Hybrid using global ensemble improved Bret track forecast 14

15. Summary & Outlook The DTC is a community facility with a mission to: Accelerate the transition of new NWP technology into operations Maintain and support community modeling systems for research and operational NWP communities Facilitate the interaction between research and operational NWP The DTC seeks input from the community through: Participation in DTC Testing and Evaluation activities (e.g., MMET): Funding is available for off-cycle visitor proposal Suggestions for new DTC T&E activities Defining future direction of the DTC through the DTC Science Advisory Board (Cliff Mass is the chair of DTC SAB) 15

16. THANK YOU! http://www.dtcenter.org/