1. The Developmental Testbed Center: Historical Perspective and Benefits to NOAA Steve Koch DTC Deputy Director Director, NOAA/ESRL/Global Systems Division

2. Fundamental Purpose of DTC Operational Codes DTC To serve as a bridge between research and operations to facilitate the activities of both halves of the NWP Community in pursuit of their own objectives: Research community gets a functionally equivalent operational environment to test and evaluate new NWP methods in retrospective extended period tests using advanced tools Operational community benefits from DTC testing and evaluation of strengths and weaknesses of new NWP advances prior to consideration for operational implementation

3. NOAA will use the DTC as the primary gateway through which promising well-tested NWP science and technology originating in the research community will be selected for further development and evaluation by NCEP/EMC EMC resources will be used much more efficiently since DTC will filter out less promising technologies and greatly reduce the time needed for the better ones to become ready for operational implementation DTC activity links with Weather and Water Goal’s STI program, with future support to be provided primarily through the Environmental Modeling Program

4. Agency priorities for DTC NOAA: Top priority of DTC is “to manage the WRF community code and support the WRF modeling system to the research community… the DTC is the critical link to reaping the benefits of the WRF Program.” April 28, 2006 WRF ExOB AFWA: Top priority of DTC is “to manage the WRF baseline. DTC is the key component/enabler for establishing, maintaining, and supporting the ‘Official’ (ARW- and NMM-WRF cores) baseline of the community’s end-to-end WRF ‘Reference-grade’ code.” June 5, 2006 SOW

5. Operational Community  Provide guidance for next generation Operational Configurations Research Community  Provide baselines for evaluating impacts of new techniques  Aid in selecting configurations for research projects Testing and Evaluation Goals

6.  DTC demonstrated the capabilities of the candidate dynamical cores to qualify them for a 6-member WRF IOC ensemble system to run daily in High Resolution Window (HRW) domains: ◦ NMM + ARW dynamical WRF cores with NCEP + NCAR physics ◦ NMM and ARW cores with switched physics packages ◦ Model variants using bred Initial / Boundary Conditions  Ensemble improves accuracy over a single deterministic forecast and provides for a measure of uncertainty. WRF Test Plan: Getting to the NCEP IOC

7.  DTC testing and evaluation demonstrated to NCEP that the 6 WRF members tested were qualified to run as an ensemble system  21 Sept 2004: IOC implemented with only 2 members (the two dynamical cores - ARW and NMM - without physics swapping)  6 Dec 2005: Six WRF members added to NCEP Short-Range Ensemble Forecast (SREF) included physics swapping between the cores and 2 members from NCEP breeding system  New WRF-based SREF outperformed the current one in all ensemble aspects including mean, spread and probability distributions Impact of the WRF Test Plan on NCEP

9. DTC Winter Forecast Experiment Goal: Provide real-time model guidance for winter weather over a large domain using 2 WRF configurations with explicit convection Objectives  NWS: Expose forecasters to WRF at very high (5-km) resolution prior to the first scheduled operational implementation at NCEP.  NCEP: Evaluate WRF in HRW at 8-10 km resolution relative to WRF run at 4-5 km over CONUS with explicit microphysics only.  NCAR: Determine whether encouraging results from warm season 4-km WRF runs carry over to winter events out to 48h.  Real-time displays: ◦ NCAR: DTC website ◦ GSD: FX-Net ◦ GSD: 2D AWIPS fields-NMM only  Objective & subjective evaluations by DTC and NWS, respectively

10.  Provided valuable hands-on experience to forecasters with high-resolution WRF model output over CONUS  NCEP added simulated reflectivity to diagnostic fields available to WFOs  Results helped NCEP decide to upgrade resolution of HRW domains (NMM 8 to 5.1 km / ARW 10 to 5.8 km)  DTC demonstrated importance of statistical tests of significance when comparing forecast skill from multiple models  NMM 5 km real-time runs were continued into Spring at request of forecast offices (NMM5-CONUS) DWFE Impact on NWS

11. 0600 UTC Radar Mosaic ARW 3-h forecast Composite Reflectivity Lake-effect snowbands Narrow Cold- Frontal Rainband Observed Composite Reflectivity at 03Z

12. WRF-ARW 00Z 22Jan05 F27 988 Verifying 5-km Surface Mesoanalysis for 0300 UTC 995 Stationary Mesoscale Wave-Precip Systems

13. Eta-12 WRF-NMM WRF-ARW DWFE Verification Activities

14. Key questions still remain: What are the implications of choosing a single dynamic core for WRF? Do the potential cost savings overcome the loss of model diversity for ensemble predictions? Previous DTC experiments provided some idea of how the two WRF cores compare in terms of forecast skill, but no tests have been totally “clean” so far – this requires each core to be run with the same physics package, initial and boundary conditions, horizontal and vertical resolution, & run over the same domain This was done with the WRF-Rapid Refresh “core test” WRF-RR is the first operational WRF model at NCEP for which the DTC can perform such tests and evaluations prior to implementation Never before has such a clean dynamic core test been performed 2 tests completed: using NMM physics in both cores and RUC-like physics 13-km WRF run to 24h even though RUC only extends to 12h WRF-Rapid Refresh “Core Testing”

15. WRF Core Test: Rapid Refresh (RR) Goals  Conduct controlled experiments carefully comparing the 2 WRF dynamic cores.  Provide datasets to support GSD’s dynamic core recommendation to NCEP for WRF-RR Impacts  1 st “clean” WRF dynamic core comparison  ARW selected for initial implementation of WRF-RR

16. WRF Core Test Rapid Refresh Setup Domain: CONUSDt: ARW 72 s, NMM 30 s ICs: RUC13 BCs: NAM Forecast length: 24 h Forecast cycles: 00 & 12 UTC Upper BC: NMM – default ARW – damping layer (5 km) Physics frequency: Radiation - 30 min Other - ARW 72 s, NMM 60 s Dx: 13 km Vertical levels: 50 Summer15 Jul - 15 Aug 05 Fall1 - 30 Nov 05 Winter15 Jan - 15 Feb 06 Spring25 Mar - 25 Apr 06 PhysicsPhase 1Phase 2 MicrophysicsFerrierThompson RadiationGFDL PBLMYJ LSMNoah-99RUC CumulusBMJGrell-Devenyi

17. Red = Physics 1 = NMM set Blue = Physics 2 = RUC-like set 2-sigma = 95% CI

18. Red = Physics 1 = NMM set Blue = Physics 2 = RUC-like set 2-sigma = 95% CI

19. Objective  Assess whether the medium range (60h) skill of the two dynamic cores is similar enough to allow the direct transfer of research results obtained with one core to the other. Approach  Use Rapid Refresh Core Test as basic template for test design  Use current code for all components of end-to-end system ◦ Relevance to community! ◦ Maximize support from developers WRF Core Test - Extended

20. Wind errors are maximum at 250 hPa NMM has best forecast at 850 and 700 hPa 60-h bias-corrected RMSE (NMM in red; ARW in blue; ARW-NMM in green) Temperature errors show max at 850 and 200 hPa Core differences are not statistically significant RH errors increase with height to 500 hPa Core differences are not statistically significant

21. Most variables and levels, such as surface temperature, do not show an increase in ARW – NMM differences in time out to 60h. However, core differences do grow in time for a few variables and levels, such as temperature at 100 hPa. Evolution in Time 2-m temperature bias100 hPa temperature bias

22. Extended Core Test  ARW and NMM results are not substantially different  Inter-core differences do not grow in time  Potential for transferring research done with one core to the other

23. IBM vs Linux Comparison

24.  WRF Test Plan ◦ NMM ported to WRF Framework (increased number of options available to community)  DWFE ◦ Problem related to initialization of small lakes during winter season brought to attention of developers  WRF-RR Core Test ◦ Significant increase in physics options available for NMM  Extended Core Test ◦ More consistency between pre-processing options available for 2 dynamic cores  Pre-release and system testing ◦ Bug fixes for both dynamic cores ◦ Improved interoperability (more microphysics & PBL schemes available for NMM) Impact of DTC T & E WRF Community Code