1. Developmental Testbed Center: Overview
Bill Kuo

2. Outline DTC mission, history, and governance
Planning process for DTC annual operating plan (AOP)
Next phase of DTC – NOAA-UCAR Cooperative Agreement
2011 SAB recommendations and incorporation into AOP 2012
DTC AOP 2012 and budget
Looking ahead – AOP 2013 & beyond
Challenges and opportunities

3. DTC Mission
The fundamental purpose of the DTC is to facilitate the interaction & transition of NWP technology between research & operations. DTC facilitates:
R2O transition by performing testing & evaluation of new NWP innovations in a functionally similar operational environment over an extended period
O2R transition by making the operational NWP systems available to the research community & providing community user support
Interaction between research & operational NWP through the organization of community workshops/meetings on important topics of interest to the NWP community & hosting a DTC Visitor Program

4. DTC Overview
DTC was established in 2003, with initial funding from USWRP and NCAR.
The DTC Charter provides guidance for the governance of the DTC.
DTC AOP and budget need to be endorsed by the DTC Management Board (MB) and approved by the DTC Executive Committee (EC).
DTC Science Advisory Board (SAB) provides recommendations on codes/systems to be tested, visitor program, and future direction of DTC.
For FY2012, DTC has a budget of $5.3 M, with funding provided by NOAA/OAR, NWS/HFIP, USWRP, Air Force, NSF, GSD, and NCAR.
DTC has a staff of ~35 (total of ~22 FTE), distributed between NOAA/ESRL/GSD (1/3), and NCAR/RAL/JNT and NESL/MMM (2/3). The DTC Director’s Office is hosted at NCAR/RAL.

5. DTC governance and operation
Operation of DTC: Bill Kuo (Director), Louisa Nance (Assistant Director), Zoltan Toth (ESRL/GSD/FAB Branch Chief), Barb Brown (NCAR/RAL/JNT Director)
Priorities set by funding agencies with guidance from Science Advisory Board
Executive Committee
Members from funding agencies
Appoint DTC director, and provide executive oversight; Approve DTC AOP and budget; Approve membership of MB and SAB
Management Board (MB)
Director, deputy directors, members from funding organizations
Work with DTC Director to prepare DTC Annual Operating Plan (AOP) and budget; Assist DTC Director with the execution of DTC tasks
Science Advisory Board (SAB)
Members from operational and research communities
Provide advice to DTC Director on strategic direction and objectives; Review DTC Visitor Program proposals; Recommend codes for DTC testing
Executive Committee (representatives from funding organizations): appoints DTC Director and performs executive oversight on policy and resource commitments.
Management Board (MB - DTC Director, deputy directors, additional member from each chartering organization): assists Director in preparing Annual Operating Plan, annual budget, and selecting testing activities.
Advisory Board (nominated by MB, members from both operational and research communities): assists the Director with strategic direction and objectives of DTC.
BB: CHANGED LOUISA’S AFFILIATION TO CLARIFY THAT SHE IS DTC ASSISTANT DIRECTOR

6. NWS-UCAR Cooperative Agreement
To support the DTC activities at NCAR, a five-year Cooperative Agreement (CA) between NWS and UCAR was established covering the period of FY09 – FY13.
NOAA needs to prepare RFP (request for proposal) for the next phase of DTC.
Recommendations from SAB will serve as important community input for the next phase of DTC.
Final decision will be by NOAA senior management, in consultation with DTC Executive Committee.

7. DTC budget Sources: NOAA, AFWA, NCAR, NSF
NWS-UCAR CA on DTC FY09 – FY13
Millions of $
AFWA+
NOAA+
HFIP

8. Collaboration with NOAA testbeds
AOP 2011 Tasks
DTC Director’s Office
Mesoscale Modeling
Hurricanes
Data Assimilation
Ensembles
Verification
DTC Visitor Program
Activities are divided into basic task areas / Staff may be assigned to contribute to multiple tasks/ Tasks also utilize common tool sets / oversight structure where each task is assigned a lead…
Collaboration with NOAA testbeds
DTC activities focused on five key areas: Mesoscale Modeling, Hurricanes, Data Assimilation, Ensembles and Verification.
Collaborations with NOAA testbeds contribute to DTC task areas.

9. DTC Annual Operating Plan Process
DTC is funded by OAR, NWS (HFIP), USWRP, Air Force, NSF, NCAR and GSD, each with different periods of performance.
DTC has adopted March 1 to February 28th for its AOP.
Planning of 2012 AOP took into consideration recommendations from the DTC Science Advisory Board (SAB).
DTC is half way into the execution of AOP 2012.
Draft DTC AOP 2013 needs to be developed by Dec 2012 for DTC MB review and DTC EC approval in early 2013.
Comments and recommendations of DTC SAB will be taken into consideration for the preparation of DTC AOP 2013 as well as providing guidance for the long-term direction of DTC.
Funding from NWS/HFIP, USWRP, Air Force and NSF ($1.8M) is “targeted” for specific tasks, funding from OAR, NCAR, and GSD ($3.5M) is more flexible.

10. 2011 SAB Recommendations and Their Incorporation into AOP 2012

11. General Comments
The essential role of the DTC is to act as a testbed for research innovations in numerical weather prediction.... If the DTC does not succeed in its testbed role, the organization should not be seen as a success, no matter what other NWP-related activities it completes. Thus, the SAB strongly recommends that DTC management & staff concentrate sufficient resources to ensure an effective testbed system is in place & active by next year. Agree. DTC placed much greater emphasis on T&E for all DTC tasks in AOP 2012, with a focus on R2O. Highlights will be presented later in the meeting. While T&E activities are key, code management & maintaining connections w/ the community are also essential pieces to being a testbed, so it’s a delicate balancing act.

12. Specific recommendations (I)
DTC evaluate 5-10 community innovations during the next year, using a suite of case studies & multi-week runs. Agree. The Mesoscale Modeling, Hurricane, & Ensemble tasks plan to conduct at least 2 comprehensive tests of community innovations during AOP 2012, which would lead to testing at least 6 innovations overall. Attain & maintain competency in NEMS, but not support to the community. Concept of friendly users release is encouraged. Agree - included in AOP 2012 DTC should put in place the ability to comprehensively evaluate the next WRF ARW release & present results at the June WRF User’s workshop. Agree. DTC presented results on ARW Reference Configuration testing in June In the future, DTC will coordinate with other interested organizations (e.g., U of Washington, MMM, ESRL/GSD, OSU, CWB, etc) to provide comprehensive evaluation of ARW and present these results at future WRF workshop.

13. Specific recommendations (II)
Facilitate discussions of next generation hurricane model & model framework to assure affective R2O & O2R. DTC will work w/ EMC & HFIP on the planning & development of next generation hurricane model for operations. Establish an evaluation dataset composed of special interest cases (NCEP/EMC’s top 30 complaints), major historical weather events & well observed cases suitable for thorough evaluation, including both observations & gridded data needed to run model tests. Agree - included in AOP 2012 In collaboration w/ ensemble task, support HFIP community in adopting the GSI/EnKF hybrid system. Agree – included in AOP 2012 Full community support of the GSI/EnKF system should be considered in 1-2 years, when the system is more mature. Agree – community support for GSI-Hybrid system is not part of AOP 2012

14. Specific recommendations (III)
Keep focus on the development & testing of the next generation SREF – testing should be at 4 km. Documentation & community support should be a low priority Agree. DTC’s mission is to test innovations developed by the community. Focus for AOP 2012 will be on T&E, but at 10 km due to resource limitations. Testing at 4km will be considered for 2013 when more computing resources become available. Continue verification-related tutorial and expand online education materials (online MET tutorial and/or COMET module) to include more basic "education" material, including sample data and hands-on exercises to emphasize basic statistical and forecast evaluation concepts. Agree there is a need for basic verification training. We will explore ways of expanding our efforts in this area within the bounds of our resource limitations. One fairly cheap way to meet part of this need is to provide links on the MET web page to the large body of verification educational materials already available online.

15. DTC Budget

16. DTC funding sources (in $K)
FY2011
FY2012
Difference
NOAA/OAR
2,994
2,925
-69
NOAA/HFIP
708
460
-248
USWRP
281
190
-91
GSD
250
AFWA
718
855
137
NCAR
NSF
100
Carry-over
129
268*
139
Total
5,430
5,298
-132
* Due to period of performance for some projects, some funding is ‘committed carry-over’ for AOP 2012.

17. DTC budget allocations (in $K)
Funding allocation
FY2011
FY2012
Difference
Director’s Office
755
746 -9
Visitor Program
200
Mesoscale Modeling
1,037
1,029 -8
Hurricane
1,197
1,012
-185
Data Assimilation
772
569
-203
Ensemble
846
827
-19
Verification
623
915
*292
Total
5,430
5,298
-132
*Note verification support for T&E activities is included in the Verification task in AOP In AOP 2011, they resided with task areas conducting the test.

18. Planning for AOP 2013
Preliminary guidance for DTC is ‘flat budget,’ however, potential cuts are possible.
DTC will prepare draft AOP 2013 for a budget ranging from -10% to +10%.
SAB’s recommendations will be taken into consideration for the preparation of AOP 2013.
Schedule:
Oct 2012 – DTC MB meeting (discuss guidance from agencies)
Dec 2012 – Complete the draft AOP 2013
Mid-January 2013 – DTC MB review and approval
Early Feb 2013 – DTC EC review and approval
1 March 2013 – Execution begins

19. DTC Testing and Evaluation

20. DTC Testing and Evaluation (T&E)
Goals
Provide operational community with guidance for selecting new NWP techniques with potential added value
Provide research community with baselines against which new techniques can be evaluated
Philosophy
Maintain a neutral position in order to provide a trusted, unbiased assessment
Conduct comprehensive testing for a broad range of weather regimes
Evaluate based on extensive objective
verification statistics
Traditional metrics
New, relevant verification techniques
(e.g., spatial methods)
Statistical significance assessment

21. DTC Test & Evaluation Activities (I)
Mesoscale Modeling (WRF)
Reference Configuration testing to assess performance of WRF releases
Physics inter-comparison testing in a functionally similar operational environment
SREF membership configuration testing
Mesoscale Model Evaluation Testbed (MMET)
QPF verification of GSF and NAM
NCEP/MET verification system comparison
Hurricanes (HWRF)
Reference Configuration testing to assess performance of HWRF release
Test HWRF configured from community WRF repository to ascertain robustness
Test HWRF cumulus parameterization options
Benchmark HWRF skill in predicting intensity change and rapid intensity change
Perform diagnostics studies (large and storm scale) to examine the strengths & weaknesses of HWRF
Collaborate with FSU and UCLA to identify sources of errors in HWRF physical parameterizations

22. DTC Test & Evaluation Activities (II)
Data Assimilation
Test & tune GSI configurations &, therefore, set up a candidate implementation system for AFWA (against the operational DA system)
Estimate the impact of using regional background error (RBE) statistics instead of the global statistics from NCEP
Evaluate the performance of the regional BE generation using ensembles
Ensembles
Test bias-correction & down-scaling schemes for SREF
Verification of storm-scale ensemble systems for severe weather & QPF (HWT)
Demonstration of real-time QPF verification for mesoscale ensemble system (HMT)
Data Assimilation/Ensembles/Hurricanes
Assess the impact of GSI-hybrid DA on HWRF forecasts

23. Some thoughts on R2O
Major decisions are needed for next generation operational NWP systems in 3-5 years:
Global modeling
Hurricane prediction
Storm-scale ensemble prediction
Data assimilation from storm-scale to global scale
What is DTC’s role in facilitating closer collaboration between research and operations for future systems?
Does this country need 5 hurricane prediction models?
How do we avoid divergence of modeling systems between research and operational communities (e.g., WRF vs NEMS)?
Should DTC be involved in global modeling? [At least in facilitating the collaboration between research and operations?]

24. Challenges
To better serve research and operational NWP communities, DTC needs to take on new opportunities and challenges. However, DTC’s budget is not expected to increase in the future.
Traditional O2R, R2O process may not be efficient or cost-effective
Significant effort is required to transform operational codes into community codes and to support them to the broad users
Majority of research community users do not contribute back with new developments
Should DTC consider possible paradigm shift?
DTC reduces community support for operational codes
DTC provides code management and focused workshops for ‘expert groups’ and operational centers (e.g., EMC and AFWA)
DTC focuses its efforts on T&E to accelerate R2O
Should DTC consider reducing the number of systems that it support?

25. Comparison of global model five-day forecast among different centers
Cliffmass.blogspot.com/2012/03/us-fallen-behind-in-numerical-weather.html: “The NWS numerical weather prediction effort has been isolated and has not taken advantage of the research community,” said Cliff Mass

26. Next Generation Global Models
Next generation global models with unstructured grids provide interesting possibilities for a ‘unified’ model for operation. Challenges: Scale-dependent physics. How do we make a physics package work across varying grid sizes? What can the DTC do to help in this regard?

27. How to benefit operations from research done by the large WRF community?
The WRF community has significant experience in convective scale physics.

28. THANK YOU!