Developmental Testbed Center: Overview Bill Kuo

Outline DTC mission, history, and governance Planning process for DTC annual operating plan (AOP) Next phase of DTC – NOAA-UCAR Cooperative Agreement 2011 SAB recommendations and incorporation into AOP 2012 DTC AOP 2012 and budget Looking ahead – AOP 2013 & beyond Challenges and opportunities

DTC Mission The fundamental purpose of the DTC is to facilitate the interaction & transition of NWP technology between research & operations. DTC facilitates: R2O transition by performing testing & evaluation of new NWP innovations in a functionally similar operational environment over an extended period O2R transition by making the operational NWP systems available to the research community & providing community user support Interaction between research & operational NWP through the organization of community workshops/meetings on important topics of interest to the NWP community & hosting a DTC Visitor Program

DTC Overview DTC was established in 2003, with initial funding from USWRP and NCAR. The DTC Charter provides guidance for the governance of the DTC. DTC AOP and budget need to be endorsed by the DTC Management Board (MB) and approved by the DTC Executive Committee (EC). DTC Science Advisory Board (SAB) provides recommendations on codes/systems to be tested, visitor program, and future direction of DTC. For FY2012, DTC has a budget of $5.3 M, with funding provided by NOAA/OAR, NWS/HFIP, USWRP, Air Force, NSF, GSD, and NCAR. DTC has a staff of ~35 (total of ~22 FTE), distributed between NOAA/ESRL/GSD (1/3), and NCAR/RAL/JNT and NESL/MMM (2/3). The DTC Director’s Office is hosted at NCAR/RAL.

DTC governance and operation Operation of DTC: Bill Kuo (Director), Louisa Nance (Assistant Director), Zoltan Toth (ESRL/GSD/FAB Branch Chief), Barb Brown (NCAR/RAL/JNT Director) Priorities set by funding agencies with guidance from Science Advisory Board Executive Committee Members from funding agencies Appoint DTC director, and provide executive oversight; Approve DTC AOP and budget; Approve membership of MB and SAB Management Board (MB) Director, deputy directors, members from funding organizations Work with DTC Director to prepare DTC Annual Operating Plan (AOP) and budget; Assist DTC Director with the execution of DTC tasks Science Advisory Board (SAB) Members from operational and research communities Provide advice to DTC Director on strategic direction and objectives; Review DTC Visitor Program proposals; Recommend codes for DTC testing Executive Committee (representatives from funding organizations): appoints DTC Director and performs executive oversight on policy and resource commitments. Management Board (MB - DTC Director, deputy directors, additional member from each chartering organization): assists Director in preparing Annual Operating Plan, annual budget, and selecting testing activities. Advisory Board (nominated by MB, members from both operational and research communities): assists the Director with strategic direction and objectives of DTC. BB: CHANGED LOUISA’S AFFILIATION TO CLARIFY THAT SHE IS DTC ASSISTANT DIRECTOR

NWS-UCAR Cooperative Agreement To support the DTC activities at NCAR, a five-year Cooperative Agreement (CA) between NWS and UCAR was established covering the period of FY09 – FY13. NOAA needs to prepare RFP (request for proposal) for the next phase of DTC. Recommendations from SAB will serve as important community input for the next phase of DTC. Final decision will be by NOAA senior management, in consultation with DTC Executive Committee.

DTC budget Sources: NOAA, AFWA, NCAR, NSF NWS-UCAR CA on DTC FY09 – FY13 Millions of $ AFWA+ NOAA+ HFIP

Collaboration with NOAA testbeds AOP 2011 Tasks DTC Director’s Office Mesoscale Modeling Hurricanes Data Assimilation Ensembles Verification DTC Visitor Program Activities are divided into basic task areas / Staff may be assigned to contribute to multiple tasks/ Tasks also utilize common tool sets / oversight structure where each task is assigned a lead… Collaboration with NOAA testbeds DTC activities focused on five key areas: Mesoscale Modeling, Hurricanes, Data Assimilation, Ensembles and Verification. Collaborations with NOAA testbeds contribute to DTC task areas.

DTC Annual Operating Plan Process DTC is funded by OAR, NWS (HFIP), USWRP, Air Force, NSF, NCAR and GSD, each with different periods of performance. DTC has adopted March 1 to February 28th for its AOP. Planning of 2012 AOP took into consideration recommendations from the DTC Science Advisory Board (SAB). DTC is half way into the execution of AOP 2012. Draft DTC AOP 2013 needs to be developed by Dec 2012 for DTC MB review and DTC EC approval in early 2013. Comments and recommendations of DTC SAB will be taken into consideration for the preparation of DTC AOP 2013 as well as providing guidance for the long-term direction of DTC. Funding from NWS/HFIP, USWRP, Air Force and NSF ($1.8M) is “targeted” for specific tasks, funding from OAR, NCAR, and GSD ($3.5M) is more flexible.

2011 SAB Recommendations and Their Incorporation into AOP 2012

General Comments The essential role of the DTC is to act as a testbed for research innovations in numerical weather prediction.... If the DTC does not succeed in its testbed role, the organization should not be seen as a success, no matter what other NWP-related activities it completes. Thus, the SAB strongly recommends that DTC management & staff concentrate sufficient resources to ensure an effective testbed system is in place & active by next year. Agree. DTC placed much greater emphasis on T&E for all DTC tasks in AOP 2012, with a focus on R2O. Highlights will be presented later in the meeting. While T&E activities are key, code management & maintaining connections w/ the community are also essential pieces to being a testbed, so it’s a delicate balancing act.

Specific recommendations (I) DTC evaluate 5-10 community innovations during the next year, using a suite of case studies & multi-week runs. Agree. The Mesoscale Modeling, Hurricane, & Ensemble tasks plan to conduct at least 2 comprehensive tests of community innovations during AOP 2012, which would lead to testing at least 6 innovations overall. Attain & maintain competency in NEMS, but not support to the community. Concept of friendly users release is encouraged. Agree - included in AOP 2012 DTC should put in place the ability to comprehensively evaluate the next WRF ARW release & present results at the June WRF User’s workshop. Agree. DTC presented results on ARW Reference Configuration testing in June 2012. In the future, DTC will coordinate with other interested organizations (e.g., U of Washington, MMM, ESRL/GSD, OSU, CWB, etc) to provide comprehensive evaluation of ARW and present these results at future WRF workshop.

Specific recommendations (II) Facilitate discussions of next generation hurricane model & model framework to assure affective R2O & O2R. DTC will work w/ EMC & HFIP on the planning & development of next generation hurricane model for operations. Establish an evaluation dataset composed of special interest cases (NCEP/EMC’s top 30 complaints), major historical weather events & well observed cases suitable for thorough evaluation, including both observations & gridded data needed to run model tests. Agree - included in AOP 2012 In collaboration w/ ensemble task, support HFIP community in adopting the GSI/EnKF hybrid system. Agree – included in AOP 2012 Full community support of the GSI/EnKF system should be considered in 1-2 years, when the system is more mature. Agree – community support for GSI-Hybrid system is not part of AOP 2012

Specific recommendations (III) Keep focus on the development & testing of the next generation SREF – testing should be at 4 km. Documentation & community support should be a low priority Agree. DTC’s mission is to test innovations developed by the community. Focus for AOP 2012 will be on T&E, but at 10 km due to resource limitations. Testing at 4km will be considered for 2013 when more computing resources become available. Continue verification-related tutorial and expand online education materials (online MET tutorial and/or COMET module) to include more basic "education" material, including sample data and hands-on exercises to emphasize basic statistical and forecast evaluation concepts. Agree there is a need for basic verification training. We will explore ways of expanding our efforts in this area within the bounds of our resource limitations. One fairly cheap way to meet part of this need is to provide links on the MET web page to the large body of verification educational materials already available online.

DTC Budget

DTC funding sources (in $K) FY2011 FY2012 Difference NOAA/OAR 2,994 2,925 -69 NOAA/HFIP 708 460 -248 USWRP 281 190 -91 GSD 250 AFWA 718 855 137 NCAR NSF 100 Carry-over 129 268* 139 Total 5,430 5,298 -132 * Due to period of performance for some projects, some funding is ‘committed carry-over’ for AOP 2012.

DTC budget allocations (in $K) Funding allocation FY2011 FY2012 Difference Director’s Office 755 746 -9 Visitor Program 200 Mesoscale Modeling 1,037 1,029 -8 Hurricane 1,197 1,012 -185 Data Assimilation 772 569 -203 Ensemble 846 827 -19 Verification 623 915 *292 Total 5,430 5,298 -132 *Note verification support for T&E activities is included in the Verification task in AOP 2012. In AOP 2011, they resided with task areas conducting the test.

Planning for AOP 2013 Preliminary guidance for DTC is ‘flat budget,’ however, potential cuts are possible. DTC will prepare draft AOP 2013 for a budget ranging from -10% to +10%. SAB’s recommendations will be taken into consideration for the preparation of AOP 2013. Schedule: Oct 2012 – DTC MB meeting (discuss guidance from agencies) Dec 2012 – Complete the draft AOP 2013 Mid-January 2013 – DTC MB review and approval Early Feb 2013 – DTC EC review and approval 1 March 2013 – Execution begins

DTC Testing and Evaluation

DTC Testing and Evaluation (T&E) Goals Provide operational community with guidance for selecting new NWP techniques with potential added value Provide research community with baselines against which new techniques can be evaluated Philosophy Maintain a neutral position in order to provide a trusted, unbiased assessment Conduct comprehensive testing for a broad range of weather regimes Evaluate based on extensive objective verification statistics Traditional metrics New, relevant verification techniques (e.g., spatial methods) Statistical significance assessment

DTC Test & Evaluation Activities (I) Mesoscale Modeling (WRF) Reference Configuration testing to assess performance of WRF releases Physics inter-comparison testing in a functionally similar operational environment SREF membership configuration testing Mesoscale Model Evaluation Testbed (MMET) QPF verification of GSF and NAM NCEP/MET verification system comparison Hurricanes (HWRF) Reference Configuration testing to assess performance of HWRF release Test HWRF configured from community WRF repository to ascertain robustness Test HWRF cumulus parameterization options Benchmark HWRF skill in predicting intensity change and rapid intensity change Perform diagnostics studies (large and storm scale) to examine the strengths & weaknesses of HWRF Collaborate with FSU and UCLA to identify sources of errors in HWRF physical parameterizations

DTC Test & Evaluation Activities (II) Data Assimilation Test & tune GSI configurations &, therefore, set up a candidate implementation system for AFWA (against the operational DA system) Estimate the impact of using regional background error (RBE) statistics instead of the global statistics from NCEP Evaluate the performance of the regional BE generation using ensembles Ensembles Test bias-correction & down-scaling schemes for SREF Verification of storm-scale ensemble systems for severe weather & QPF (HWT) Demonstration of real-time QPF verification for mesoscale ensemble system (HMT) Data Assimilation/Ensembles/Hurricanes Assess the impact of GSI-hybrid DA on HWRF forecasts

Some thoughts on R2O Major decisions are needed for next generation operational NWP systems in 3-5 years: Global modeling Hurricane prediction Storm-scale ensemble prediction Data assimilation from storm-scale to global scale What is DTC’s role in facilitating closer collaboration between research and operations for future systems? Does this country need 5 hurricane prediction models? How do we avoid divergence of modeling systems between research and operational communities (e.g., WRF vs NEMS)? Should DTC be involved in global modeling? [At least in facilitating the collaboration between research and operations?]

Challenges To better serve research and operational NWP communities, DTC needs to take on new opportunities and challenges. However, DTC’s budget is not expected to increase in the future. Traditional O2R, R2O process may not be efficient or cost-effective Significant effort is required to transform operational codes into community codes and to support them to the broad users Majority of research community users do not contribute back with new developments Should DTC consider possible paradigm shift? DTC reduces community support for operational codes DTC provides code management and focused workshops for ‘expert groups’ and operational centers (e.g., EMC and AFWA) DTC focuses its efforts on T&E to accelerate R2O Should DTC consider reducing the number of systems that it support?

Comparison of global model five-day forecast among different centers Cliffmass.blogspot.com/2012/03/us-fallen-behind-in-numerical-weather.html: “The NWS numerical weather prediction effort has been isolated and has not taken advantage of the research community,” said Cliff Mass

Next Generation Global Models Next generation global models with unstructured grids provide interesting possibilities for a ‘unified’ model for operation. Challenges: Scale-dependent physics. How do we make a physics package work across varying grid sizes? What can the DTC do to help in this regard?

How to benefit operations from research done by the large WRF community? The WRF community has significant experience in convective scale physics.

THANK YOU! http://www.dtcenter.org/