1. Improved CTB R2O Process Annarita Mariotti (OAR/CPO) and Jin Huang (NWS/NCEP) November 10, 2015

2. Outline Two-phased R2O process o Priority setting and selection of demonstration projects o Requirements for demonstration projects o Monitoring progress during project execution o Post-Project review and operational deployment CTB Measures of success and progress Summary

3. How are CTB priorities set and projects selected? The CTB R2O priorities are based on the NWS operational requirements, the gaps and needs in the current NWS operational products, science readiness and programmatic opportunities; The selection of the CTB demonstration projects is conducted through a competitive proposal review process managed by the CPO/MAPP Program.

4. A Two-Phased R2O Process Demonstration Phase Operational Deployment Phase Part of OAR Research and Development Also supported by NWS/R2O program Part of NWS operations ? Demonstration Project Review Climate Test Bed: a prime example of OAR–NWS partnership in R2O/O2R

5. High relevance to solicitation priorities and to NCEP Involve an NCEP co-PI or collaborator (contact Jin Huang to make a connection with NCEP) Clear testing objectives and metrics (what will be tested for transition into NCEP and how) Transition for “Technical Readiness Levels” (TRL) 5–8 Feasibility and support from NCEP Must include a transition plan, for both demonstration and potential operational deployment at a later stage Technical Readiness Levels (TRLs) 5678 9 (Deployment) Demo Project TRL 5-7 Demo Project TRL 6-8 Requirements for Demonstration Projects: Demonstration projects at different TRL, all need to show a pathway to operations 1) TRL 5-7: early state testing and more risky 2) TRL 6-8: more mature testing

6. Technical Readiness Levels* NAO 216-105 What is the Technical Readiness Level?

7. MAPP–CTB Proposal Preparation Process External community PIs and NCEP co-PIs interact on LOI ideas. NCEP management provides guidance and feedback to NCEP co- PIs on the preparation of the LOIs. CPO provides feedback to the LOIs regarding the submission of a full proposal as stated in solicitation. NCEP management provides feedback to NCEP co-PIs on transition plans before full proposals are submitted to CPO for peer review in terms of relevance to NCEP operational priorities and NWS commitment Transition plans signed by NWS AA and OAR AA or designees before funding decisions are made

8. CTB Transition Plans to Ensure Effective R2O Demonstration Phase Plan: What would be deployed operationally as a result of this research Metrics for measuring the success of the demonstration phase Tasks, deliverables and timeline Costs to OAR NOAA HPC requirements during demonstration (if any) Deployment Phase Plan: Tasks and timeline for deployment Cost to NWS for deployment (if any) HPC requirements during deployment (if any) Signatures of the NWS AA and the OAR AA are required before the projects are approved for funding A transition plan is required for both the demonstration activities and potential operational deployment as part of the proposal

9. Project Execution -The CTB Management Team meets quarterly to monitor demonstration projects execution and plan for potential operational deployment -Besides annual reports, the Team considers progress and issues for each project presented by the CTB Director. -CTB Director checks in with each project (at least quarterly) to see how things are going, update project plans (milestones) as necessary and find out if there are any current issues or anticipated risks that require assistance from the federal side.

10. Project Execution CTB Director maintains a quad chart for each project to keep up to date on project plans and document issues or risks that need assistance from federal side. - PIs have opportunity to communicate any issues or risks or changes to the project schedule, i.e. : -“Don’t have enough computing to do our experiments” -“We lost critical staff member so project timeline/milestones will need to slip. New schedule will be … (This will require a no-cost extension) -About the quad chart: -Simple way for tracking changes/updates to the project plan. -It isn’t a grading device -Will be maintained by CTB Director. PIs only provide updates. -Shouldn’t take more than 10 minutes per quarter of PIs time to provide updates.

11. Example of Quad Chart to Track Demonstration Projects during execution A CPT for improving Turbulence and Cloud Processes in NCEP Global Models Issues/Risks Issues: 1)O2R: Pis need to access the latest GFS model code Mitigation: Finances Scheduling Project Information and Highlights Funding Sources: 1)Demonstration Phase: FY13: $5K; FY14: $348K; FY15: $358K; FY16: 360K Status: Management Attention Required Potential Management Attention Needed On Target Updated on 11/04/2015 Lead: Steven K. Krueger and Shrinivas Moorthi (EMC) Scope: - Test impacts of improved Turbulence and Cloud Processes in NCEP Global Models Estimated Benefits: A PDF-based SGS turbulence and cloudiness scheme, a cumulus parameterization that scales continuously between simulating individual clouds and conventional parameterization of deep convection an improved representation of the interactions of clouds, radiation, and microphysics MilestoneDateStatus Install Simplified Higher Order Closure (SHOC) scheme in in a version of GFS FY16Q2 Install Chikira and Sugiyama (CS) and Arakawa-Wu (AW) convection parameterizations in GFS. FY16Q* Install SHOC and CS in NEMSFY16Q* Unifying the cloud-radiation-microphysics interaction Post-Project ReviewFY16Q4 (Sep) GYR GG G Y Note: This is an sample chart The PIs will discuss the milestones in their project-level meetings

12. Post-Project Project Review and Potential Operational Deployment Post-Project Review: within 90 days of end of project – Assesses whether the demonstration activities were executed according to plan – Assesses the outcome of tests that were executed and whether operational deployment is recommended NCEP considers outcome of review and make a decision on whether to deploy operationally – Operational deployment plan is put in place conditional on NCEP’s decision CTB Demonstration Phase Operational Deployment Phase ? Demonstration Project Review

13. Measure of Progress and Success: Project level metrics Metrics and Protocols for Evaluating NCEP Climate Models and Forecasts (July 26, 2013; shown in CTB website; Referred by MAPP-CTB call for proposals) 1.Climate Model performance metrics mean and biases modes of variabilities (ENSO, MJO, ENSO); Teleconnections Forecast Skills of SSTs, ENSO, Temp, precipitation 2.Climate forecast evaluation metrics Deterministic/continuous forecasts: AC, RMSE, mean error, biases Categorical: e.g., Heidke score Probabilistic: BSS, RPSS, reliability 3.Climate Forecast and Hindcast Evaluation Protocol Hindcast period: 30 year for seasonal; 15 years for sub-seasonal Forecast lead time What should be the metrics for monitoring products ?

14. Measure of Progress and Success: Overall CTB performance measures Metrics: Total number of CTB projects successfully executed Annual number of CTB projects’ transitioned to a higher TRL Considerations: The ultimate goal is to transfer new methodologies or models components to improve those currently at NCEP as part of its operational suite. CTB performance measures need to capture the progressive transition to higher RLs, not just the RL 8-9 transition. It is possible that a demonstration project which demonstrates a forecast/analysis benefit although the final project review still may not result in operational transition due to other considerations

15. Summary A two-phased R2O process with upfront planning and constant monitoring during project execution o Careful selection of demonstration projects which involves having specific requirements o Monitoring progress during project execution o Post-Project review to inform operational deployment A set of measures of success and progress at the project and overall CTB level