1. ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE Jeff Waldstreicher Scientific Services Division – Eastern Region Northeast Regional Operational Workshop (NROW) November 4-5, 2003 Albany, NY

2. ASSESSING THE IMPACTS OF COLLABORATIVE PROJECTS IMPACT ON VERIFICATION SCORES –Performance Metrics CASE STUDY ANALYSIS –Event Verification –Subjective Evaluation of Impact on Forecast Process AFDs Event Reviews and Impact Reports

3. COMPLICATIONS Many Factors Influence Performance Metrics –Factors Not Independent –Difficult to tie performance changes to a specific factor Cannot Analyze “Null Case” –If forecasters have “knowledge” or “data,” cannot directly answer “What if they did not have the knowledge?”

4. FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES Infusion of New Technologies –Hardware (New systems or Processors) –Software (New algorithms or Models) Applied Research and Development –National (Research Laboratory) –Local/Regional –Independent –Collaborative Changes to Operational Procedures –Implementation of Best Practices

5. FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES Climate Variability –Variations in frequency and type of events External Outreach and Education –Including development of spotter networks Personnel/Staffing Issues –Including forecaster experience levels Training –All other factors are also tied to training issues

6. HOW DO THESE FACTORS IMPACT PERFORMANCE METRICS? Can be global (national), regional, or local –Technology usually national –Climate Variability typically regional –Staffing usually local Impact can be long or short term –Technology infusion typically has long term impact –Climate Variability is usually short term Factors are often inter-related –New technology or staffing changes require training –Applied R&D often yields/suggests changes to operations

7. COMET PROJECTS IN EASTERN REGION Different types of projects –Cooperative (2-3 year ~$35K/yr) 19 ER Cooperative Projects funded since 1991 –Partners (1 year ~$9K/yr) 39 ER Partners Projects funded since 1991 20 ER WFOs and 3 RFC have participated –90 Offices Nationwide 21 Universities have participated in ER projects –70+ Universities Nationwide

8. EVALUATING COMET PROJECTS Examined COMET Cooperative and Partners Projects in Eastern Region –Projects completed between 1995 and mid-2001 –Projects specifically addressing warning programs Tornado Severe Thunderstorms Flash Flooding Winter Storms Study was designed to minimize as much as possible the impact of the factors previously discussed.

9. METHODOLOGY 3-year running verification scores used –Minimize impact of short-term factors such as variability of events Compared 3 years before project to 3 years following project –1996-mid 2001 period helped ensure 3-years of post-88D data in “before” scores and a full 3 years of “after” scores 3-year “expected” improvements calculated based on long term trend of ER-wide scores –Compared rate of improvement for WFO involved in collaborative project to ER-wide improvement –Use of ER scores as a baseline minimizes the impact of national/region- wide factors such as AWIPS and radar system improvements Very difficult to evaluate the impact of project results beyond the primary WFO

16. INDIVIDUAL PROJECT COMPARISON Compare 3-year performance change of WFOs involved in COMET projects to the region-wide improvements for the same time period

17. PROBABILITY OF DETECTION (POD) # of COMET Projects With WFO Greater Improvement # of COMET Projects With ER Greater Improvement # of COMET Projects With No Difference in Improvement Tornado Warnings 32 1 Severe Tstm Warnings 610 Flash Flood Warnings 510 Winter Storm Warnings 430

18. FALSE ALARM RATIO (FAR) # of COMET Projects With WFO Greater Improvement # of COMET Projects With ER Greater Improvement # of COMET Projects With No Difference in Improvement Tornado Warnings 240 Severe Tstm Warnings 340 Flash Flood Warnings 330 Winter Storm Warnings 331

19. LEAD TIME # of COMET Projects With WFO Greater Improvement # of COMET Projects With ER Greater Improvement # of COMET Projects With No Difference in Improvement Tornado Warnings 510 Severe Tstm Warnings 511 Flash Flood Warnings 411 Winter Storm Warnings 430

20. IMPACT OF LONG TERM COLLABORATIVE ACTIVITIES WFO RAH and NCSU –Continuous collaborative projects since January 1991 –COMET Projects 1991-2000 3 Cooperative Projects 3 Partners Projects 1 Graduate Fellowship –CSTAR Projects 2000-2003 2003-2006 Unique “laboratory” to examine the impact of long-term collaborative activities

27. IMPACT OF LONG TERM COLLABORATIVE ACTIVITIES WFO ALY and U-Albany –Continuous since February 1995 –Numerous COMET Projects 3 Cooperative Projects 7 Partners Projects Several additional projects between UA and NCEP –CSTAR PROJECT 2001-2004

34. SUMMARY Verification scores for ER WFOs involved in COMET collaborative research projects appear to improve at a greater rate than overall ER performance There are indications that certain performance metrics (e.g., lead time) are more responsive to improved scientific understanding, while others (POD) are more dependent on technology upgrades.