1. Subjective Probability Forecasts at the NWS Storm Prediction Center Dr. Russell S. Schneider Steven J. Weiss NWS Storm Prediction Center Model Development Laboratory - December 2009 “Where America’s Weather and Climate Services Begin”

2. Outline Motivation Background SPC Probabilistic Products SPC Unique Guidance & Forecast Tools Advanced Verification Current Development Challenges

3. NOAA NWS Storm Prediction Center Forecast tornadoes, thunderstorms, and wildfires nationwide Forecast information from 8 days to a few minutes in advance World class team engaged with the research community Partner with over 120 local National Weather Service offices

4. Each Year: About 1,500 tornadoes touch down across the United States causing an estimated $1.1 billion in damages and over 80 deaths

5. Motivation for Probabilities Capture forecaster’s assessment Quantify uncertainty inherent in a forecast Enable partner-specific risk assessment & decision support Improve depth & quality end-to-end of forecast process

6. Scientific Background Subjective Probability Forecasting (Sanders, 1963) Probability is proper internal language of forecasters Forecasters use: Sorting Process & Labeling Process Murphy and Collaborators (1974-1995) Early work with SPC (NSSFC) Forecast Process Murphy & Winkler (1982): SPC Tornado Watches in 1976-1977 Murphy et al. (1993): Experimental Convective Outlooks Other notable papers Murphy (1991): Probabilities, Odds, and Forecasts of Rare Events Murphy (1993): Essay on the Nature of Forecast Goodness –Need to effectively capture forecaster judgment –Avoid forecaster product-embedded estimates of user cost-loss model

7. Basic Methodology Guidance & Forecast Tools Forecaster Experience (rare events) SPC Specialists for Severe Convection Jack Hales: Lead for 35 years = over 1/5 th Conceptual models important with limited data Capture Forecaster Information Communicate Information (multi-tier) Timely & Effective Feedback

8. SPC Probabilistic Products

9. Tornadoes, Hail & Wind Fire weather (Day 1- 8) Winter weather Excessive rainfall Storm Prediction Center Hazardous Phenomena

10. “Super Tuesday” Outbreak 5 February 2008 56 Fatalities - 15 th Largest Death Toll since 1950

11. Day 4-8 Convective Outlook Operational since March 2007 30% Probability of an Event within 25 mi of a point (“high end” Slight Risk) “Potential too Low” <30 % Probability “Predictability too Low” Widespread severe weather possible, but too much uncertainty to forecast an area Graphic & Short Discussion Issued at 4:00 am Central

12. Day 3 & Day 2 Convective Outlook Operational since: 2001 & 1986 (categorical); 2001 & 2000 (probabilistic) Probability of Severe Weather within 25 miles of a point Hatched Area --- 10% or greater chance of an extreme event (EF2 or greater tornado, 2” or larger hail, 65 kt or faster gust) Graphic & Discussion Valid for 24 hour period 1200 UTC – 1200 UTC Issued at 2:30 am Central Issued at 1:00 am Central & updated at 1730 UTC DAY AFTER TOMORROW TOMORROW

13. Day 1 Convective Outlook Operational since 1955 (catagorical) & 2000 (probabilistic) Probability of individual hazards within 25 miles of a point Hatched Areas --- 10% or greater chance of an extreme event (F2 or greater tornado, 2” or larger hail, 65 kt or faster gust) Issued at 0600 UTC; updated at 1300, 1630, 2000, 0100 UTC Valid for period ending at 1200 UTC Categorical Risk Hail Probability Tornado Probability Wind Probability

14. Reliability of Probabilistic Day 1 Outlooks 2008 Forecast Obs 10% Sig Torn 7.2% 10% Sig Gust 4.5% 10% Sig Hail 7.6%

15. Day 2 Forecast Observed 10% Prob. Sig 13.3% Convective Outlook Probabilities ROC Charts The concaveness of the ROC curve wrt to the diagonal indicates the value the forecast Day 3 Forecast Observed 10% Prob. Sig 9.1% POD = fraction of events that were correctly forecast FAR = fraction of nonevents that were incorrectly forecast

16. Relationship between Categorical and Probabilistic Outlooks

17. Watch Hazard Probabilities All watches are not created equal Use the probabilities to refine the specific threat within the watch and gauge forecaster confidence in the specific convective hazards Operational since 2006

18. Reliability of Watch Hazard Probabilities Problem with “significant wind gusts … Observational issue, few measurements… Most estimated from damage (e.g., only 193 reports in 2008)

19. 2008 Watch Hazard Probability Verification Worst Case Best Case POD = fraction of events that were correctly forecast FAR = fraction of nonevents that were incorrectly forecast

20. Thunderstorm Outlooks to Support Aviation 1300UTC 9 April 2009 1300Z 9 April 09 VT 16Z-20Z 1300Z 9 April 09 VT 00Z-04Z 1300Z 9 April 09 VT 20Z-00Z 0600Z 9 April 09 VT 20Z-00Z 0600UTC 9 April 2009 0600Z 9 April 09 VT 16Z-20Z 0600Z 9 April 09 VT 00Z-04Z Observed Cloud-to-Ground Lightning 16Z-20Z 20Z-00Z 00Z-04Z

21. SPC Scientific Approach Engage the Community (SSB & Forecasters) Professional Meetings and Conferences Collaborative Research Projects Hazardous Weather Testbed Sift & Winnow Eye toward specialized mission specific data & tools Hazardous Weather Testbed (key intermediate step) Early integration of ensemble forecast systems Storm-resolving WRF NWP Storm-scale Ensemble Forecast System SSB focus on Science Enabling “Structure” Flexible. Adapt to specific trends as path becomes clear Multiple dimensions for success and synergy

22. SPC Science Support Branch Foci Collaborative Research HWT Spring Experiments NSSL, OU, EMC, NCAR, GSD, U. of Albany, Pacific NW Fire Lab Steve Weiss, David Bright, Phillip Bothwell, Jason Levit, Chris Siewert Forecast Tools & Guidance Regional Scale NWP Information Extraction (e.g. SREF guidance) Storm Scale NWP Information Extraction (e.g. storm scale ensemble) Interactive Forecast Tools (e.g. NSHARP) David Bright, Phillip Bothwell, Jason Levit & Steve Weiss Forecast Product & Storm Environment Database SPC Administrative & Context Based Verification Understanding Storms & Supporting Forecast Tools Andy Dean, Phillip Bothwell, Greg Carbin additional support: Gregg Grosshans, Jay Liang, Joe Byerly

23. Guidance & Forecast Tools

24. Forecaster GUI for Probabilities … climatology as a reference constrained by “policy” and based on climatology

25. Perfect Prog. Lightning Forecasts including Dry Thunderstorm Potential (Bothwell, 2008)

26. NOAA Hazardous Weather Testbed EFP EWP GOES-R PG Experimental Forecast Program Experimental Warning Program Prediction of hazardous weather events from a few hours to a week in advance Detection and prediction of hazardous weather events up to several hours in advance

27. Key HWT Contributions to Advances in SPC Operations Short Range Ensemble Forecast System (SREF) SREF Workshop (NCEP 1994) Collaboration on SREF with EMC & NSSL (1999-2002) SPC focused SREF guidance in Operations (2003) Developing additional calibrated high impact wx. guidance Global Ensemble Forecast System (GEFS) SPC Focused GEFS Guidance in Operations (2005) Working toward NAEFS tools (2010) Convection Allowing High Resolution NWP Experimental WRF data in Operations (2004) Operational WRF data in Operations (2008)

28. Example of 39 Hour SREF Calibrated Thunderstorm Forecast Valid: 03-06 UTC 03 May 2008 Bright et al. 2005

29. Example of 39 Hour SREF Calibrated Thunderstorm Forecast Valid: 03-06 UTC 03 May 2008 Bright et al. 2005

30. Logistic Regression applied to sounding parameters, yielded the following significant predictors: 1) maximum bulk shear (m/s) in the 0-1 and 6-10 km layer 2) 3-8 km lapse rate (degrees C/km) 3) most unstable CAPE 4) 3-12 km mean wind speed (m/s) Probability of MCS Maintenance Coniglio et al. 2006

31. Post-Processed Short Range Ensemble Forecast Calibrated Severe Thunderstorm Probability Calibration – Computed from conditional probability of Severe Weather given a Thunderstorm Bright & Wandishin 2006

32. Wide range of high impact SREF guidance are available at the SPC website http://www.spc.noaa.gov/exper/sref/

33. MDL has a Distinguished History of Probabilistic Forecast Guidance Pioneering work documented in formal publications Charba (MWR 1979) Reap and Foster (JAM 1979) Technical Reports & Conferences with SPC R. M. Reap, D. S. Foster, and S. J. Weiss, 1981 R. M. Reap, D. S. Foster, and S. J. Weiss, 1982 S. J. Weiss and R. M. Reap, 1984

34. SPC Day 3 Probabilistic Outlook for Tuesday 8 December Day 3 SPC Outlook valid 12Z 8 Dec. 24hr Unconditional Severe Thunderstorm Probability (within 25 mi of point) Preliminary Reports valid 12Z 8 Dec. 24hr Compilation 12Z-12Z

35. NAM & GFS based Severe Thunderstorm Probabilities NAM MOS: 72hr Projection valid 12Z 8 Dec. 24hr Unconditional Severe Thunderstorm Probability GFS MOS: 72hr Projection valid 12Z 8 Dec. 24hr Unconditional Severe Thunderstorm Probability

36. SREF based Calibrated Severe Thunderstorm Probabilities SREF 69hr Projection valid 12Z 8 Dec. 24hr Calibrated Unconditional Severe Thunderstorm Probability

38. We need data … robust verification In a World Without Data, Opinion Prevails We tend to research the atmosphere, but not ourselves From Scholtes (1998) “The Leader’s Handbook”

39. Context Based Verification Support a dynamic learning environment Provide insights into tactical & strategic challenges

40. Data & Methodology: –Severe weather reports for 2003-2008 (2009 through August) –Environmental estimate from hourly SPC - RUC meso-analysis –Extract from nearest grid point at hour prior to event –Link report and estimated environment databases by event time & location 175,658 reports | 8334 tornadoes

41. High CAPE - Strong Shear Low CAPE - Strong Shear Severe Reports Environment Hours ~50hr/yr ~100hr/yr 24 % of F2+ tornadoes local axis ~ 40 hr / year 39 % of F2+ tornadoes widespread ~ 80 hr / year

42. All Tornadoes (total of 6643) 0-6 km Shear 100 mb ML CAPE Thunderstorms Common Significant Tornadoes 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1

43. 0-6 km Shear 100 mb ML CAPE Conditional Probability of Tornadoes (given lightning) 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1

44. 0-6 km Shear 100 mb ML CAPE All Watches: Probability of Tornado Detection 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1 Overall POD: 0.69; F2+: 0.94

45. 0-6 km Shear 100 mb ML CAPE Tornado Watch: Probability of Tornado Detection 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1 Overall POD: 0.58; F2+: 0.83

46. 0-6 km Shear 100 mb ML CAPE Tornado Watch: Probability of Tornado Detection 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1 Overall POD: 0.58; F2+: 0.83 > 60%

47. 0-6 km Shear 100 mb ML CAPE Tornado Watch: Fraction of Missed Tornadoes 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1

48. 0-6 km Shear 100 mb ML CAPE Tornado Watch: False Alarm Area 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1

49. 0-6 km Shear 100 mb ML CAPE Tornado Watch: False Alarm Area 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1

50. 0-6 km Shear 100 mb ML CAPE Tornado Watch: Fraction of Missed Tornadoes 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1

51. Regional Tornado Variability 0-6 km Shear 100 mb ML CAPE High False Alarm Area High Tornado POD (in tor) Overall POD: 0.58; F2+: 0.83

52. All Tornadoes 2003-2008 (total of 9178) 0-6 km Shear 100 mb ML CAPE 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1

53. All Central Region Tornadoes (total of 4369) 0-6 km Shear 100 mb ML CAPE 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1

54. All Southern Region Tornadoes (total of 3622) 0-6 km Shear 100 mb ML CAPE 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1

55. All Eastern Region Tornadoes (total of 981) 0-6 km Shear 100 mb ML CAPE 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1

56. All Western Region Tornadoes (total of 205) 0-6 km Shear 100 mb ML CAPE 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1

57. Tornado Variability (Tornado in Tornado Watch) 0-6 km Shear 100 mb ML CAPE High False Alarm Area High Tornado POD (in tor) Overall POD: 0.58; F2+: 0.83

58. Tornadoes: 2003 (1484 tornado segments) 0-6 km Shear 100 mb ML CAPE 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1 Overall POD: 0.61; F2+: 0.90

59. Tornadoes: 2004 (1941 tornado segments) 0-6 km Shear 100 mb ML CAPE 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1 Overall POD: 0.57; F2+: 0.82

60. Tornadoes: 2005 (1327 tornado segments) 0-6 km Shear 100 mb ML CAPE 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1 Overall POD: 0.60; F2+: 0.77

61. Tornadoes: 2006 (1255 tornado segments) 0-6 km Shear 100 mb ML CAPE 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1 Overall POD: 0.55; F2+: 0.82

62. Tornadoes: 2007 (1614 tornado segments) 0-6 km Shear 100 mb ML CAPE 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1 Overall POD: 0.60; F2+: 0.84

63. Tornadoes: 2008 (1950 tornado segments) 0-6 km Shear 100 mb ML CAPE 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1 Overall POD: 0.70; F2+: 0.82

64. Tornadoes: 2009 (Jan-Aug) (1130 tornado segments) 0-6 km Shear 100 mb ML CAPE 20 ms -1 10 ms -1 30 ms -1 40 ms -1 50 ms -1

65. Summary Tornado forecast improvement needed for marginally supportive environments Low CAPE – High Shear Environments –low conditional probability, very high potential impact –distinct challenges for communication and societal response Modest CAPE – Modest Shear Environments Dimensions of Forecast Skill Variability Regional & Annual Variation in Environments Supports queries to explore research and operational questions Provides an initial estimate of forecast difficulty

66. New Guidance

67. NOAA Hazardous Weather Testbed EFP EWP GOES-R PG Experimental Forecast Program Experimental Warning Program Prediction of hazardous weather events from a few hours to a week in advance Detection and prediction of hazardous weather events up to several hours in advance

68. HWT strategy is to engage wider range of community Testbed R20 Process pathway for operational assessment and feedback SPC-NSSL HWT

69. Forecast Focused Community Collaboration

70. Collaboration has grown steadily since 1997 Focus on making an experimental forecasts Forecasters, Researchers, Numerical Modelers Community contributed experimental forecasts NOAA (NSSL, ESRL, NCEP) University of Oklahoma (CAPS) NCAR, AFWA, UK Meteorological Office Collaborations have spurred community momentum

71. We are rapidly approaching a new era in numerical weather prediction High-resolution models can provide details important to prediction of high impact weather events 21 hour forecasts for a tornado outbreak on 15 Nov 2005 12 km North American Model 3 hour precipitation Experimental 4.5 km WRF-NMM Radar Reflectivity 2005 Capability

72. Ensembles: Explore & Define Uncertainty ARW – CN ARW – C0 NMM – CN NMM – C0 ARPS – CN ARPS – C0 Observed Radar Reflectivity 18 hr Forecast - 1 km “Radar” HWT 2009 Spring Experiment

73. Probability Updraft Helicity > 50 m 2 /s 2 Supercell Forecast (26 hour) Radar BREF 0142 UTC 22 Apr 2008 F026: Valid 02 UTC 22 Apr 2008 Updaft Helicity > 50 m 2 s -2 within 25 miles Supercell Observed

74. Probability Updraft Helicity > 50 m 2 /s 2 Looking south from Norman, OK (0145 UTC 22 Apr 2008) (Numerous large hail reports up to 2.25 inch) Jack Hales

75. Vision: Warn-on-Forecast NSSL Warn on Forecast Briefing March 5, 2007 The warning issued based on the computer model forecast(s) includes detailed information on the expected event Radar and Initial Forecast at 2100 CSTRadar at 2130 CST: Accurate Forecast Most Likely Tornado Path T=2120 CST T=2150 T=2130 T=2140 70% 50% 30% T=2200 CST Developing Thunderstorm Most Likely Tornado Path T=2120 CST T=2150 T=2130 T=2140 T=2200 CST An ensemble of storm-scale NWP models predict the path of a potentially tornadic storm. A Warning is Issued! 70% 50% 30% Triple current Tornado Warning lead times Support enhanced community response

76. Aviation Impacts from Thunderstorms Observed Aviation impacts 7:00 pm 18 April 2008 26 hour Forecast of a Squall Line at 7:00 pm 18 April 2008 Hazardous Weather Testbed Experimental Forecast Adverse weather costs approximately $30B / year Thunderstorms cause two-thirds of all delays Support improved airspace management

77. Challenges Service & Societal Challenges Effective communication of threat & uncertainty Create a continuous decision support hazardous weather information stream Major Science Challenges the convective initiation challenge impacts of mesoscale variability on storm predictability appropriate NWP resolutions for severe storm prediction advanced cloud microphysics & BL for storm scale NWP radar (& other) data assimilation for storm scale NWP extraction of storm information from high resolution NWP perturbation strategies for storm scale ensembles extraction of probabilistic information from storm scale ensembles

78. New Software Tools

79. Real Time Queries to Storm DB

80. Ensemble Vertical Profiles

81. New Verification Support a dynamic learning environment Provide insights into tactical & strategic challenges

82. Strategic Verification e.g. “Roebber” diagrams WAF (2008): Visualize Multiple Dimensions of Quality Frequency of Hits (1-FAR)

83. Daily Context Based Verification

84. 70 % of all tornadoes (2006) 89 % of F2+ tornadoes 13 % of tornadoes (2006) 10 % of F2+ tornadoes All Supercells (2006) Weak Storm Organization (2006) QLCS & Bows - no Supercells (2006) All “Significant” Reports (2006)

85. Future DB & Verification Work Add storm attributes supercell, bow echo, QLCS Also applicable to NWS Warning problem Automated storm categorization algorithms in future Improve near Real Time Forecaster Feedback Refine measures of forecast difficulty Provide interactive forecaster access to database Extend using climate reanalysis Both Strategic & Tactical (Forecaster) Efforts

86. Forecast Information Decision Support Services

87. Watch Hazard Probabilities All watches are not created equal Use the probabilities to refine the specific threat within the watch and gauge forecaster confidence in the specific convective hazards Operational since 2006

88. Current Internet Test for Public Watch… Hazard Probability Based Message (CTA) URGENT - IMMEDIATE BROADCAST REQUESTED SEVERE THUNDERSTORM WATCH NUMBER 266 NWS STORM PREDICTION CENTER NORMAN OK 455 AM CDT FRI MAY 8 2009 THE NWS STORM PREDICTION CENTER HAS ISSUED A SEVERE THUNDERSTORM WATCH FOR PORTIONS OF NORTHERN ARKANSAS SOUTHEAST KANSAS SOUTHERN MISSOURI NORTHEAST OKLAHOMA EFFECTIVE THIS FRIDAY MORNING FROM 455 AM UNTIL NOON CDT....THIS IS A PARTICULARLY DANGEROUS SITUATION... STORMS WITHIN THE WATCH AREA ARE EXPECTED TO PRODUCE WIDESPREAD DAMAGING WINDS... WITH WIND GUSTS TO 80 MPH LIKELY. WIDESPREAD LARGE HAIL TO 2.0 INCHES IN DIAMETER. ISOLATED TORNADOES. DANGEROUS LIGHTNING WILL ALSO ACCOMPANY THESE STORMS. REMEMBER...A SEVERE THUNDERSTORM WATCH MEANS CONDITIONS ARE FAVORABLE FOR SEVERE THUNDERSTORMS IN AND CLOSE TO THE WATCH AREA. PERSONS IN THESE AREAS SHOULD BE ON THE LOOKOUT FOR THREATENING WEATHER CONDITIONS AND LISTEN FOR LATER STATEMENTS … A Derecho is Forecast

89. Current Internet Test for Public Watch … Hazard Probability Based Message (CTA) URGENT - IMMEDIATE BROADCAST REQUESTED SEVERE THUNDERSTORM WATCH NUMBER 266 NWS STORM PREDICTION CENTER NORMAN OK 455 AM CDT FRI MAY 8 2009 THE NWS STORM PREDICTION CENTER HAS ISSUED A SEVERE THUNDERSTORM WATCH FOR PORTIONS OF NORTHERN ARKANSAS SOUTHEAST KANSAS SOUTHERN MISSOURI NORTHEAST OKLAHOMA EFFECTIVE THIS FRIDAY MORNING FROM 455 AM UNTIL NOON CDT....THIS IS A PARTICULARLY DANGEROUS SITUATION... STORMS WITHIN THE WATCH AREA ARE EXPECTED TO PRODUCE WIDESPREAD DAMAGING WINDS... WITH WIND GUSTS TO 80 MPH LIKELY. WIDESPREAD LARGE HAIL TO 2.0 INCHES IN DIAMETER. ISOLATED TORNADOES. DANGEROUS LIGHTNING WILL ALSO ACCOMPANY THESE STORMS. REMEMBER...A SEVERE THUNDERSTORM WATCH MEANS CONDITIONS ARE FAVORABLE FOR SEVERE THUNDERSTORMS IN AND CLOSE TO THE WATCH AREA. PERSONS IN THESE AREAS SHOULD BE ON THE LOOKOUT FOR THREATENING WEATHER CONDITIONS AND LISTEN FOR LATER STATEMENTS … A Derecho is Forecast Add Societal Dimensions

90. U.S. Population Density Data (persons km -2 ) 2000 Census: 30 arc sec grid (calculations on 1km grid)

91. May 5, 2007 Tornado High Risk 354,381 people in risk area 1.7% of area has pop >10/km 2 0.6% of area has pop >100/km 2 Feb 5, 2008 Tornado High Risk 2,826,940 people in risk area 22.7% of area has pop >10/km 2 4.1% of area has pop >100/km 2 Hypothetical Tornado High Risk 9,893,380 people in risk area 70.7% of area has pop >10/km 2 14.6% of area has pop >100/km 2

92. Some Current SPC Initiatives Refine and Expand Public Weather Outlook Add criteria based on potential societal impact Overnight violent tornado threat (began 1 December) Refine message to focus response & improve dissemination Work with partners to identify additional needs Refine Convective Watch Suite (FY11) Hazard probability based wording to highlight threat Refine focus on partner & customer needs Adapt HWT for Diverse Challenges Aviation, Cool Season Severe, Regional Hazards Science, Forecast Tools, Services & Society

93. Continuing SPC Challenges Services to Support Improved Decisions continuous stream of probabilistic hazard information –rapid update, enhanced-time-resolution outlooks –evolving convective watches with partner driven attributes –support and seamlessly blend with local office & partner services –support cascade to Warn-on-Forecast (Warn-on-Detection) distinguish forecast information from service delivery Adapt services to evolving needs –high impact weather (severe, fire, aviation, convection, OCONUS) Adapt staff to evolving science & business model Create a dynamic and sustained learning environment

94. www.spc.noaa.gov Russell.Schneider@noaa.gov