Pablo Santos WFO Miami, FL Mark DeMaria NOAA/NESDIS David Sharp WFO Melbourne, FL 2010 AMS Annual Meeting, Atlanta, GA PS/MM/DS Determining Optimal Thresholds of Tropical Cyclone Incremental Wind Speed Probabilities to Support Expressions of Uncertainty within Text Forecasts

2 Probability Definitions Cumulative – The probability of sustained winds reaching or exceeding the specified wind speed threshold between the 00 hour forecast and a specified forecast hour. Available in Grid (NDFD)/Graphic Forms as well as within PWSAT# product. Individual (Interval) – The probability of sustained winds reaching or exceeding the specified wind speed threshold beginning during a specified forecast period ending at the specified forecast hour (e.g., period of onset). Computed from Cumulative and explicitly available in PWSAT# text product. Incremental –The probability of sustained winds reaching or exceeding the specified threshold during a specified forecast period ending at the specified forecast hour. Available in Grid (NDFD) but not within the PWSAT#. These are the probabilities referenced throughout this presentation. (A Quick Review)

3 Hypothetical Test Case Wind Impacts Are A Concern Well Outside Of The Error Cone 3 Based on 1938 Long Island Express Hurricane - a large fast-moving storm Tropical Storm Force wind probabilities are almost certain in a large area from the Carolinas to New England well outside the cone!

4 During potential high impact weather events such as tropical cyclones (TC), users not only demand a WFO’s ‘best forecast’, but also require a corresponding expression of uncertainty for decision- making purposes. Therefore, a methodology was developed that uses TC Incremental Wind Speed Probabilities to elevate the utility of official text forecast products: ZFP CWF –By including situational expressions of uncertainty within the Zone Forecast Product (ZFP) and the Coastal Waters Forecast (CWF) which responsibly ‘scopes’ NHC’s forecast (temporally and spatially) for use within local forecasts. For WFOs, this effort crosses a major threshold for incorporating probabilistic data (other than PoP) within their text forecast products. Proposed WFO Use Proposed WFO Use (Item from NOAA Hurricane Conference, 2005)

5 (The Nine Expressions of Uncertainty) HURRICANE CONDITIONS * HURRICANE CONDITIONS EXPECTED HURRICANE CONDITIONS POSSIBLE TROPICAL STORM CONDITIONS * TROPICAL STORM CONDITIONS * WITH HURRICANE CONDITIONS POSSIBLE TROPICAL STORM CONDITIONS EXPECTED TROPICAL STORM CONDITIONS EXPECTED WITH HURRICANE CONDITIONS POSSIBLE TROPICAL STORM CONDITIONS POSSIBLE TROPICAL STORM CONDITIONS POSSIBLE WITH HURRICANE CONDITIONS ALSO POSSIBLE The * phrases are equivalent to imminent or ongoing conditions as reflected in the hour period simultaneously by the hazard, wind, and probability grids. The word EXPECTED is used in situations mainly during the Warning period, but is also selectively used in the Watch period (e.g., Day 1 & 2, or out to period 4). The word POSSIBLE is used during the extended period (e.g., Day 3 - 5), but is also used in specific situations within Watch/Warning period. Also, certain situations require compound phrases (e.g., when there is a Tropical Storm Warning with a Hurricane Watch). Baseline Phraseology

6 Determining Optimal Probability Thresholds NHC verification program evaluates gridded products. In this case as it pertains to WFO Forecast applications. –Sample: 19 storms cases with U.S. hurricane warnings along East and Gulf of Mexico Coasts from Brownsville, TX to Portland, Maine. All probability runs starting 3 days before 1 st warning. –2008 MC model re-runs for 400 forecast cases. Evaluated probabilities at 343 coastal “breakpoints” as well as 286 inland points (approx. 50 km inland). –Use Threat Score or ROC diagrams to select optimal probability thresholds. Emphasis placed on 12 h incremental probabilities. Results for 34 & 64 knot probabilities were based on –6 hourly NHC best track wind radii interpolated every hour to produce the model verification grid (used to determine points with observed hurricane/tropical storm force winds).

2010 Wind Speed Probability Model Track and Intensity error statistics updated with previous 5 year sample – Track error distributions stratified by Goerss Predicted Consensus Error (GPCE) –GPCE estimates track forecast uncertainty from model spread and other forecast parameters GPCE version implemented at NHC in 2010 –Includes option to run without GPCE input 2010 model (GPCE and non-GPCE versions) run for all WFO test cases –Coastal and inland points 7

8 Storm Cases For 400 MC Model Runs Coastal Points Inland Points

9 Conversion of Probabilities into Yes-No Forecast Pick probability threshold (P t ) If P ≥ P t, forecast YES If P < P t, forecast NO Best choice of P t depends on particular application Use appropriate validation metric to select P t objectively Metrics determined from contingency table

10 Contingency Table for Validation of Yes-No Forecasts Observed Yes No Yes Forecast No a b c d Threat Score (TS) = a/(a + b + c) Hit Rate (HR) or Probability of Detection (POD) = a/(a + c) False Alarm Rate (FR) = b/(b + d)

11 Regional Stratification Estimate optimal probabilities using Threat Score –ROC score probabilities too low Region 1 –Brownsville, TX to Mobile, AL Region 2 –Mobile, AL to GA/SC Border Includes all Florida breakpoints Region 3 –GA/SC Border to Eastport, ME

12 Optimal 34 knot Probability Thresholds (Coastal vs. Inland) Based on Threat Score…collectively, and by each region.

13 Optimal 64 knot Probability Thresholds (Coastal vs. Inland) Based on Threat Score…collectively, and by each region.

14 Optimal 34 & 64 knot Probability Thresholds (Combined) Based on Threat Score…by region, for all points.

15 Associated Reliability Diagrams (Coastal vs. Inland)

16 Associated Reliability Diagrams (Combined)

2010 Evaluation of GPCE input: Reduction in Brier Score 17

18Summary Optimal Probability Threshold curves decrease almost exponentially with time (for both 34 & 64 knot). Optimal thresholds are correspondingly larger for inland points, particularly in the shorter ranges (e.g., watch/warning periods). Optimal thresholds are a function of geographical location: –More so for 34 knot probabilities, in the short range, and for coastal points. Overall, probabilities are deemed to be “reliable” although: –They tend to under predict 64 knot or greater events particular for coastal points. For all points combined, this bias is smaller. Results indicate that the empirically-derived thresholds (original; based on histograms) need an upward adjustment in the short range to better support operational use.

19Summary That there is inherent operational sensitivity regarding event-specific reliability and continuity (forecast-to- forecast, region-to-region, coastal-to- inland, etc.), That the current ZFP/CWF text formatters will only support one set (each) of 34 and 64 knot thresholds, And that newer Monte Carlo based results by region are still forthcoming which will also include more cases, …the following (interim) values to the right are the ones currently in effect for operational use. PeriodPWS64PWS hr30%55% hr25%45% hr20%40% hr15%35% hr10%30% hr7%25% hr6%20% hr5%15% hr4%12.5% hr3%10% So Given this and… Proposed Single Set Values

20 Future Work Iteratively transition optimal thresholds to ZFP and CWF text formatters, as well as point-n-click web forecast code. –single set –multi-set coastal vs. inland by region Complete verification using 2010 version of the Monte Carlo model by regions.

21 END

22 Discussion Items to Official Implementation of EoU Problems Encountered During 2010 Season: –ER problem with phrases dropping Wx in presence of EoU. Investigating. –Offices editing probability grids and questioning EoU with winds of 10 to 15 mph. Training Issue. –However, technical team suggests we can consider dropping the phrase if winds less than 15 kts and there is no warning. Iteratively transition optimal thresholds to ZFP and CWF text formatters, as well as point-n-click web forecast code. –single set, multi-set, coastal vs. inland, by region

23 Discussion Items to Official Implementation of EoU Back Up sites will need training to be aware of this capability. Configuration wise it should not be a problem. Public and Marine Directives will need to be updated to include set of rules that prompts EoU. Work on instructions for field offices to enable Tropical SAF Formatter. MIA and MLB already have it. Baseline code is the same as the ZFP. It is a matter of configuration. RECOMMENDATION: Official or Not? Reconvene Wind Team to oversee all issues above are taken care of whether it is by 2011 or 2012.

24 BACK UP SLIDES

25REFERENCES DeMaria, M., J. A. Knaff, R. Knabb, C. Lauer, C. R. Sampson, and R. T. DeMaria, 2009: A new method for estimating tropical cyclone wind speed probabilities. Wea. Forecasting, 24, Santos, P., D. Sharp, M. DeMaria, and S. Kiser, 2009: The Determination of Optimal Thresholds of Tropical Cyclone Incremental Wind Speed Probabilities to Support Expressions of Uncertainty in Text Forecasts. Preprints, Symposium on Urban High Impact Weather/89 th AMS Annual Meeting, P1.5, Phoenix, AZ. [Available online Santos, P., D. W. Sharp, M. Volkmer, and G. Rader, 2008: Employing Hurricane Wind Probabilities to Convey Forecast Uncertainty and Potential Impact Through NWS Field Office Forecast Products. Preprints, Tropical Meteorology Special Symposium/19 th Conf. on Prob. And Statistics, JP1.3, New Orleans, LA. [Available online Sharp, D. W., P. Santos, M. Volkmer, G. Rader, and M. Sardi, 2006: Employing Hurricane Wind Probabilities to Enhance Local Forecasts and Improve Guidance for Decision-Makers. Preprints, 27th Conf. on Hurricanes and Tropical Meteorology, 9A.3, Monterey, CA. [Available online

26 Wind Impacts Are A Concern Well Outside Of The Error Cone 26 Based on 1938 Long Island Express Hurricane - a large fast-moving storm Hurricane-force winds extended outside the cone over much of NJ, srn NY, LI, and CT Hypothetical Test Case

27 Storm Cases For 400 MC Model Runs

28 Optimal 34 and 64 kt Probability Thresholds based on Threat Score and ROC Diagram for Coastal Points

29 Note: Go to weather.gov Click on Graphical Forecasts Scroll to the Bottom Click on Tropical Notice both Cumulative and Incremental Probabilities are available here Notice the Resolution What About Interval or Individual Probabilities? Tropical Cyclone Wind Speed Probabilities Through NDFD

30 How Can I get the grids from the NDFD Download DeGrib (NDFD GRIB2 Decoder from): Then go to this site and read on how turn the NDFD gridded data into GIS Files (Creating GIS files from the downloaded NDFD grids):

31 P t Optimization Procedures Threat Score –Threat Score useful for low probability events since it does not include No-No cases –TS ranges from 0 to 1 (1 is best) –Pick P t that maximizes TS Hit Rate/False Alarm Rate –Plot HR vs. FR in 2-D plane for range of P t Relative Operating Characteristic (ROC) diagram –Pick P t for point closest to upper left corner in ROC diagram Upper left corner has Hit Rate=1, False Alarm Rate=0

32 Hurricane Charley - Update

33 Hurricane Charley (ZFP) CHARLEY (ZFP) * Punta Gorda Advisory Time: _1500 (0-12 Hours – First Period) A Second Algorithm Example

34 Fay – WFOs MFL and MLB