1. Spatial Verification Methods for Ensemble Forecasts of Low-Level Rotation in Supercells Patrick S. Skinner 1, Louis J. Wicker 1, Dustan M. Wheatley 1,2, and Kent H. Knopfmeier 1,2 1 - NOAA/National Severe Storms Laboratory 2 - Cooperative Institute for Mesoscale Meteorological Studies, University of Oklahoma Thanks To: Thomas Jones, Corey Potvin, Gerry Creager, Jeff Snyder, and Chris Karstens National Weather Association 40th Annual Meeting

2. “Next Day” May 31st, 2013 6-18 hour forecast probability swath of updraft helicity (shading) and SPC storm reports (yellow markers) Goal is to verify forecasts of regions of thunderstorms Wheatley et al. 2015 Schwartz et al. 2015 Convection allowing ensemble forecast with mesoscale data assimilation: Convection allowing ensemble forecast with storm-scale (radar) data assimilation: “Next Hour” May 31st, 2013 0-1 hour forecast probability swath of vertical vorticity (shading), reflectivity (contours), and El Reno, OK tornado track (blue outline) Goal is to verify forecasts of specific thunderstorms

3. Verification of Low-level Rotation Overall goal of verification is to reproduce subjective interpretation of ensemble low-level rotation forecasts Neither observations or forecasts resolve the hazard being forecast NEWS-e horizontal grid spacing is 3 km: Can partially resolve mesocyclones, but not storm-scale processes (i.e. cyclic mesocyclogenesis) (Potvin and Flora 2015) The predicted model fields cannot be directly verified (e.g. only have WSR-88D data) Requires development of proxies for tornado likelihood in both forecasts and observations Best available proxy for tornado occurrence in modeled supercells is the presence of a low-level (z<1km AGL) mesocyclone (Trapp et al. 2005) Best available proxy for low-level mesocyclone occurrence in WSR-88D data is low-level azimuthal wind shear (AWS)

4. Postprocessing Example: 20 May 2013, Moore, OK Supercell NEWS-e ζ (s -1 ) 25-min Forecast Valid: 20:10 Ens. Mem. 14 KTLX AWS (s -1 ) Valid: 20:10 KTLX azimuthal wind shear field interpolated to model grid Postprocessing method: Based on Method for Object- Based Diagnostic Evaluation (MODE; Davis et al. 2006a,b) Damage track of Moore Tornado Contour of 0.001 s -1 final KTLX AWS Step 1:

5. NEWS-e ζ (s -1 ) 25-min Forecast Valid: 20:10 Ens. Mem. 14 KTLX AWS (s -1 ) Valid: 20:10 Apply 3x3 grid point maximum value filter to spread information for both modeled vorticity and AWS fields Postprocessing Example: 20 May 2013, Moore, OK Supercell Step 2:

6. NEWS-e ζ (s -1 ) 25-min Forecast Valid: 20:10 Ens. Mem. 14 KTLX AWS (s -1 ) Valid: 20:10 Apply 5x5 grid point convolution filter to smooth information Postprocessing Example: 20 May 2013, Moore, OK Supercell Step 3:

7. NEWS-e ζ (s -1 ) 25-min Forecast Valid: 20:10 Ens. Mem. 14 KTLX AWS (s -1 ) Valid: 20:10 Threshold smoothed AWS field at 0.001 s -1 and restore unsmoothed values to grid points exceeding threshold Threshold smoothed ζ field at 0.003 s -1 and restore unsmoothed values to grid points exceeding threshold Postprocessing Example: 20 May 2013, Moore, OK Supercell Step 4:

8. NEWS-e ζ (s -1 ) 1945 - 2045 Forecast Ens. Mem. 14 KTLX AWS (s -1 ) 1945 - 2045 Rotation Track Values may be combined over time to create rotation tracks Note: A poorly performing ensemble member is shown to illustrate differences between observations and forecasts Postprocessing Example: 20 May 2013, Moore, OK Supercell Time Aggregation:

9. NEWS-e 1945 - 2045 Probabilistic Forecast KTLX AWS (s -1 ) 1945 - 2045 Rotation Track Rotation tracks for each ensemble member may be used to create a probabilistic rotation forecast Postprocessing Example: 20 May 2013, Moore, OK Supercell Ensemble Aggregation:

10. 20 May 2013: Subjective Verification Moore tornado genesis at ~2000 Improved probabilistic swaths in successive forecasts for Moore supercell Four 60-min NEWS-e forecasts issued at 15-min intervals prior to Moore tornado Increasing probabilities along the rotation track of the Moore supercell Lower probabilities along the tracks of the weakly tornadic supercell in the center of the domain and nontornadic southern supercell Forecast Time

11. 20 May 2013: Subjective Verification Orange: Model forecast information Dots represent vorticity maxima in individual members Contours are ensemble mean reflectivity of 20, 30, 40, and 50 dBZ Gray: WSR-88D information: Large dot is observed maximum in AWS Forecast accuracy degrades with increasing forecast time Strong bias in storm speed observed in all members Similar to many recent storm-scale data assimilation studies (e.g. Wheatley et al. 2015, Yussouf et al. 2015) 15-min NEWS-e forecast 45-min NEWS-e forecast

12. Object-Based Verification NEWS-e ζ (s -1 ) 25-min Forecast Valid: 20:10 Ens. Mem. 14 KTLX AWS (s -1 ) Valid: 20:10 Verification object Forecast object Each low-level mesocyclone isolated in forecast and verification fields is considered an object and is compared to all other objects in space and time

13. 20 May 2013: Object-Based Verification Total interest score (I ij ; Davis et al. 2006) is calculated for each pair of verification and forecast objects (“ij” represents obs/fcst object) 90% of weight is on proximity in space in time 10% of weight is on area difference between objects Match: Highest total interest between objects, provided I ij > 0.2 False Alarm: Unmatched forecast objects Miss: Unmatched verification objects 15-min NEWS-e forecast 45-min NEWS-e forecast 15-min NEWS-e Objects 45-min NEWS-e Objects

14. 20 May 2013: Object-Based Verification Object matches, false alarms, and misses can be used as quadrants in the standard contingency table (e.g. Wilks et al. 2006) Allows an Object-based Threat Score (OTS) to be calculated (Johnson et al. 2011) OTS scores reproduce subjective assessment of NEWS-e forecasts Scores improve with each successive forecast Scores decrease with increasing forecast time Results are more dramatic when subdomain containing only the Moore supercell (right panel) is considered Initial AWS maximum observed

15. Comparison Between Different Cases “Good”“Average” “Poor” OTS score can capture relative differences in quality between different forecasts El Reno, OK 2013 Moore, OK 2013 Nebraska, 2014 T-30 Min T-15 Min T-0 Min T-15 Min T-30 Min T-15 Min T-0 Min

16. Summary Object-based verification can reproduce subjective assessments of forecast quality for ensemble forecasts of low-level rotation Object characteristics, such as centroid distance, may be used to quantify specific forecast errors Summary object-based threat score can discriminate between relative forecast quality for different cases Long term goal is to develop automated, near real time capabilities for assessing Warn-on-Forecast system performance Quantify improvements resulting from changes in system configuration Quantify system performance for varying storm modes and environments patrick.skinner@noaa.gov

17. 20 May 2013: Object-Based Verification Characteristics of matched objects can be used to quantify specific errors Centroid distance between forecast and observed rotation objects in the Moore supercell captures positive bias in storm speed