1. Overview of SPC Efforts in Objective Verification of Convection-Allowing Models and Ensembles Israel Jirak, Chris Melick, Patrick Marsh, Andy Dean and Steven Weiss Storm Prediction Center

2. Background SPC began looking at near real-time objective verification of simulated reflectivity forecasts from convection-allowing models (CAMs) during the 2012 HWT Spring Forecasting Experiment (SFE2012). Compared subjective evaluation of forecast skill to objective grid-point and spatial neighborhood metrics. Neighborhood metrics were overwhelmingly favored by SFE2012 participants in agreeing better with the perceived quality of reflectivity forecasts. We developed a year-round framework to examine objective neighborhood statistics in near-real time for forecaster look-back at model performance and for evaluating parallel CAMs.

3. CAM Neighborhood Approach NSSL-WRFNMQ OBS Find the neighborhood maximum reflectivity within a 40 km radius. Hit = 32 Hit = 11,555 Get credit for “near hits” @ 40 dBZ

4. CAM Neighborhood Metrics NSSL-WRFNMQ OBS Still calculate traditional contingency table statistics (i.e., POD, FAR, Bias, CSI, etc.) on neighborhood fields By applying a 2-D Gaussian smoother (with   40 km) to both model and obs (e.g., Sobash et al. 2011, Marsh et al. 2012), get a probabilistic field (purple lines – 40 dBZ) from which Fractions Skill Score (FSS; Roberts and Lean 2008; Schwartz et al. 2010; Sobash et al. 2011) can be calculated. FSS = 0.806 90

5. CAM Neighborhood Statistics April 1 – July 15, 2015 : National Reflectivity Stats for 40-km Neighborhood NSSL-WRF (blue) has the highest CSI for all dBZ thresholds. Although it also has largest bias, the POD is much higher with little increase in FAR. The operational models (NAM Nest & HRWs) now all have similar statistical characteristics after upgrades in 2014. The WRF-NMM (red; “SPC run”) has the lowest CSI at all dBZ thresholds with a high bias. FSS 0.447 0.406 0.381 0.439 0.432

6. Ensemble Neighborhood Probability SSEO NMQ OBS Find percentage of ensemble members that exceed a specified threshold within a given radius of influence. Apply a 2-D Gaussian smoother to account for uncertainty and to produce a refined probabilistic field. ROI = 40 km  = 10 gp = 40 km

7. FSS 0.563 0.562 0.541 0.549 0.546 Ensemble Neighborhood Metrics Calculate FSS for ensemble neighborhood probability of reflectivity ≥40 dBZ verified with observed radar (using Gaussian smoother as applied before). During SFE2015, all convection-allowing ensembles had similar FSS during the peak of the convective cycle (20- 02Z); FSS for SSEO was more uniform from f13-f30 Ensemble Probability of Reflectivity ≥40 dBZ Observed “Probability” of Reflectivity ≥40 dBZ