HIC Meeting, 02/25/2010 NWS Hydrologic Forecast Verification Team: Status and Discussion Julie Demargne OHD/HSMB Hydrologic Ensemble Prediction (HEP) group UCAR

HIC Meeting, 02/25/ Databases Archive files CHPS Verification Service IVP EVS WR Water Supply Web FEWS Spatial Display FEWS Time Series Display Graphics Generator External data Verification ComponentDisplay/Product Generator Real-Time Verification (analogs, summary stats…) Verification Statistics & Products Forecast Products Interactive Forecasting System

HIC Meeting, 02/25/ RFC Contribution Input from RFCs is crucial to provide useful verification information to  modelers and forecasters to guide improvements of forecasting system  users to maximize utility of forecasts in their decisions Key questions:  What verification metrics/products are the most meaningful?  What are the best methods to supply verification information?

HIC Meeting, 02/25/ st Team Charter July 07: 1 st team charter to propose standard verification strategies (metrics, products, and verification analyses) by Sep August 07: 1 st RFC verification workshop November 08: 2 nd RFC verification workshop July-August 09: presentation at HIC meeting, feedback from HICs on 2 nd team charter draft (5 HICs replied)  Main feedback: include SCHs to develop meaningful products September 09: final team report on recommended standard verification strategies and future team activities (2 nd team charter)

HIC Meeting, 02/25/ nd Team Charter October 09: finalized 2 nd team charter for Oct Sep  Mission:  test verification standards (RFC case studies)  perform user analysis of products w/ SCHs and OCWWS/HSD and help develop requirements for dissemination  support design/development of CHPS Verification Service  Main deliverables:  report on improved verification standards and case studies done by all RFCs and by OHD/HSMB/HEP  prototype functionalities based on IVP, EVS, and CHPS to generate standard verification products

HIC Meeting, 02/25/ Status on 2 nd Team 2 meetings in Dec. 09 and Feb. 10:  presentation of OHD verification activities (FY10 plan)  presentation of MBRFC and NCRFC QPF verification case studies  survey on real-time verification (analog query and summary verification displays): feedback from all RFCs

HIC Meeting, 02/25/ Clarifications on Team Activities Verification team report listed only recommendations  Proposed sensitivity analyses for RFCs: 1) what is the optimized QPF horizon for hydrologic forecasts? 2) do run-time mods made on the fly improve forecasts? Each HIC decides how the RFC team member(s) may contribute to the team Each RFC defines its own verification case study to test proposed verification standards

HIC Meeting, 02/25/ Recommendations: 1) QPF horizon Goal: what is the optimized QPF horizon for hydrologic forecasts? QPF horizon to test  0 (no QPF), 6-hr, 12-hr, 18-hr, 24-hr, 30-hr, 36-hr, 48-hr, 72-hr  Longer (96-hr, 120-hr…): depends on RFC and location Model states to use  Similar to operational mods except mods that impact QPF  Metadata to store which mods were used in these runs What forecast to verify  6-hr stage forecasts for 7-day window (longer window for slow response basins)

HIC Meeting, 02/25/ Recommendations: 2) run-time MODs Goal: do run-time mods made on the fly improve forecasts? 4 scenarios  Operational forecasts (w/ all mods)  Forecasts w/ best available obs. and fcst. inputs wo/ on-the-fly mods  Forecasts w/ best available obs. inputs (no fcst) w/ all mods  Forecasts w/ best available obs. inputs (no fcst) wo/ on-the-fly mods What forecast to verify  6-hr stage forecasts for same window as in operations Model states  Carryover from 5 days ago (w/ past mods) + a priori mods (known before producing any forecast)

HIC Meeting, 02/25/ Discussion Other ways to explore QPF horizon and run-time mods? Feedback on RFC’s contribution to verification? How to best coordinate between Verification Team, CAT/CAT2 Teams, Graphics Generator Requirements Team, SCHs and OCWWS/HSD? Feedback on future verification activities at OHD?

HIC Meeting, 02/25/ Thank you! Extra slides

HIC Meeting, 02/25/ Final team report: recommendations Key verification metrics for 4 levels of information for single-valued and probabilistic forecasts 1.Data information (scatter plots, box plots, time series plots) 2.Summary information (e.g. skill scores) 3.More detailed information (e.g. measures of reliability, resolution, correlation) 4.Sophisticated information (e.g. for specific events) Corresponding verification products

HIC Meeting, 02/25/ Recommended metrics 4 different levels of information

HIC Meeting, 02/25/ Recommended analyses Analyze any new forecast process with verification Use different temporal aggregations (e.g. weekly max. flow)  Analyze verification score as a function of lead time; if similar performance, data can be pooled from different lead times Perform spatial aggregation carefully  Analyze results for each basin and results plotted on spatial maps  Use normalized metrics (e.g. skill scores) Evaluate forecast performance under different conditions  w/ time conditioning (by month and season)  w/ atmospheric/hydrologic conditioning using probability and absolute thresholds (e.g. 90 th percentile vs. Flood Stage) Report verification scores with sample size (in the future, confidence intervals)

HIC Meeting, 02/25/ Diagnostic verification products Example: skill score maps by months January AprilOctober Smaller score, better

HIC Meeting, 02/25/ Approach: select analogs from a pre-defined set of historical events and compare with ‘live’ forecast Real-time verification: analogs Analog 1 Observed Live forecast Analog Forecast Analog Observed Analog 2 Analog 3 “Live forecast for Flood is likely to be too high”

HIC Meeting, 02/25/