RFC Verification Workshop What to Expect Julie Demargne and Mary Mullusky RFC Verification Workshop, August

2 Outline Context Goals Agenda Survey Analysis Verification Questions Logistics

3 Approach to River Forecasting Observations Models Predictions Forecasts BULLETIN FLOOD WARNING NATIONAL WEATHER SERVICE SHREVEPORT LA 1033 AM CDT WED APR

4 Where is the …? Verification … Verification Little verification of hydrologic forecasts has been conducted to date. Key question: How good are the forecasts? Need to fill this void for Research, Operations and Users ???

5 V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N Improving the Approach to River Forecasting Observations Models Predictions Forecasts BULLETIN FLOOD WARNING NATIONAL WEATHER SERVICE SHREVEPORT LA 1033 AM CDT WED APR

6 Hydrologic Verification Verification helps us answer these questions: –How good are the forecasts? –What are the sources of skill and error? –How is new science improving the forecasts? –What should be done to improve the forecasts? Verification supports: –Research by identifying needs and clarifying value of results –Operational agencies by defining acceptable methods and satisfying user demands –Users by quantifying forecast skill and uncertainty for use in decision making

7 Hydrologic Verification in NWS NWS River Forecast Verification System under development –Requirements and Verification Plan defined by Verification System Requirements Team in October 2006 On-going verification activities to be coordinated and expanded Verification Model Setup Calibration Operation Installation State Updating Data Quality Control Runtime Simulations Data Assimilation Forecast Computation Hydrologic and Hydraulic Models Postprocessor Product Review and Issuance Forecaster Analysis and Quality Control

8 Hydrologic Verification in NWS !!! Verification Experts x Verification System Forecasts + Verification Info Papers

9 Workshop Goals OBJ1: Review verification concepts and methodologies OBJ2: Review existing verification software capabilities OBJ3: Discuss strategies to communicate verification results OBJ4: Discuss recommendations in NWS River Forecast Verification Plan report Answer your verification questions Help you develop verification expertise

10 Workshop Agenda: Highlights Day 1 –Verification concepts and metrics –Verification strategy -> lab with your forecast products –Local verification activities Day 2 –National verification –Archive database –Logistical verification -> lab –Deterministic verification: IVP -> lab Day 3 –Probabilistic forecast verification at University of Iowa –Ensemble verification: EVS -> lab

11 Survey Analysis Goal: better understand forecast processes and forecaster expertise Forecast Processes –Deterministic forecast –Ensemble forecast Verification Processes and Expertise

12 Survey Analysis: Mods All sorts of Mods are used Significant effort from forecasters

13 Survey Analysis: Mods Various factors used to define Mods Large expertise of forecasters

14 Survey Analysis: Mods Numerous iterations of Mods Significant effort from forecasters

15 Survey Analysis: Variables All sorts of variables are important Each variable may require a different verification strategy

16 Survey Analysis: Sources of Error Many sources of error How do you separate the different sources of error

17 Survey Analysis: Verification Metrics A significant part of Day 1 will be devoted to verification metrics

18 Survey Analysis: Reference Forecast Discussion on reference forecasts will be part of the presentation of skill score metrics (Day 1)

19 Survey Analysis: Other Findings Ensembles –Benefits not identified primarily because of lack of rigorous verification Quality of real-time forecasts (deterministic and ensemble) Role of hindcasting experiments (deterministic and ensemble)

20 Verification Questions List of questions provided (sorted by agenda topics) –Verification Concepts and methodologies –Effective Strategies to Communicate Verification (deterministic and probabilistic) –Findings and Recommendations of the Hydrologic Verification System Requirements Team –National Verification Program –Archiving requirements (deterministic and probabilistic) –IVP Software Capabilities –EVS software capabilities –Verification Teams (Workshop Follow Up and Future Training) Strategy: Questions answered during the session. Review questions at the end of the day to be certain questions were answered. Remaining questions will be placed on a list for the new NWS Hydrology Forecast Verification Team.

21 Workshop Logistics To eat –Lunch –Dinner at Squatters Brew Pub at 6:30pm tonight (head count) Lab sessions –Workstation Internet access CBRFC workshop contact point Carpool CBRFC staff: thank you!

22 Thank you … let’s start!