1. 1 Probabilistic Forecast Verification Allen Bradley IIHR Hydroscience & Engineering The University of Iowa RFC Verification Workshop 16 August 2007 Salt Lake City

2. 2 Advanced Hydrologic Prediction Service Ensemble streamflow forecasts

3. 3 Advanced Hydrologic Prediction Service Ensemble streamflow forecasts Multiple forecast locations

4. 4 Advanced Hydrologic Prediction Service Ensemble streamflow forecasts Multiple forecast locations Throughout the United States

5. 5 Forecast Location Forecast Date Forecast Variable How good are the ensemble forecasts produced by AHPS? AHPS Verification

6. 6 Outline Illustrate a consistent diagnostic framework for verification of AHPS ensemble forecasts Describe a prototype interactive web- based system for implementing this verification framework within an RFC Present a future vision for the role of verification archives in AHPS forecasting 6

7. 7 Forecast Verification Framework

8. 8 Perspective: Forecast Users Evaluate the quality of forecasts at a specific location for a particular forecast variable and date Examine one “element” in the data cube 8

9. 9 Elemental Problem Use a distributions-oriented approach (DO) to evaluate probability forecasts for “events” defined by a flow threshold Forecast quality attributes quantified over a range of flow thresholds 9

10. 10 Ensemble Forecast

11. 11 Ensemble Forecast Probability forecast of a discrete eventy f

12. 12 Ensemble Forecast Probability forecast of a discrete event Probability forecasts of multicategory eventsWet Near Avg Dry f dry f avg f wet 12

13. 13 Ensemble Forecast Generalize by defining event forecasts as a continuous function of threshold f 0.50 y 0.50 y 0.75 f 0.75 y 0.25 f 0.25 Index function by the threshold’s climatological probability 13

14. 14 Ensemble Forecast Verification Compute forecast- observations pairs for specific thresholds y p Evaluate forecast quality for a range of thresholds y p

15. 15 Des Moines River near Stratford Standard Errors Skill Skill depends on the threshold Uncertainty is greater for extremes April 1 st Forecasts 15

16. 16 Distributions-Oriented Measures Skill Score Decomposition: (SS) Skill (RES) Resolution (CB) Conditional Bias (UB) Unconditional Bias Slope Reliability Standardized Mean Error Potential Skill

17. 17 Implications for Verification Increase Probability forecast skill Eliminate with bias- correction Minimum 7-Day Flow SS RES CB UB 17

18. 18 Perspective: NWS RFC Forecaster Assess the overall performance of the forecasting system Diagnose attributes limiting forecast skill (e.g., biases) Examine “slices” and “blocks” of the data cube 18

19. 19 Multidimensional Problem The forecaster needs summary verification measures suitable for comparing forecasts at different locations and/or forecasts issued on different dates Summary measures describe attributes of the skill functions derived from the elemental verification problem 19

20. 20 Summary Verification Measures Ranked Probability Skill Score (RPSS): Weighted-average skill over probability thresholds

21. 21 Summary Verification Measures Skill RPSS shows average skill Center of mass shows asymmetries in the skill function RPSS Center of Mass 21

22. 22 Hypothetical Skill Functions All skill functions have same average skill 22

23. 23 Hypothetical Skill Functions All skill functions have same average skill Second central moment shows shape 23

24. 24 Hypothetical Skill Functions All skill functions have same average skill Second central moment shows shape 24

25. 25 NCRFC Forecasts 7-day minimum flow forecasts for mainstem locations for three rivers Minnesota River (MIN) Des Moines River (DES) Rock River (RCK) 25

26. 26 Forecast Skill Attributes Forecasts made at the 1 st of the month 26

27. 27 Forecast Skill Attributes Average skill is highest for DES The skill function is peaked in the middle 27

28. 28 Summary Measure Decomposition Skill Score Decomposition: Skill ResolutionConditional Bias Unconditional Bias Weighted-average measures of resolution and biases

29. 29 AHPS Minimum 7-Day Flow A single MIN site has large biases for low flows The largest biases for other sites centered on higher flows 29

30. 30 Forecast Bias Attributes Unconditional bias is dominate Simple bias- correction can significantly improve forecasts Simple bias- correction Post-hoc calibration 30

31. 31 Verification Framework Forecast quality for ensemble forecasts (e.g., skill) is a continuous function of the forecast outcome (or its climatological probability) Summary measures can be interpreted as measures of the “geometric shape” of the forecast quality function This interpretation provides a framework for concisely summarizing the attributes of ensemble forecasts 31

32. 32 AHPS Verification System

33. 33 AHPS Verification System Web-based tools for online access, analysis, and comparison of retrospective AHPS forecasts for River Forecast Centers (RFCs) http://www.iihr.uiowa.edu/ahps_ver

34. 34 Map-Based Navigation 2 1 3

35. 35 Verification Data Archive 35

36. 36 Verification Data Archive Retrospective forecasts for a 50-year period 36

37. 37 Verification Data Archive Retrospective forecasts for a 50-year period Processed ensemble forecasts & observations 37

38. 38 Verification Data Archive Retrospective forecasts for a 50-year period Processed ensemble forecasts & observations Verification results 38

39. 39 Verification System Concepts Retrospective ensemble traces available in their native format (*.VS files) Processed ensemble forecasts & observations for a suite of variables Uses *.qme files from the calb system Forecast quality measures based on the ensemble forecasts 39

40. 40 Disk Requirements 6 Forecast periods per month (72 per year) All segments have 50 years observed record

41. 41 Advantages Interactive exploration of verification results Provides a diagnostic “report card” for sites within an RFC Instant access to forecasts and quality measures for verification sites Seamless integration with other components of the NWSRFS system 41

42. 42 A Vision for the Future

43. 43 Vision Generation and archival of retrospective forecasts will be a routine component of forecasting systems Verification methods can assess quality Verification results would form the basis for accepting (or rejecting) proposed improvements to the forecasting system Archival information will form the basis for generating improved forecast products

44. 44 Product Generation with Archive Raw ESP forecast Archive verification indicates biases and skill Optimal merging and bias correction Ensemble Forecast Verification Archive Optimized CS

45. 45 Conclusions

46. 46 Conclusions A consistent verification framework provides both users and forecasters with the means evaluating forecast products (exploring the “data cube”) AHPS-VS integrates retrospective forecast generation and forecast verification within the operational setting of an RFC Retrospective forecast archives will become a routine component of a hydrologic forecasting system, enhancing forecast evaluation and product generation

47. 47

48. 48 Des Moines Forecast Skill Skill is higher (lower) downstream (upstream) Skill decline from April to June JCKM5 DESI4 STRI4 Total Bias