1. NOAA CLIMAS NASA EOS NSF SAHRA NOAA GAPP Forecast Assessment: Tactics, Techniques, and Tools Holly C. Hartmann Department of Hydrology and Water Resources University of Arizona hollyh@hwr.arizona.edu hollyoregon@juno.com NASA HyDIS Raytheon Synergy

2. NOAA CLIMAS NASA EOS NSF SAHRA NOAA GAPP NASA HyDIS Raytheon Synergy Concepts of integration Our hydrology team’s experience Examples of usable science Lessons learned Forecast Assessment: Tactics, Techniques, and Tools

3. Calls for Societally Relevant Research and Products Information is appropriate to the knowledge and concerns of the recipient. Ensure that.. modeling improvements and data products are useful to the water resources management community. Develop a strategy for… how these could be made more useful for [user] purposes. Need studies of the benefits and costs of [hydroclimatic] information services. Increase the value of weather and related … information to society. Bring scientific outputs and users’ needs together. Make climate forecasts more socially useful. Stronger sense of responsibility for delivering timely and relevant tools. Accelerate activities to integrate science with the needs of decision makers. Integrate user needs… and ensure that research results are provided in a form useful for users. Sources: Various USGCRP and NRC reports, 1997-2004

4. CLIMAS: A Regional Integrated Science Assessment Purpose of NOAA RISAs Assess impacts of climate variability on human & natural systems Improve the ability of stake- holders to anticipate & respond to climate variability & change - lower risk (vulnerability) - increase opportunity - ensure sustainability Educate a new generation of interdisciplinary scientists www.ispe.arizona.edu/CLIMAS A Research Process – Not a Research Product

5. CLIMAS: A Regional Integrated Science Assessment www.ispe.arizona.edu/CLIMAS Purpose of NOAA RISAs Pilot for climate services Hydroclimatology research to address regional priorities … driven by stakeholders … hand-in-hand with social science research & sustained stakeholder interactions A Research Process – Not a Research Product A Real Partnership with Users

6. Four Types of Integration 1. Integration of decision makers, researchers & other stakeholders in developing research agenda One-on-One Interviews Town Hall Meetings Conferences and Workshops Product Evaluation

7. Four Types of Integration 1. Integration of decision makers, researchers & other stakeholders in developing research agenda 2. Disciplinary integration of research team to address science questions Stakeholders Information needs, understanding, access Social Science Effective communication Natural Science Forecast skill, interpretation

8. Four Types of Integration 1. Integration of decision makers, researchers & other stakeholders in developing research agenda 2. Disciplinary integration of research team to address science questions 3. End-to-end integration of climate issues Variability  Impacts  Response Options Global ocean/atmosphere/land  Regional climate  hydrology  water resources and demand  water management Multiples stresses, multiple pathways

9. Four Types of Integration 1. Integration of decision makers, researchers & other stakeholders in developing research agenda 2. Disciplinary integration of research team to address science questions 3. End-to-end integration of climate issues 4. Integration of research and services Research Operations Decision Makers Public groups

10. Issue: So Many Stakeholders! Continental Scale: Focus of much research Watershed/Local Scale: Where impacts happen Where stakeholders exist Different Scales (time & space) Different Issues Different Stakeholders

11. Issue: Many Space & Time Scales

12. Changed decisions & decision processes Transferability, Scalability (products and process) to enable systemic change Public support for climate research Concerns for Climate Science Enterprise Project Objectives Affect… Metrics Structure of stakeholder interactions Research products Perceptions of climate science enterprise Research funding Evaluating Societally Relevant Research and Products

13. Objective: Economic Efficiency Metrics: Cost/benefits. Return on Investment. Stakeholder Interaction Structure: Consultant-client relationship with high-value clients (e.g., hydropower). Research Products: Customized Decision Support Systems. System optimization rules. Perceptions: Science serving special interests. Increasing competitive imbalances. Research Funding: By clients through private sector.

14. Objective: Agency Impact Metrics: Policy and regulatory impact. Stakeholder Interaction Structure: Work with agencies. Important role for policy analysts/scientists. Research Products: Traditional products. Refereed methodology and results. Hold up in court. Perceptions: Science serving special interests, agendas. Increasing regulatory burden. Research Funding: By managed sector, perhaps public.

15. Objective: Societal Equity Metrics: Breadth/diversity of applicability, accessibility, usability. Sectoral ‘market’ penetration. Stakeholder Interaction Structure: Engagement with diversity of stakeholders. Important role for social scientists. Potentially huge demand on researchers’ time. Research Products: Diverse. Non-traditional, but not “dumbed down”. Note: data << information << knowledge << wisdom Perceptions: Science providing useable information and practical tools. Increasing capacity to adapt to climate variability. Research Funding: Public.

16. Structuring Stakeholder Integration What kind of relationship? Consultant-Client Education & Outreach Research Partnership Public Participation: targeted groups first, general groups eventually Stakeholders Natural Science Social Science

17. Stakeholder-Driven Research Process Using public participation for stakeholder integration - from Hartmann et al., submitted to Global Environmental Change

18. Structured discussions: meetings, workshops, conferences - foster mutual learning - techniques not well developed or assessed - social scientists or physical scientists can’t do it alone Structuring Stakeholder Integration Getting input from stakeholders: Surveys, In-depth interviews - efficient implementation - provide snapshots Structure of products and tools for stakeholders    

19. Evaluating Success of Products and Process ??? PRODUCTS: Forecasts - traditional publications - MS/PhD degrees - newsletter outreach - database of forecasts - forecast evaluation tool - “Climate in a Nutshell” (450+) - presentations to stakeholder groups (25+) - workshops (research/forecast/stakeholder) (8+) Frequent interaction, from the outset Interaction… not outreach! Getting and giving Starting where the stakeholders are Moving dialogue & action forward

20. Why Focus on Forecasts? 6-year paleodrought as reconstructed with tree-rings After E. Cook et al., Summer PDSI 1818-1824 Current drought (precipitation deficit) Monitoring and historical analysis are NOT enough!

21. Forecasts link science and society Resource management decisions require forecasts: formal or ad hoc, large scale or small scale Predictions are tests of the state of the science Confront scientific understanding with every prediction and decision Why Focus on Forecasts?

22. Climate predictability Hydrologic predictability Risk-based decision making Forecast misinterpretation Unknown forecast quality Lack of historical context Institutional/social factors Need different variables Use of new seasonal forecasts Improvements Barriers Changing the Balance…

23. Common across all groups Uninformed, mistaken about forecast interpretation Understand implications of “normal” vs. “unknown” forecasts Use of forecasts limited by lack of demonstrated forecast skill Stakeholder Use of Hydroclimatic Forecasts Unique among stakeholders Relevant forecast variables, regions (location & scale), seasons, lead times, performance characteristics Role of of forecasts in decision making Technical sophistication: base probabilities, distributions, math Common across many, but not all, stakeholders Have difficulty distinguishing between “good” & “bad” products Have difficulty placing forecasts in historical context

24. http://hydis6.hwr.arizona.edu/ForecastEvaluationTool/ Initially for NWS CPC climate forecasts Adding water supply forecasts Six elements in our webtool: Exploring Forecast Progression Forecast Interpretation – Tutorials Historical Context Forecast Performance Use in Decision Making Details: Forecast Techniques, Research

25. 1. Forecast Progression

26. Warm Near Normal Cold Weighted “climate dice” Probability Concepts Tutorials Self-Quizzes 2. Forecast Tutorial

27. Unknown EC Sometimes forecasters don’t know what the chances are… EC - EQUAL CHANCES THE PROBABILITY OF THE MOST LIKELY CATEGORY CANNOT BE DETERMINED = Unknown Chances!!! 63% 33% 3% “+30% Chance of Warm” Each colored contour indicates a shift in the normal chances. 33% Climatology Climatology is only a reference (1971- 2000), not a substitute forecast 2. Forecast Tutorial

28. 3. Historical Context for Forecasts Recent History | Possible Futures Requested by Fire managers… Applicable to any climate variable 20032002 Neutral Non-ENSO sequences 2004 La Nina

29. Wet Near-Dry Normal 10 years had more than 3.7 inches 10 years had less than 1.9 inches 10 years were in the middle Willcox Jan-March Total Precipitation 1930-2001 Year Precipitation (Inches) 1971-2000 Willcox Jan-March Total Precip. 1971-2000 Dry Norm Wet 0” 1.9” 3.7” 8+” Exceedance Probability 3. Historical Context for Forecasts

30. La NinaEl Nino El Nino La Nina 50% Wet 0% 30% Norm 25% 20% Dry 75% Willcox, AZ: Precipitation, JFM 3. Historical Context for Forecasts

31. 4. Forecast Performance Evaluation

32. “Today’s high will be 76 degrees, and it will be partly cloudy, with a 30% chance of rain.” Deterministic Categorical Probabilistic Categorical Deterministic Different Forecasts, Information, Evaluation 4. Forecast Performance Evaluation

33. Forecast evaluation concepts All happy families are alike; each unhappy family is unhappy in its own way. -- Leo Tolstoy (1876) All perfect forecasts are alike; each imperfect forecast is imperfect in its own way. -- Forecast evaluation is very multi-faceted! 4. Forecast Performance Evaluation

34. Deterministic Bias Correlation RMSE Standardized RMSE Nash-Sutcliffe Linear Error in Probability Space Categorical Hit Rate Surprise rate Threat Score Gerrity Score Success Ratio Post-agreement Percent Correct Pierce Skill Score Gilbert Skill Score Heidke Skill Score Critical Success index Percent N-class errors Modified Heidke Skill Score Hannsen and Kuipers Score Gandin and Murphy Skill Scores… Probabilistic Brier Score Ranked Probability Score Distributions- oriented Measures Reliability Discrimination Sharpness 4. Forecast Performance Evaluation

35. False AlarmsSurprises warning without eventevent without warning No fire “False Alarm Rate”“Probability of Detection” A forecaster’s fundamental challenge is balancing these two. But decision makers may be more sensitive to one. 4. Forecast Performance Evaluation

36. Observed? Yes NoTotal Forecast? Total No Yes 10 20 30 3535 70 45 55 100 False Alarm Rate: 20/30 = 66% Probability of detection: 10/45 = 22% “How often were you not surprised?” Categorical Forecast Evaluation

37. Observed Outcome Probabilistic Forecast Evaluation: “Brier” Score 80% 20% Forecast: “80% chance of rain” Rain 55% 45% Forecast: “55% chance of rain” The “Brier” Score considers: Direction of forecast Strength of forecast Category of interest

38. (Your Forecast - The Observed) 2 = Your Performance 80% If rain happens… 100% 0.2 2 0.04 20% 100% 0.8 2 0.64 If rain doesn’t happen… (Your Forecast - The Observed) 2 = Your Performance Probabilistic Forecast Evaluation: Brier Score

39. (0.50 – 0.54)/(1.00-0.54) = -8.6% ~worse than guessing~ Skill Score = Forecast - Baseline Perfect - Baseline

40. (Your Forecast - The Observed) 2 = Your Performance 80% If rain happens… 100% 0.2 2 0.04 50% 100% 0.5 2 0.25 (Climatology - The Observed) 2 = Climatology’s Performance Skill Score = (0.04-0.25)/(0-0.25) = +84% Forecast Evaluation: Brier Skill Score

41. (Your Forecast - The Observed) 2 = Your Performance 20% If rain doesn’t happen… 100% 0.2 2 0.04 50% 100% 0.5 2 0.25 (Climatology - The Observed) 2 = Climatology’s Performance Skill Score = (0.04-0.25)/(0-0.25) = -156% Forecast Evaluation: Brier Skill Score

42. Observed Outcome Probabilistic Forecast Evaluation: “Brier” Score 50% 80% 20% Climatology (Baseline chances) Forecast: “80% chance of rain” Rain No Rain Skill Scores +84% -156%

43. Observed Outcome 50% 80% 20% Climatology (Baseline chances) Forecast: Good Not Good 33% 63% 33% 3% “+30% chance of warm” Warm Near- Normal Cold Really Bad Ranked Probability Score is like Brier Score, except it penalizes for being off by more than 1 category Probabilistic Evaluation: Ranked Probability Score

44. Example RPS = (0.10 - 0) 2 + (0.43 - 0) 2 + (1 - 1) 2 = 0.20 RPS clim = (0.33 - 0) 2 + (0.67 - 0) 2 + (1 - 1) 2 = 0.55 SS rps = (0.20 - 0.55)/(0 - 0.55) = 0.63 Brier Score: 2 categories Ranked Probability Score: multiple categories - similar to MAE, but for cumulative probability - observation ‘probability’ = 0 or 1 Skill Score: improvement over baseline forecast 57% 33% 10% 33%

45. Distributions-Oriented Evaluation Reliability P[O|F] Does the frequency of occurrence match your probability statement? Relative frequency of observations Forecasted probability

46. Reliability: CPC climate forecasts & water management CPC forecast performance varies among regions, with important implications for resource management. Seasonal climate forecasts have been much better for the Lower Colorado Basin than for the Upper Basin. Lower Basin Upper Basin Upper Colorado River Basin Lower Colorado River Basin Forecasts “better” than expected Forecast probability for “wet” Relative frequency of observations 0.2 0.4 0.6 1 0.8 1 0.2 0 0 Precipitation forecasts accurately reflect expected performance perfect reliability

47. Forecasted Probability Relative frequency of this strength of forecast Climatology 0.00 0.33 1.00 Good discrimination! Forecasted Probability Relative frequency of this strength of forecast Climatology 0.00 0.33 1.00 Not much discrimination! Probability of dry Probability of wet Distributions-oriented Evaluation Probability of dry Probability of wet Discrimination: P[F|O] Can the forecasts distinguish among different events?

48. Relative Frequency of Forecasts high 30% mid 40% low 30% 1)High flows less likely. 2) No discrimination between mid and low flows. 3) Both UC and LC show good discrimination for low flows at 2-month lead time. Discrimination of ESP Water Supply Outlooks Jan 1 Forecast probability Apr 1 Lower Colorado Basin Jan-May (5 mo. lead) April-May (2 mo. lead) Jan 1 Jun 1 Upper Colorado Basin Jan-July (7 mo. lead) June-July (2 mo. lead) For observed flows in lowest 30% of historic distribution (Franz, 2001)

49. Conclusion There’s only one way to be “right” Many ways to be “wrong” Given that forecasts aren’t perfect… Which imperfections can you tolerate? Which can’t you tolerate?

51. 4. Forecast Performance Evaluation Sub-setting: Seasons, Leadtimes, Regions Criteria: Simple/Intuitive to Complex/Informative Transparency: Data behind analysis

52. 4. Forecast Performance Evaluation Sub-setting: Seasons, Leadtimes, Regions Criteria: Simple/Intuitive to Complex/Informative Transparency: Data behind analysis

53. Custom real-time data access, analysis, and information Value-added interpretation Multiple entry points along continuum of sophistication Opportunities and tools for increasing sophistication Knowledge development emphasis vs. decision support Data << Information << Knowledge << Wisdom We can provide forecasts with: Accessibility (physical & conceptual), Relevance, Credibility Lessons of FET for HydroClimate Services Other issues: Ease of use (information management), Assisting information intermediaries

54. Ease of Use  Profile and Projects: save a history of your work on each "project", so you can return to your work any time, easily repeat past analyses using updated data. Facilitating Information Intermediaries Accessibility  Report Generation create PDF reports of your analyses for non-Internet users automatically includes legends, data sourcing, contact information, caveats, explanations sections for user-customized (value-added) comments Future: Automated Updating & Additional Products: water supply forecasts, experimental climate forecasts, drought monitoring, river flow and storage monitoring

55. Lessons Learned: Knowledge Development Tools Stakeholders Information needs, understanding, access Social Science Effective communication Natural Science Forecast skill, interpretation Transferable, scalable tools are possible! Focus on knowledge development, not just data & information.

56. Lessons Learned: Knowledge Development Tools Stakeholders Information needs, understanding, access Social Science Effective communication Natural Science Forecast skill, interpretation Computer Science Web programming Transferable, scalable tools are possible! Focus on knowledge development, not just data & information. Interactive webtools require major commitment and resources. Prototypes insufficient! Stakeholders need reliable tools, which require solid software foundation, organized development, sustainability for maintenance and expansion.