Communicating uncertainty in seasonal climate forecasts to stakeholders Andrea L. Taylor 1,2 Suraje Dessai 2 Wändi Bruine de Bruin 2,3 1 Sustainability Research Institute, University of Leeds 2 Centre for Decision Research, University of Leeds 3 Department for Engineering and Public Policy, Carnegie Mellon University

Paper presented at the Centre for Climate and Energy Decision Making (CEDM) funded by the US National Science Foundation. We thank CEDM and the US National Science Foundation for the opportunity to share this work with the wider CEDM community. EUPORIAS European Provision Of Regional Impacts Assessments on Seasonal and Decadal Timescales This project has received funding from the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement no

Dr Andrea TaylorProf Suraje DessaiProf Wӓndi Bruine de Bruin Taylor, A. L., Dessai, S., & de Bruin, W. B. (2015). Communicating uncertainty in seasonal and interannual climate forecasts in Europe. Phil. Trans. R. Soc. A, 373(2055),

Seasonal Forecasts: Sources of Uncertainty Uncertainty in seasonal climate predictions comes from multiple sources. Forecasts are probabilistic. No Forecast perfectly captures the world’s climate system. Example of ensemble seasonal temperature forecast (ECMWF – provided by Jean Pierre Ceron, Meteo France)

Seasonal Forecasts: The challenge of communicating uncertainty Forecast distribution used to obtain measures of ‘likelihood’. Forecast performance measured by comparing predictions to observations. – Reliability: How much better does the forecast perform than chance? – Skill: To what extent does the forecast perform better (or worse) than historical data at predicting future conditions?

Different ways of representing seasonal forecasts…. … but relatively little testing with users Top left example wind forecast (IC3). Top middle seasonal temperature forecast (MeteoSwiss), Top right precipitation forecast (Met Office). Bottom left Temperature anomaly forecast (ECMWF )

Reviewing the broader literature on communicating uncertainty Interpretation and use of uncertain information may be influenced by a number of factors: – ‘ambiguity aversion’ (e.g. Ellsberg, 1961) – institutional protocol (e.g. Demeritt et al., 2010; Ramos et al., 2010) – technical and statistical expertise (e.g. Gregory et al., 2012) – cognitive biases (e.g. Kahneman, 2011) – use of colour in visualisations (e.g. Kaye et al., 2012) Tradeoffs between detail, robustness and understandability (Stephenson et al., 2012)

Communicating confidence and uncertainty in seasonal forecasts: Surveying user needs Sample 50 respondents from stakeholder organisations started the survey. 45 provided full completions. Use weather forecastsn = 45 Use seasonal forecastsn = 33 Use interannual forecastn = 18

Communicating confidence and uncertainty in seasonal forecasts: Surveying user needs Key questions 1.How accessible, understandable and useful do users perceived seasonal forecasts to be? 2.What information about uncertainty in seasonal forecasts do users currently receive, and what would they like to receive? 3.When it comes to representing likelihood what are participants’ preferences?

How accessible, understandable and useful do users perceived forecasts to be? Mean ratings of accessibility, understandability and usefulness amongst survey respondents.

What information about uncertainty in seasonal forecasts do users currently receive, and what would they like to receive?

When it comes to representing likelihood what are respondents’ preferences?

When it comes to representing uncertainty what are respondents’ preferences? Correlations (Spearman’s ρ) PreferenceFamiliarity Preference with Familiarity Preference with statistical comfort Familiarity with statistical comfort Mean (SD) ρρρ Map3.9 (0.7)3.3(1.1).60*** Fan chart3.9 (0.7)3.4(1.1).52*** Error bar3.7 (0.7)3.5(1.1).74***.38**.34* Bar graph: Distribution3.4 (0.8)3.0(1.1).52***.26†.19 Pie chart3.2 (0.9)2.9(1.1).71*** Spaghetti plot3.1 (1.0)3.0(1.2).69*** Bar graph: Tercile3.0 (0.9)2.5(1.1).76***.08.26† †Marginally significant at p ≤.10 *Significant at p ≤.05 **Significant at p ≤.01 ***Significant at p ≤.001 Mean ratings of preference and familiarity and correlation between preference, familiarity and statistical comfort

Conclusions and practical implications Seasonal predictions are not judged to be as understandable as more familiar forecasts. Information about forecast performance (skill) is not being clearly communicated to many current users. Preference for different types of visualisation influenced by both familiarity and statistical expertise.

….Follow up stakeholder Decision Lab Pr eference for particular communication formats related to perceived familiarity, but not objective understanding. Evidence that perceived familiarity can sometimes suppress objective understanding. Even when forecasts do not perform well enough to provide useful information, “likelihoods” still affect judgement. Highlights the importance of testing user understanding. Should forecasts “without skill” be provided at all?

Centre for Decision Research Left to right: Prof Barbara Summers, Dr Andrea Taylor, Dr Yasmina Okan, Dr Simon McNair, Prof Wӓndi Bruine de Bruin, Dr Richard Hodgett, Prof Alan Pearman, Dr Gulbanu Kaplan

Climate Change Adaptation Group Left to right: Prof Suraje Dessai, Dr Marta Bruno Soares, Dr Susanne Lorenz, Dr Paola Sakai, Dr Andrea Taylor.