Forecast Uncertainty & the End User Susan Joslyn

Uncertainty forecasts for the general public  Why?  Tailor forecast to risk tolerance and parameter threshold  Increase trust in the forecast  Acknowledge uncertainty  Seem less “wrong” when single value forecast fails to verify

Do people have varying uncertainty thresholds?  Yes!  Thresholds for everyday decisions (Morss, et al., 2010)  moving a picnic indoors  protecting plants against frost damage  Reasons for hurricane evacuation (Zhang et al., 2007)  16% because they were told to evacuate  60% relative vulnerability or the costs weighed against risk

Do people realize there is error in the forecast?  Yes!  Survey of 1,340 residents of Pacific Northwest  98% west of Cascades  Given 52 forecasts & asked what they expected to observe  Wide range of expectations Forecast:Night time low in December: 32ºF Participants expect: I----36ºF  Extreme forecasts Forecast: Daytime high in August: 100ºF Participants expect more normal values: I--102ºF  Contrary to verification: observed values tend to be slightly MORE extreme (Joslyn & Savelli, 2010)

Survey of general public users  Could benefit from uncertainty forecasts  Wide variation in thresholds for action  People attempt to estimate it themselves  Unjustified biases  Good uncertainty forecasts  Allow them to make decisions tailored to their own needs  Improve people’s understanding of future weather events

Can people understand uncertainty estimates?  Cognitive Psychology: uncertainty leads to errors  Understanding is influenced by a number of biases  Systematic errors in interpretation  Decision making is non-optimal from an economic standpoint

Cognitive Psychology: biases  Framing: choice is influenced by wording  People make different medical treatment choices 90% short-term survival 10% immediate mortality  Experienced physicians as well as patients  McNeil, Pauker, Sox, and Tversky (1982)

Probcast Website: 80% Predictive Interval for Temperature  Unlikely but possible values  90% Upper Bound  10% Lower Bound

How do we define it? Upper Bound is: 10% chance observed temperature will be greater 90% chance observed temperature will be less Framing: People will think the temperature will be higher

We tested this  Asked people  Rate the amount of uncertainty expressed by the forecast  Make decision based on the forecast  NO framing effect  10% greater = 90% less  Understanding depended on decision task  Uncertainty expression compatible with decision Joslyn, et al High Wind Advisory (wind >20 knots) Freeze warning (temp < 32ºF) 90% chance lessconfusingunderstandable 10% chance greaterunderstandableconfusing

17 k 15 k 12 k 9 k Median & Upper Bound 17k Median, Upper Bound & Lower Bound 15 k Median 12 k Median & Lower Bound 9 k Should we provide all information?  Don’t confuse them:  Just give them the worst case scenario  Worst Case: upper bound of 80% predictive interval (highest possible wind speeds). “Most likely wind speed is 15 knots, but there is a 10% chance the wind speed > 27 knots”

Cognitive Psychology: biases  Anchoring: Numeric estimates are unduly influenced by a previously considered standard  Estimate proportion of African Countries in the UN  Influenced number spun on a “wheel of fortune” (Tversky and Kahneman, 1974).  Unintentional and unconscious  Not reduced when explained to participants  Not reduced when they are given incentives to overcome it (Wilson et al., 1996, Chapman & Johnson, 1994)

We tested this: Anchoring for worst case scenario weather forecast 17 k 15 k 12 k 9 k The most likely wind speed will be 15 knots “What do you think the wind speed will be?” Upper Bound Lower Bound Control Lower Bound Upper Bound knots

We tested this: Correcting for deterministic forecast 17 k 15 k 12 k 9 k The most likely wind speed will be 15 knots “What do you think the wind speed will be?” Upper Bound Lower Bound ControlLower BoundUpper Bound knots

We tested this: Upper & Lower Bound believed the forecast 17 k 15 k 12 k 9 k The most likely wind speed will be 15 knots Upper Bound Lower Bound ControlLower BoundUpper Bound knots “What do you think the wind speed will be?”

Can people understand uncertainty?  Depends on how it is expressed  Communication compatible with use  Balanced picture  Basic cognitive research does not translate directly into rules for applied situations  Sometimes it does: Anchoring  Sometimes it doesn't: Framing  Tested in very simple lab based situations where all other variables are eliminated and controlled.  Applied situations: complex interaction of cognitive processes  Applied research on weather related decisions in necessary  Complex realistic situations  Real forecasts-- expectations from prior experience

Applied Studies: misinterpretation  Misunderstanding probability of precipitation (PoP)  Point or area forecast (Murphy et al., 1980)  Deterministic conversion error  70% chance of precipitation-----> 70 % time 70 % area  Error only reduced by  70% chance rain & 30% chance of no rain (Joslyn, et al., 2008)   Predictive interval  Deterministic 12 hour range  Despite providing key Night Time Low Temperature Night time low temperature

Deterministic Conversion Error  People have a tendency to interpret probability as a deterministic quantity such as proportion or range  Not necessarily a conscious choice  Reduces processing load

Decisions without uncertainty Decide Water Plants Water Plants Do Nothing Do Nothing

Uncertainty increases cognitive processing load  Consider both outcomes & the probability for each Decide Water Plants Water Plants Do Nothing Do Nothing Rain Waste water Rain Waste water No rain Plants happy No rain Plants happy Rain Save water Rain Save water 70% chance 70% chance

Do people make better decisions with uncertainty?  Basic research  Decisions is not economically optimal  Too much risk in some situations  Too little risk in others  Choices between hypothetical gambles

How does this translate to weather related decisions?  We don’t care if decisions are economically optimal  We care whether decision with uncertainty forecasts are better than decisions without it  Different question

Applied Research: Weather forecast uncertainty improves decisions  Farmers in Zimbabwe (Patt, 2000)  Half of whom had finished high school  Decisions improved with probability forecasts  College student increase “expected value” (Roulston, 2006)  Not optimal but BETTER with than without uncertainty Expected value: value of outcome weighted by probability that it will occur

Applied Research: apply salt brine to roads to prevent icing Salt more when freezing is likely Salt less when freezing is unlikely %% % People with uncertainty forecast had significantly higher trust in the forecast

Economically optimal : Salt whenever expected value of penalty exceeds cost of treatment Cost to treat roads = -$1,000 Penalty for not treating & freeze = -$6,000 Economically optimal strategy = 1000/6000 = 16.67% Salt whenever the probability of freezing Is 17% or more

Decision Making  People make better decisions with uncertainty information  Not necessarily optimal  Uncertainty increases trust in the forecast

Conclusions  People need uncertainty estimates  Make better decisions  Recognize uncertainty and try to estimate it themselves  Unjustified biases  Uncertainty is difficult for people to process  Increases processing load  Limited conscious level processing capacity  Communication is key  Compatible with tasks  Provide balanced picture  We need much more applied experimental research

The End This research was supported by the National Science Foundation under Grant No. ATM

Study 3: Strategy change in high error trials % %% High Error Low Error Probability of Freezing Optimal Decision is to salt (PoF > 17%) People salt less often when forecast error is high

Basic Cognitive Research: Decision making is not optimal  From an economic standpoint:  Risk seeking: too much risk  Risk averse: too little risk  Choices between hypothetical gambles such as: Which would you prefer? Alternative 1Alternative 2 50% loose $1,000 sure loss $450 50% loose 0

Basic Cognitive Research: Decision making is not optimal  From an economic standpoint:  Risk seeking: too much risk  Risk averse: too little risk  Choices between hypothetical gambles such as:  Which would you prefer? Alternative 1Alternative 2 50% win $1,000 sure gain $450 50% win 0

Sensory Memory Working Memory Consciousness Long Term Memory Information Processing Model Physical Energy Light waves, Sound waves Large capacity Small capacity “Bottleneck” Decisions are made here