1. The Impact of Probabilistic Information on Deterministic & Threshold Forecasts Susan Joslyn, Earl Hunt & Karla Schweitzer University of Washington References Gonzalez, R., & Wu, G. (1999). On the form of the probability weighting function. Cognitive Psychology, 38, 129-166. Eddy, D.M. (1982). Probabilistic reasoning in clinical medicine: Problems and opportunities. In D. Kahneman, P. Slovic, & A. Tversky (Eds), Judgment under uncertainty: Heuristics and biases pp. 249-267) Baars, J. A. & Mass, C. F. (2004) Performance of National Weather Service Forecasts Compared to Operational, Consensus and Weighted Model Output Statistics. Joslyn, S. Jones, D.W. & Tewson, P.(2005) Designing Tools for Uncertainty Estimation in Naval Weather Forecasting. 7 th International Conference on Naturalistic Decision Making Roulston, M.S. & Smith, L. A. (2004) The boy who cried wolf revisited: The impact of false alarm intolerance on cost-loss scenarios. Weather & Forecasting (19) 391-397. This research was supported by the DOD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745 Background How do people understand and use probabilistic information? –People, in general, do not treat probability linearly (Gonzalez & Wu, 1999) –Even experts have trouble incorporating prior probabilities (Eddy 1982 ) –However weather forecasters are good at estimating probabilities of e.g precipitation (Baars & Mass, 2004) Probabilistic information is increasingly important product of ensemble forecasts. Yet few operational weather forecasters make use of available probability products (Joslyn, Jones & Tewson, 2005) EXPERIMENT OUR QUESTION: Can weather forecasters incorporate uncertainty information expressed in probabilities into a non-probabilistic forecasting decision in a way that improves the forecast? Without bias? Information Provided Historical data for all products used in previous cognitive task analysis (Joslyn et al, submitted) Radar Imagery Satellite Imagery TAFs and current METARs Model output (AVN, MM5 & NGM) Instructions: Free choice of products Except: forecasts with the probability product Required to read & record the range provided by the probability product We conducted regression analyses predicting observed wind speeds from forecasted wind speeds. See R 2 for each condition in the table to the right. For 3 dates people did better with the probability product Note especially 3/21 which was the most difficult and for which the probability product made a significant difference. Posting wind advisories: Threshold forecast d with and without the probability product were similar However, WITH the probability product: –the criterion was higher (less willing to post advisory) –Fewer advisories overall. Same forecasters/same weather Difference must be due to probability product Subjects tended to –Over forecast in low-mid probability situations –Underforecast in high probability ranges – Attenuated with probability product Probability product –Good in very low probability situation –Overforecast in the midranges –In general, improved subjects performance CONCLUSIONS RESULTS Probability product had a slight impact on the deterministic wind speed forecast. Wind Advisory Forecasts: –Forecasters, in general, had a liberal bias in the low to mid probability ranges: Biased to post wind advisories Makes sense in this task: being cautious, keeping people off the water if there is any chance of danger However, boaters may come to disregard the advisory if it often proves to be a false alarm (Roulston & Smith, 2004) –Probability product attenuated this tendency without causing them to lose sensitivity--they weren’t more likely to miss high wind situations –Forecasters, in general, had a conservative bias in the highest probability ranges They did not post advisory as often as they should have –Probability product attenuated this tendency more advisories in high range with probability product Manipulation On 1/2 of the forecasts, participants had the MM5 ACME Ensemble Probability of winds greater than 20 knots 10 Participants : Atmospheric Science Students Task: Forecast wind speed and direction for a 48 hour period.. Decide whether to issue high wind advisory (20 + knots). Forecasts were made for 4 days for four different locations on each day. R2 Predicting observed from forecasted wind speeds Wind speed forecast: Deterministic forecast Design: Within subjects Each subject made 2 forecasts with the probability product, 2 forecasts without probability product All subjects saw weather data from all of the same dates Probability product rotated through the dates, making sure that no subject saw the same date twice PROCEDURE