1. © Crown copyright Met Office Towards an optimized communication of probabilistic weather forecasts via the internet Liz Stephens 1, Ken Mylne and David Spiegelhalter 2 1. University of Bristol, 2: University of Cambridge

2. © Crown copyright Met Office Defining project Designing representations Developing game Running game Outcomes Intro Project Brief Summary review existing ways of representing probabilistic information develop online surveys to ascertain user preferences for specific designs analyse the outputs provide recommendations regarding which designs work best with people from different demographic profiles

3. © Crown copyright Met Office Defining project Designing representations Developing game Running game Outcomes Intro Current representations Summary Metcheck yr.no Australia Probcast (UW) UK Met Office

4. © Crown copyright Met Office Defining project Designing representations Developing game Running game Outcomes Intro Objectives, scope and limitations Addressing assumptions: “the public don’t understand uncertainty” “uncertainty is too difficult to communicate” …everyday weather not extremes …basic representations …no implicit representations …temperature and rainfall only …6 months Can players make better decisions? How does this vary with presentation type? Demographic influences Preference vs understanding Summary

5. © Crown copyright Met Office Defining project Designing representations Developing game Running game Outcomes Intro Summary

6. © Crown copyright Met Office Defining project Designing representations Developing game Running game Outcomes Intro Deterministic Basic Rating Percentage (rounded) Range of values & percentage Probability distribution & percentage Summary

7. © Crown copyright Met Office Defining project Designing representations Developing game Running game Outcomes Intro Maximise participants Start & finish game Want to do well To share with friends Useful tests Correct decision (relative) Understanding of actual confidence Appropriate Consider theme, style and complexity Not too specific Appeal to all demographics Realistic Associate with real weather situations Hint at utility of uncertainty information Summary

8. © Crown copyright Met Office Defining project Designing representations Developing game Running game Outcomes Intro Age, postcode, gender… Randomly assign presentation type Multiple questions: expect different responses depending on presentation type Outcomes generated on-the-fly (based on probabilities) Only look at 1 st time players Quadratic scoring rule Summary

9. © Crown copyright Met Office Playing the game How confident are you about each shift – tests understanding of presentation

10. © Crown copyright Met Office Scoring Each shift is allocated an outcome randomly selected from the forecast distribution Element of luck! Amount of ice cream bought and sold depends on advice and outcome Total score at end of game:

11. © Crown copyright Met Office Defining project Designing representations Developing game Running game Outcomes Intro Summary Outcomes >8000 people – largest survey of its kind (aiming for 10,000+) Hope to be able to: Identify the percentage of correct answers from each presentation type Identify the skill score of confidence ratings for each presentation type Link this with demographics

12. © Crown copyright Met Office DeterministicProbabilistic Defining project Designing representations Developing game Running game Outcomes Intro Summary Early Results Brad makes more money with probabilistic forecasts!

13. © Crown copyright Met Office

15. Under 16 16-2425-34 35-44 45-5455+ Defining project Designing representations Developing game Running game Outcomes Intro Summary

16. © Crown copyright Met Office Defining project Designing representations Developing game Running game Outcomes Intro Summary Game carefully designed to answer a few simple questions Large sample achieved across different demographics Early results show benefit of probabilistic forecasts Detailed analysis of different presentations is underway Men over 55 do spectacularly badly! You have a few days left to play: www.metoffice.gov.uk/weather/weather-game Not too late to win a T-shirt! liz.stephens@bristol.ac.uk Thanks!