1. 1 WMO SWFDP Macau 9 April 2013 Anders Persson Decision making process and blending ensemble and deterministic forecasts

2. 2 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 1. What do good forecasters do?

3. 3 WMO SWFDP Macau 9 April 2013 Anders Persson Blending deterministic and probabilistic forecast information has been a challenge since Fitzroy started weather forecasting 150 years ago An overcast evening outside London in January 1863: Low clouds over snow covered ground with +2° C The clouds will disperse and the temperature drop to -6° C But will the clouds disperse? Probably? (=60% chance?) How will this affect the forecast?

4. 4 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 The same situation in our days : The NWP tells the clouds will clear and the temperature drop +2º-6º A classical, physical-meteorological, deterministic problem The skilled weather forecaster is invited to “add value” to the NWP by modifying the -6° forecast However, the real added value might be of some other kind... assume the clouds do not lift? →

5. 5 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 Assume the probability of clearing = 60% Three different forecasts might be provided I. A compromise forecast -3º for verifications II. A missed event is considered worse than a false alarm so -4º or -5º is forecast III.Special customers are told that there is a slightly higher probability (60%) for the clouds to disperse with -6º, rather than not (40%) with +2º All of these involve clever use of intuitive statistics

6. 6 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 Asymmetric cost- or penalty functions Error “Pain” Possible errors ●● ● ● “Pain” Forecasts for unspecified customer or for verification purposes Specific customer with sensitivity for missed cold events Forecast too mild Forecast too cold

7. 7 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 Expected mean cost/d £30 £20 £10 £0 Value of probabilities: The school book example Loss=£100 and average probability of bad weather p clim =30% £0 £30£60£90 protection cost £30 £20 £10 0 gain Never protect Always protect Deterministic forecast Perfect forecasts Useful forecasts Ob Fc R _ R2010 - 60

8. 8 WMO SWFDP Macau 9 April 2013 Anders Persson Ob Fc R _ R2010 - 60 Ob Prob R _ 10010 0 80 8 2 60 6 4 40 4 6 20 2 8 0 050 Ob Fc R - R10 0 ??20 - 050 CategoricalNon-categorical The value of uncertain weather forecasts Probabilistic

9. 9 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 Loss=£100 and average probability of bad weather p clim =30% gain Never protect Always protect Deterministic forecast Probabilistic forecasts Ob % R_ 10010 0 80 8 2 60 6 4 40 4 6 20 2 8 0 050 Expected mean cost/d £30 £20 £10 £0 £0 £30£60£90 protection cost £30 £20 £10 0

10. 10 WMO SWFDP Macau 9 April 2013 Anders Persson The intuitive-statistical nature of routine forecasting The forecasters work in an environment with a flow of information from different sources that might be incorrect, contradictory and unrepresentative

11. 11 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 2. The need of good “statistical intuition” has been the subject of learned books

12. 12 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 Not only meteorologists are concerned with risks and uncertainties

13. 13 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 Time constrains, limited and sometimes misleading information, stress and outside distraction (Almost) unlimited time, a wide range of reliable information and full concentration Fast thinking: Meteorologists in the forecast office Slow thinking: Meteorologists attending a seminar The title of Kahneman´s book refers to

14. 14 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 3. Five points where we humans have to improve on how to deal with uncertainties

15. 15 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/201515 Common human intuitive weaknesses 1. Over-confidence 2. Underestimation of randomness 3. Problems estimating uncertainty 4. Communicating this uncertainty 5. Drawing the conclusions from uncertainty

16. 16 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 3.1 Overconfidence

17. 17 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 17 3.1 Overconfidence: Before 2000 Concord was regarded as the safest airplane Concorde Other company __0___ 100 000 Flight hours < __1_____ 1 000 000 Flight hours __1___ 100 001 Flight hours > __1_____ 1 000 000 Flight hours...after the 2000 crash the most unsafe accidents

18. 18 WMO SWFDP Macau 9 April 2013 Anders Persson Mon 00Mon 12Tue 00Tue 12Wed 00Wed 12 Model 1 Model 2 Model 3 Model 1 Model 2 Model 3 Mon 00Mon 12Tue 00Tue 12Wed 00Wed 12 Over confidence Three forecasts from different NWP models valid at the same time in 5% of the cases Surely dry! in 80% of the cases Surely rain!

19. 19 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 3.2.Underestimating randomness

20. 20 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 20 Conditional sampling Comments from ECMWF Member States: 1.You overforecast Portuguese cut-offs at D+5, only 50% verify 2.You overforecast >25 mm/day events at D+3, only 50% verify 3.You overforecast gales at D+4, only 50% verify OB FC OB FC Many hits Under forecasting Few misses Over forecasting Well tuned forecasts

21. 21 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 3.3 Estimating uncertainty (probabilities)

22. 22 WMO SWFDP Macau 9 April 2013 Anders Persson Mon 00Mon 12Tue 00Tue 12Wed 00Wed 12 Forecast from model A Forecast from model B Forecast from model C The Halo Effect “…the atmosphere is inherently unpredictable due to the chaotic nature of its motions (Ørgård, 1963)…” “…the atmosphere is inherently unpredictable due to the chaotic nature of its motions (Lorenz, 1963)…” In forecasting meteorology: to weight one’s favourite NWP too much

23. 23 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 Barcelona 5 Sep 2012 Anders Persson 15 UTC chart15 UTC forecast 03 UTC chart (≡) = = 15 UTC forecast ☼ Δ Δ Δ Δ +24 h forecast ☼☼ The availability effect ( ) TS-risk 70% TS-risk 30% +12 h forecast

24. 24 WMO SWFDP Macau 9 April 2013 Anders Persson Mon 00Mon 12Tue 00Tue 12Wed 00Wed 12 Model 1 Model 2 Model 3 Model 1 Model 2 Model 3 Mon 00Mon 12Tue 00Tue 12Wed 00Wed 12 The primacy effect To order of arrival of the NWP may also affect the assessment “Yes, rain is possible” “No, I do not believe in rain”

25. 25 WMO SWFDP Macau 9 April 2013 Anders Persson in 63% of the cases Mon 00Mon 12Tue 00Tue 12Wed 00Wed 12 Forecast 1 Forecast 2 Forecast 3 Forecast 1 Forecast 2 Forecast 3 Mon 00Mon 12Tue 00Tue 12Wed 00Wed 12 Misleading consistency Three consecutive NWP from the same model valid at the same time in 58% of the cases Very “jumpy” Rather consistent

26. 26 WMO SWFDP Macau 9 April 2013 Anders Persson Consecutive forecasts tend to be correlated since the new observations do not change the “first guess” entirely The two best on average but also the most correlated ones i.e. their mutual agreement is less significant The best and the worst on average but also the least correlated ones. Their mutual agreement becomes more significant +24h +36h +48h +24h +36h +48h

27. 27 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 3.4 Communicating uncertainty

28. 28 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/201528 People react differently to a statement like: “-There is a 30% risk of rain” compared to ”- A 70% chance of dry weather” This is the “Framing effect” 3.4 Example of communicating probabilities

29. 29 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/201529 An example of a meteorological framing effect: The authorities react more appropriately to a probability forecast of 60% for a whole region (Midlands) than to 10- 20% for an individual location (Birmingham) 20% 60% 15% 10% Thunderstorm risk

30. 30 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 30 The Base rate effect 50% probability means different things Base rate 1.Tossing a coin: 50-50? = I do not know 50% 2.Snowfall in Barcelona: 50% very high risk! 2% 3.<4/8 clouds in Barcelona: 50% is a low “risk”! 80% It all depends on the “base rate”

31. 31 WMO SWFDP Macau 9 April 2013 Anders Persson The base rate in meteorology is the climatology The ECMWF:s Extreme Forecast Index (EFI) relates the probabilities to the climatology 09/08/201531 ECMWF’s new EFI chart 16 August 2012 12 UTC +60 to +84 h

32. 32 WMO SWFDP Macau 9 April 2013 Anders Persson Excessive rain risk Excessive hot

33. 33 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 D+8 forecast 7 December D+7 forecast 8 December D+6 forecast 9 December D+5 forecast 10 December The predicted arrivals of the 15-16 December storm (ECMWF and UKMO alike)

34. 34 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 The cyclone has changed track several times - we have revised our calculations No blame on the computer for the “jumpiness”

35. 35 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 35 The way the Met Office and BBC forecasters handled the weather situation was “very well received by senior managers in the BBC and the Met Office….and had been praised by the section of government which is responsible for the Met Office. “ No direct surveys of public opinion were made, “but informal feedback has been positive.” I’ll come back into more detail what the BBC/Met Office did

36. 36 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 3.5 Drawing conclusions from probabilities -What do you prefer? -An 80% chance of winning £1000 or -Get £700 directly?

37. 37 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 Summary of part 1-3: The forecasters will increasingly deal with forecast uncertainty and risk assessments, which will increase the public’s confidence and improve the weather forecasters´ reputation A five-point program is suggested on how to change the current deterministic culture: The greatest “threat” to the meteorological weather forecaster is not the computer but the growing number of non- meteorological weather forecasters with a modern outlook a)Reduce forecast over-confidence b)Understand the effects of randomness c)To estimate forecast uncertainty d)To convey probabilistic information e)To help the customers to make decisions

38. 38 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 4. Updating deterministic and ensemble forecasts

39. 39 WMO SWFDP Macau 9 April 2013 Anders Persson Meteorologists have five sources of information: 1.Observations 2.Deterministic NWP 3.Statistical interpretation 4.Ensemble forecasts 5.Climatological information Let’s start with the last one, point 5. (systematic errors and “jumpiness”) (irregular, varying quality and representative) (outdated or unrepresentative) (a lot of information with “probs”) (“only” background information)

40. 40 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 4.1 The problem seen from a typical PDF (probability density function) perspective – climate distribution

41. 41 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ 2% 14% 34% 34% 14% 2% Mean value +1 SD+2 SD-1 SD-2 SD Most likely values Higher than normal Much higher than normal Lower than normal Much lower than normal A typical climatological distribution (temperature)

42. 42 WMO SWFDP Macau 9 April 2013 Anders Persson very tricky for bi-modal distributions More tricky for rain or wind Mode (the most likely value) Median (divides the data into two equal halves) Mean (the average) Mode Median Mean Mode Median Mean What is one single “representative” value?

43. 43 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ Climatological average Probability The analysis observations The forecast

44. 44 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ Larger area = more certain Smaller area = less certain If the forecast is wrong it is more likely to be wrong “to the left” (less anomalous) than “to the right” (even more anomalous) The “Regression to the Mean Effect” Probability

45. 45 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ Climatological average More probable Less probable Probability

46. 46 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ Probability How certain is this NWP? We do not know! It could be very certain... or very uncertain NWP

47. 47 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ Probability Investigations show that “jumpiness” correlates badly to the accuracy of the last forecast NWP today NWP yesterday NWP the day before yesterday We might an opinion by looking at the last NWPs from the same model, so called “lagged” forecasts

48. 48 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ Probability The “jumpiness” correlates fairly well to the accuracy of the weighted average of the three NWP NWP today NWP yesterday NWP the day before yesterday This “poor man’s ensemble captures the essentials of the “rich man’s” ensemble 1.Ensemble mean 2.Spread 3.Rough “probs” Weighted average of the three NWP

49. 49 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ Probability Consensus NWP There are no exact rules on how to merge manual and NWP information Subjectively weighted average of the manual and NWP Manual Final?

50. 50 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ Probability Then arrives the EPS, the Ensemble Prediction System forecast Final Again, there are no exact rules on how to merge manual and NWP forecasts

51. 51 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ Probability Again, there are no exact rules on how to merge manual and EPS information EPS Manual+NWP New

52. 52 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ Probability The change in the most representative value, from the manual+NWP to the new forecast, is not much affected Manual+ NWP EPS New Minor change of deterministic value

53. 53 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ Probability The major change is in the spread of the forecasts, the (un)certainty Major change of probabilistic values Increased risk for extremes

54. 54 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 4.2 The problem seen from an EPS meteogram perspective

55. 55 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 Medium range forecasting…with deterministic and EPS information Latest three NWP Latest EPS Most common case with good agreement between EPS spread and NWP “jumpiness” Most common case

56. 56 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 Rather poor agreement between larger EPS spread and small NWP “jumpiness”. The analysis system has obviously managed to avoid possible problems because the NWP is not very “jumpy” Should the forecasters be more certain than the EPS indicates? Rather common case

57. 57 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 Rather poor agreement between small EPS spread and large NWP “jumpiness”. The perturbations have not been quite able to cover the analysis uncertainties Should the forecasters be more uncertain than the EPS indicates? Not uncommon case

58. 58 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 Poor agreement between the main directions of the EPS and the NWP This puts the forecasters in a very difficult situation and there is not enough experience or investigations about this situation Rare case Best choice: create a “super ensemble”

59. 59 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 4.3 The same seen from a PDF-perspective

60. 60 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ Probability Again, there are no exact rules on how to merge manual and EPS information EPS Manual+NWP New

61. 61 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ Probability EPS NWP Final Case 1: Lagged NWP agree with EPS and about the same spread

62. 62 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ Probability EPS NWP Final Case 2: Lagged NWP agree with EPS but has smaller spread

63. 63 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ Probability EPS NWP Final Case 3: Lagged NWP agree with EPS but has larger spread

64. 64 WMO SWFDP Macau 9 April 2013 Anders Persson Ψ Probability EPS NWP Final Case 4: Lagged NWP agree with EPS and about the same spread but quite different means

65. 65 WMO SWFDP Macau 9 April 2013 Anders Persson Summary of part 4 1.The forecaster has an increasing role to play as an “intuitive statistician” 2.The EPS must be compared and blended with more than one NWP, preferably 3-4 NWP 3.In the blending the spread and probabilities will normally be affected more than the “representative” value. Ensemble information tend to make us more uncertain – is that good?

66. 66 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 5. The value of uncertainty “per se”

67. 67 WMO SWFDP Macau 9 April 2013 Anders Persson Ob Fc R _ R2010 - 60 Ob Fc R - R10 0 ??20 - 050 CategoricalNon-categorical The value of uncertain weather forecasts Ob Fc R _ R3020 - 050 Ob Fc R _ R10 0 - 2070 Low protection cost High protection cost

68. 68 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 Loss=£100 and average probability of bad weather p clim =30% gain Ob Fc R _ R 2010 - 60 Low protection cost High protection cost Ob Fc R - R2010 ??20 -1060 Expected mean cost/d £30 £20 £10 £0 £0 £30£60£90 protection cost £30 £20 £10 0 This is not just playing with mathematics – this was the approach actually used by the Met Office and BBC in December 2011

69. 69 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 “Some terrible weather will come on Thursday- Friday” The BBC forecasters avoided going into detail and did not show any isobar maps

70. 70 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 On Wednesday 14 Dec still large uncertainty about the storm track

71. 71 WMO SWFDP Macau 9 April 2013 Anders Persson... and then finally the day before

72. 72 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 The Met Office repeated the approach 1 ½ month later

73. 73 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 The Met Office and the BBC didn’t hide, but made use of the uncertainty

74. 74 WMO SWFDP Macau 9 April 2013 Anders Persson My conclusions from the Met Office and BBC experience 1.Uncertainties can be communicated without numbers 2.The meteorologist must appear to be in control 3.Tell the background to the uncertainty – tell a “story” 4.Do not hesitate to give advice such as “if I were you…” 5.Follow up the forecast – but do not take to much credit

75. 75 WMO SWFDP Macau 9 April 2013 Anders Persson 09/08/2015 75 End

76. 76 WMO SWFDP Macau 9 April 2013 Anders Persson Observed rain 9-12 July 2004 50 mm Prognosis 8 July 30 mm Prognosis 7 July 40 mm Prognosis 5 July 30 mm Prognosis 6 July Expected rain for 9 July 2004 Example from Sweden The meteorologists’ forecasts

77. 77 WMO SWFDP Macau 9 April 2013 Anders Persson Observed rain 9-12 July 2004 Somewhere 50 mm Somewhere 30 mm Expected rain for 9 July 2004 The hydrologists’ forecasts

78. 78 WMO SWFDP Macau 9 April 2013 Anders Persson