Can we distinguish wet years from dry years? Simon Mason Seasonal Forecasting Using the Climate Predictability Tool Bangkok, Thailand, 12 – 16 January 2015

2 Seasonal Forecasting Using the Climate Predictability Tool The ROC The ROC answers the question: Can the forecasts distinguish an event from a non-event? Are we more confident it will be dry when it is dry compared to when it is not? – Do we forecast less rain when it is dry compared to when it is not dry? – Do we issue a higher forecast probability for below-normal when it is below-normal compared to when it is not?

3 Seasonal Forecasting Using the Climate Predictability Tool Two-Alternative Forced Choice Test In which of these two Januaries did El Niño occur (Niño3.4 index >27°C)? What is the probability of getting the answer correct? 50%(assuming that you do not have inside information about ENSO).

4 Seasonal Forecasting Using the Climate Predictability Tool Two-Alternative Forced Choice Test In which of these two Januaries did El Niño occur (Niño3.4 index >27°C)? What is the probability of getting the answer correct? That depends on whether we can believe the forecasts. Select the forecast with the higher probability.

5 Seasonal Forecasting Using the Climate Predictability Tool Two-Alternative Forced Choice Test Retroactive forecasts of MAM rainfall for Thailand. How well do the forecasts distinguish “dry” years (driest 20%) from other years? Is the forecast for “dry” years less than for other years?

6 Seasonal Forecasting Using the Climate Predictability Tool Two-Alternative Forced Choice Test It is easier to calculate by sorting the forecasts so the driest forecast are at the top. We can then count how many of the non-dry years are lower in the table than the dry years. For 2010: 14 of the 15 non-dry years have lower probabilities. For 1998: 14 of 15 For 1995: 14 of 15 For 2005: 14 of 15 For 1992: 12 of 15 In total: 68 of 75 ≈ 91%.

7 Seasonal Forecasting Using the Climate Predictability Tool Two-Alternative Forced Choice Test If the forecasts could perfectly discriminate the dry years, the forecastss would be drier than for all the non-dry years, and the dry years would be listed at the top of the table. If the forecasts could not discriminate the dry years at all, they would be randomly distributed through the table, and there would be a 50% chance of the forecast being drier than on a non-dry year.

8 Seasonal Forecasting Using the Climate Predictability Tool ROC Retroactive forecasts of MAM rainfall for Thailand. Which year are you most confident is a dry year?

9 Seasonal Forecasting Using the Climate Predictability Tool ROC The most sensible strategy would be to list the years in order of increasing forecast rainfall. If the forecasts are good, the “dry” years should be at the top of the list.

10 Seasonal Forecasting Using the Climate Predictability Tool ROC For the first guess: Repeat for all forecasts.

11 Seasonal Forecasting Using the Climate Predictability Tool ROC

12 Seasonal Forecasting Using the Climate Predictability Tool ROC The area beneath the red curve, 0.91, gives us the probability that we will successfully discriminate a “dry” year from a non- dry year. The area beneath the blue curve, 0.85, gives us the probability that we will successfully discriminate a “wet” year from a non- wet year.

13 Seasonal Forecasting Using the Climate Predictability Tool ROC The bottom left indicates whether the forecasts with strong indications of dry (or wet) are good. Can they indicate that an event will occur? The top right indicates whether the forecasts with strong indications of not dry (or not wet) are good. Can the indicate that an event will not occur?

14 Seasonal Forecasting Using the Climate Predictability Tool Relative Operating Characteristics

15 Seasonal Forecasting Using the Climate Predictability Tool Exercises Diagnose the quality of your forecast models by analysing the ROC graphs.