1. forecasts of rare events
Verification for
forecasts of rare events
Chris Ferro
Mathematics Research Institute
University of Exeter, UK
15 mins + 5 mins questions
11th International Meeting on Statistical Climatology
Edinburgh, 14 July 2010

2. Forecast verification
Describe and understand forecast performance.
We consider challenges raised by deterministic forecasts for the occurrence of large, rare values. Read more next year in the expanded second edition of ‘the book’!

3. Daily rainfall in mid-Wales
Observations: radar measurements Forecasts: direct output from the old 12km mesoscale version of the Met Office Unified Model 1 Jan 05 – 11 Nov 06
Data courtesy of Marion Mittermaier

4. Threshold exceedances
Observe
Forecast
Yes No 13 9 27
600
Hit rate = 13 / ( ) = 0.325

5. Threshold exceedances
Observe
Forecast
Yes No 7 3 12
627
Hit rate = 7 / (7 + 12) = 0.368

6. Threshold exceedances
Observe
Forecast
Yes No 3 5
638
Hit rate = 3 / (3 + 5) = 0.375

7. Threshold exceedances
Observe
Forecast
Yes No 1 2 4
642
Hit rate = 1 / (1 + 4) = 0.2

8. Threshold exceedances
Observe
Forecast
Yes No 1 3
645
Hit rate = 0 / (0 + 3) = 0

9. Threshold exceedances
Observe
Forecast
Yes No 1
648
Hit rate = 0 / (0 + 0) = NaN

10. Sampling variation increases
Hit rate decreases to 0
Sampling variation increases
95%
confidence
intervals

11. Proportion correct tends to 1
Sampling variation decreases
95%
confidence
intervals
(a+d)/n >= 1-2p since (b+c)/n <= 2p so constraint reduces absolute variation. Relative variation increases

12. Lessons
As we move to rarer events... forecast performance degenerates, sampling variation (uncertainty) can increase, some aspects of performance are hard to maintain e.g. hit rate, other aspects of performance are easy to maintain e.g. proportion correct.

13. Solutions
Verification measures that do not degenerate, e.g. Extreme Dependency Score measures decay rate. Reduce sampling variation by... imposing a parametric form on how the entries in the contingency table change with the threshold, estimating the parameters with moderate events, using the fitted model to extrapolate to rare events.
Ferro (2007), Stephenson et al. (2008), Ferro and Stephenson (2010)

14. Define the base rate to be p = (a + c) / n.
Table of frequencies
Observe
Forecast
Yes No a b
a + b c d
c + d
a + c
b + d n
Define the base rate to be p = (a + c) / n.

15. Table of relative frequencies
Observe
Forecast
Yes No
a / n
b / n q
c / n
d / n
1 – q p
1 – p 1
There is no theory for how q changes as p → 0 so recalibrate the forecasts to force q = p and model the bias, q / p, separately.

16. Recalibration
Use unequal thresholds to equalise the numbers of observed and forecasted events, removing bias.
Observe
Forecast
Yes No 9 10
620

17. Parametric model for the table
Observe
Forecast
Yes No
a / n • p
1 – p 1
For a wide class of (regularly varying) probability distributions, a / n ~ αpβ as p → 0 where α > 0 and β ≥ 1. For random forecasts, α = 1 and β = 2.
Ledford and Tawn (1996), Heffernan (2000)

18. Parametric model for the table
Observe
Forecast
Yes No
αpβ • p
1 – p 1
Given estimates of α and β, we can derive the behaviour of verification measures for small p, e.g. Hit rate ~ αpβ / p = αpβ–1.

19. Check model adequacy If the model fits then
log a / n ~ log α + β log p
for small p and the graph of log a / n against log p will approximate a straight line for small enough p.

20. Parameter estimation
If the model holds for p < p0 then the observations and forecasts can be transformed into a sample from a distribution for which Pr(Z > z) ≈ α exp(–βz) for z > –log p0. Choose p0 and then estimate α and β by maximum likelihood using those transformed data that exceed –log p0.

21. Daily rainfall in mid-Wales
Observe
Forecast
Yes No
0.97p1.25 • p
1 – p 1
The model is a good fit for base rates p < (not shown) with estimates α = 0.97 (0.71, 1.60) and β = 1.25 (1.11, 1.47).
90% confidence intervals

22. Standard (black) and model-based (red) estimates of
the hit rate with bootstrap 90% confidence intervals

23. Standard (black) and model-based (red) estimates of
prop. correct with bootstrap 90% confidence intervals

24. Summary
Forecast performance degenerates for rare events (some aspects are hard to maintain, others are easy to maintain) and uncertainty can increase. Probability models based on extreme-value theory can help to estimate performance more precisely and to extrapolate performance to rarer events.

25. Discussion
A model for how the bias changes with base rate could be included to estimate the performance of uncalibrated forecasts. Similar ideas can be used for other types of forecasts and observations, e.g. probabilistic forecasts, although different models are required. Other related issues include observation errors, location/timing errors, sensitivity to outliers, etc.

26. References c.a.t.ferro@ex.ac.uk
Ferro CAT (2007) A probability model for verifying deterministic forecasts of extreme events. Weather and Forecasting, 22, Ferro CAT, Stephenson DB (2010) Improved verification measures for deterministic forecasts of rare, binary events. In preparation. Heffernan JE (2000) A directory of coefficients of tail dependence. Extremes, 3, Ledford AW, Tawn JA (1996) Statistics for near independence in multivariate extreme values. Biometrika, 83, Stephenson DB, Casati B, Ferro CAT, Wilson CA (2008) The extreme dependency score: a non-vanishing verification score for deterministic forecasts of rare events. Meteorological Applications, 15,