1. Verification of probabilistic forecasts: comparing proper scoring rules
Thordis L. Thorarinsdottir and Nina Schuhen

2. Introduction Proper scoring rules:
measure the accuracy of a forecast
assign numerical penalty
Often used to rank different models or forecasters
For both deterministic and probabilistic verification
Propriety: expected score is optimized for true distribution

3. Real-life forecast szenario
Which proper scoring rule should I use?
What if they give conflicting results?
How should I report results?
Is my data set sufficient?
Which should I use for model parameter optimization?
Short: How to use proper scores in practice!

4. Proper scoring rules Squared error: Absolute error: Ignorance score:
Continuous ranked probability score:

5. Scores behave differently…

6. Simulation study: concept
Draw random data from a «true» distribution with
Verifying observations: 1000 data points
Training data: 300 data points for each observation
Estimate forecast distributions from the training data (method of moments)
Make forecasts from the estimated distributions (50 members)
Evaluate against observations

7. Forecasting distributions
Expected value
Variance
Normal
non- central t
log-normal
Gumbel
+ true distribution
= Euler-Mascheroni constant

8. Example forecast: scores vs. obs
IGN has different minimum due to skewness of the Gumbel distribution
All scores are minimized at the same value => proper

9. Mean scores and bootstrap intervals
Only IGN has large difference between Gumbel and other forecasters
1000 forecasts
log-normal is best if truth is unknown
10^6 forecasts
True distribution always has the lowest score, same ranking for all scores

10. PIT histograms (normal sample size)

11. PIT histograms (huge sample size)

12. Variation: Gumbel true distribution
Estimated Gumbel has lower mean score than the true distribution
1000 forecasts
10^6 forecasts
True distribution always has the lowest score, same ranking for all scores

13. Summary For a huge sample size, all proper scores give the same result
For more realistic sample sizes, they differ widely
The best model doesn’t always get the best score
AE and CRPS have trouble identifying appropriate distributions
Ignorance score is sensitive to shape of distributions
=> There is no «best» scoring rule

14. Summary For robust results:
Use error bars!
Use a combination of scores
CRPS is very useful if the distribution is unknown or can not be easily specified
Minimum score estimation: CRPS or Maximum Likelihood?
No clear answer
Depends on the forecast situation and model choice

15. Read more in… Statistical Postprocessing of Ensemble Forecasts
Editors: Stéphane Vannitsem, Daniel S. Wilks, Jakob W. Messner
Elsevier
Planned publication: September 2018