1. Quantitative verification of cloud fraction forecasts
Robin Hogan
Malcolm Brooks
Trevor Flint

2. Overview
Now have over a year of cloud data from Chilbolton and Cabauw, and four models to compare with.
Previous comparisons of mean and PDF of cloud fraction and water content evaluate the climate of a model but not the weather.
In this talk I use skill scores, commonly used in precipitation verification, to evaluate cloud fraction forecasts in the four models.

3. Chilbolton Radar
ChilboltonLidar
Month of target classification

4. Derived cloud fraction
Use model winds/resolution to determine averaging time, count pixels to get volumetric cloud fraction in each model grid box (different grid for each model)

5. Met Office & ECMWF cloud fraction
Met Office Unified Model (mesoscale version)
ECMWF global model

6. RACMO & Météo France cloud fraction
KNMI Regional atmospheric climate model (RACMO)
Météo France ARPEGE model

7. Contingency tables
Comparison with Met Office model over Chilbolton, October 2003
Observed cloud Observed clear-sky
Model cloud
Model clear-sky
A: Cloud hit
B: False alarm
C: Miss
D: Clear-sky hit

8. Simple skill score: Hit Rate
Met Office short range forecast
Météo France old cloud scheme
Misleading: fewer cloud events so “skill” is only in predicting clear skies
Models which underestimate cloud will do better than they should
Hit Rate: fraction of forecasts correct = (A+D)/(A+B+C+D)
Consider all Cabauw data, 1-9 km
Increase in cloud fraction threshold causes apparent increase in skill.

9. Scores independent of clearsky hits
False alarm rate: fraction of forecasts of cloud which are wrong = B/(A+B)
perfect forecast is 0
Probability of detection: fraction of clouds correctly forecast = A/(A+C)
perfect forecast is 1
Skill decreases as cloud fraction threshold increases

10. More sophisticated scores
Equitable threat score =(A-E)/(A+B+C-E) where E removes those hits that occurred by chance.
Yule’s Q =(-1)/(+1) where the odds ratio =AD/BC.
Advantage: little dependence on frequency of cloud
For both scores, 1 = perfect forecast, 0 = random forecast
From now on use Equitable threat score with threshold of 0.1.

11. Skill versus height: Chilbolton
Lower skill for low clouds:
Both clouds and model levels are thinner, so more difficult to forecast?
As seen before, much more cloud in models than observations above 8 km
Could be due to radar sensitivity

12. Skill versus height: Cabauw
Model performance:
ECMWF, RACMO, Met Office models perform similarly
Météo France not so well, much worse before April 2003
Met Office model significantly better for shorter lead time

13. Skill versus time Cabauw Equitable threat score
Cabauw mean cloud fraction
Chilbolton Equitable threat score
Chilbolton mean cloud fraction
Change in Météo France cloud scheme April 2003

14. Forecast degredation and scale
Met Office forecasts available for different lead times
Forecast degradation can be quantified
Gridbox comparisons are very tough tests
Averaging in time gives a better indication of skill in forecasting large scale features

15. Conclusions Skill scores provide a quantitative framework for
Comparing one model against another
Evaluating impact of a new cloud scheme
Results so far show that
Met Office, ECMWF and RACMO perform similarly for similar forecast lead times
Short range Met Office forecasts are generally better
Météo France scores are the lowest, although much improved since April 2003 when the cloud scheme was changed
Future work
Other scores, e.g. ROC, Brier score etc.
Horizontal & vertical averaging to test model at different scales
Compare Met Office global & mesoscale models, new cloud schemes etc.