1. Short Range Ensemble Prediction System Verification over Greece
Petroula Louka, Flora Gofa
Hellenic National Meteorological Service

2. HNMS involvement in COSMO-SREPS
Verification of LM-COSMO ensemble forecasts for MAPD-Phase (06-12/2007)
Cases of 72-hour forecast horizon, 16 members
Verification domain: Greece
Data used: SYNOP data covering Greece
Parameters verified:
2m temperature
Mean Sea Level Pressure (MSLP)
Precipitation
Statistical analysis of the results
10th COSMO General Meeting

3. COSMO-SREPS domain
10th COSMO General Meeting

4. Greek SYNOP stations
10th COSMO General Meeting

5. COSMO-SREPS members
10th COSMO General Meeting

6. Statistical analysis methods
For continuous parameters such as Temperature and MSLP
Bias
RMSE
For non-continuous parameters (precipitation)
Deterministic approach
Multi-category contingency tables
POD, FAR, ETS
Probabilistic approach (e.g., BSS, ROC diagrams, etc)
10th COSMO General Meeting

7. 2m Temperature By Member
Underestimation of Temperature.
It seems that GME driven members have better skill than the others.
IFS
GME
NCEP
UKMO
10th COSMO General Meeting

8. 2m Temperature By Month Underestimation of summer temperatures
June 2007 was exceptionally warm with strong heat waves
The maximum temperature in Athens reached 46.2 ºC !
October to December RMSE statistically acceptable.
Different pattern between summer and autumn/winter months
10th COSMO General Meeting

9. POD plots dependence on Driving Model
IFS
GME
NCEP
UKMO
10th COSMO General Meeting

10. POD plots dependence on convective scheme etc.
Tiedtke1
Kain-Fritsch1
Tiedtke2
Tiedtke3
10th COSMO General Meeting

11. FAR plots dependence on Driving Model
IFS
GME
NCEP
UKMO
10th COSMO General Meeting

12. FAR plots dependence on convective scheme etc.
Tiedtke1
Kain-Fritsch1
Tiedtke2
Tiedtke3
10th COSMO General Meeting

13. Brier Skill Score
BSS measures the improvement of the probabilistic forecast relative to the sample climatology
= total frequency of the event
(sample climatology)
The forecast system has predictive skill if BSS is positive (better than climatology), a perfect system having BSS = 1
10th COSMO General Meeting

14. BSS plots
The predictive skill is good (positive BSS) for the smaller precipitation thresholds
The first day shows better scores than the other two (especially when compared to the third)
The size of the sample affects the score for the larger precipitation thresholds
10th COSMO General Meeting

15. BSS plots dependence on Driving Model
It seems that for the 1st day all members group together
Some members show negative BSS for low thresholds
For the 2nd and 3rd days IFS seems to provide better score
The size of the sample affects the score
10th COSMO General Meeting

16. BSS plots dependence on Physical parameterisations
Members with Tiedtke convective scheme group together
The perturbations of the particular parameters for turbulent and length scale are less important than convective schemes
It seems that the members with Kain-Fritsch convective scheme have worse performance than the others
10th COSMO General Meeting

17. Reliability Diagrams
The Frequency of an observed event is plotted against the forecast probability of the event.
If the curve lies below the 45° line, the probabilities are overestimated
Points between the "no skill" line and the diagonal contribute positively to the BSS
(resolution > reliability).
10th COSMO General Meeting

18. Reliability Diagrams 48hr
Climatology – No resolution
No skill
Overestimation of the probability especially for the larger threshold, although
Small sample for large thresholds
10th COSMO General Meeting

19. Relative Operating Characteristic
ROC is a measure of forecast skill.
ROC is a tool that permits to evaluate the ability of the forecast system to discriminate between occurrence and non-occurrence of a precipitation event (to detect the event)
It measures resolution (YES or NO event), but not reliability (e.g. biased forecast)
ROC area < 0.5 indicates no skill.
10th COSMO General Meeting

20. ROC Area ROC area values are generally high for the lower thresholds
The ensemble can discriminate those events
It seems that for the first predictive period (24hr) ROC area has larger values for more precipitation thresholds (up to 15mm/day) compared with the other two periods
10th COSMO General Meeting

21. ROC Area dependence on Driving Model
In general all members contribute similarly
Small sample for larger thresholds
10th COSMO General Meeting

22. ROC Area dependence on Physical parameterisations
Members with Tiedtke convective scheme group together
The perturbations of the particular parameters for turbulent and length scale are less important than convective schemes
The members with Kain-Fritsch convective scheme seem to perform better
=> better resolution (but not good reliability, c.f. BSS diagrams)
10th COSMO General Meeting

23. ROC Curves
Hit rates are plotted against the corresponding false alarm rates to generate the ROC Curve.
The area under the ROC curve is used as a statistical measure of forecast usefulness.
10th COSMO General Meeting

24. ROC Curves
10th COSMO General Meeting

25. Remarks on Temperature
Temperature is strongly (~5°C) underestimated during the summer months
Both Bias and RMSE exhibited a diurnal cycle with the (absolute) maxima being during the hottest summer hours, while for the autumn/winter months the diurnal variation had the opposite behaviour
The initial and boundary condition perturbations contribute to the BIAS and RMSE more than the physical parameter perturbations
10th COSMO General Meeting

26. Remarks on Precipitation
Precipitation amount is overestimated
It is not evident a consistent attitude for the forecasted precipitation driven by a certain initial and boundary condition model
Perturbations of the convective schemes are more important than the perturbations of the particular parameters for turbulent and length scales used
It seems that the members with Kain-Fritsch convective scheme have better resolution but worse reliability compared to the members with Tiedtke convective scheme
10th COSMO General Meeting

27. 10th COSMO General Meeting

28. MSLP By Month
10th COSMO General Meeting

29. 2m Temperature By Member
Underestimation of Temperature.
No particular effect of the scaling parameters is evident.
Tiedtke1
Kain-Fritsch1
Tiedtke2
Tiedtke3
10th COSMO General Meeting

30. Reliability Diagrams 24hr
No skill
Climatology – No resolution
10th COSMO General Meeting

31. Reliability Diagrams 72hr
No skill
Climatology – No resolution
10th COSMO General Meeting