Short Range Ensemble Prediction System Verification over Greece Petroula Louka, Flora Gofa Hellenic National Meteorological Service

HNMS involvement in COSMO-SREPS Verification of LM-COSMO ensemble forecasts for MAPD-Phase (06-12/2007) Cases of 72-hour forecast horizon, 16 members Verification domain: Greece Data used: SYNOP data covering Greece Parameters verified: 2m temperature Mean Sea Level Pressure (MSLP) Precipitation Statistical analysis of the results 10th COSMO General Meeting

COSMO-SREPS domain 10th COSMO General Meeting

Greek SYNOP stations 10th COSMO General Meeting

COSMO-SREPS members 10th COSMO General Meeting

Statistical analysis methods For continuous parameters such as Temperature and MSLP Bias RMSE For non-continuous parameters (precipitation) Deterministic approach Multi-category contingency tables POD, FAR, ETS Probabilistic approach (e.g., BSS, ROC diagrams, etc) 10th COSMO General Meeting

2m Temperature By Member Underestimation of Temperature. It seems that GME driven members have better skill than the others. IFS GME NCEP UKMO 10th COSMO General Meeting

2m Temperature By Month Underestimation of summer temperatures June 2007 was exceptionally warm with strong heat waves The maximum temperature in Athens reached 46.2 ºC ! October to December RMSE statistically acceptable. Different pattern between summer and autumn/winter months 10th COSMO General Meeting

POD plots dependence on Driving Model IFS GME NCEP UKMO 10th COSMO General Meeting

POD plots dependence on convective scheme etc. Tiedtke1 Kain-Fritsch1 Tiedtke2 Tiedtke3 10th COSMO General Meeting

FAR plots dependence on Driving Model IFS GME NCEP UKMO 10th COSMO General Meeting

FAR plots dependence on convective scheme etc. Tiedtke1 Kain-Fritsch1 Tiedtke2 Tiedtke3 10th COSMO General Meeting

Brier Skill Score BSS measures the improvement of the probabilistic forecast relative to the sample climatology = total frequency of the event (sample climatology) The forecast system has predictive skill if BSS is positive (better than climatology), a perfect system having BSS = 1 10th COSMO General Meeting

BSS plots The predictive skill is good (positive BSS) for the smaller precipitation thresholds The first day shows better scores than the other two (especially when compared to the third) The size of the sample affects the score for the larger precipitation thresholds 10th COSMO General Meeting

BSS plots dependence on Driving Model It seems that for the 1st day all members group together Some members show negative BSS for low thresholds For the 2nd and 3rd days IFS seems to provide better score The size of the sample affects the score 10th COSMO General Meeting

BSS plots dependence on Physical parameterisations Members with Tiedtke convective scheme group together The perturbations of the particular parameters for turbulent and length scale are less important than convective schemes It seems that the members with Kain-Fritsch convective scheme have worse performance than the others 10th COSMO General Meeting

Reliability Diagrams The Frequency of an observed event is plotted against the forecast probability of the event. If the curve lies below the 45° line, the probabilities are overestimated Points between the "no skill" line and the diagonal contribute positively to the BSS (resolution > reliability). 10th COSMO General Meeting

Reliability Diagrams 48hr Climatology – No resolution No skill Overestimation of the probability especially for the larger threshold, although Small sample for large thresholds 10th COSMO General Meeting

Relative Operating Characteristic ROC is a measure of forecast skill. ROC is a tool that permits to evaluate the ability of the forecast system to discriminate between occurrence and non-occurrence of a precipitation event (to detect the event) It measures resolution (YES or NO event), but not reliability (e.g. biased forecast) ROC area < 0.5 indicates no skill. 10th COSMO General Meeting

ROC Area ROC area values are generally high for the lower thresholds The ensemble can discriminate those events It seems that for the first predictive period (24hr) ROC area has larger values for more precipitation thresholds (up to 15mm/day) compared with the other two periods 10th COSMO General Meeting

ROC Area dependence on Driving Model In general all members contribute similarly Small sample for larger thresholds 10th COSMO General Meeting

ROC Area dependence on Physical parameterisations Members with Tiedtke convective scheme group together The perturbations of the particular parameters for turbulent and length scale are less important than convective schemes The members with Kain-Fritsch convective scheme seem to perform better => better resolution (but not good reliability, c.f. BSS diagrams) 10th COSMO General Meeting

ROC Curves Hit rates are plotted against the corresponding false alarm rates to generate the ROC Curve. The area under the ROC curve is used as a statistical measure of forecast usefulness. 10th COSMO General Meeting

ROC Curves 10th COSMO General Meeting

Remarks on Temperature Temperature is strongly (~5°C) underestimated during the summer months Both Bias and RMSE exhibited a diurnal cycle with the (absolute) maxima being during the hottest summer hours, while for the autumn/winter months the diurnal variation had the opposite behaviour The initial and boundary condition perturbations contribute to the BIAS and RMSE more than the physical parameter perturbations 10th COSMO General Meeting

Remarks on Precipitation Precipitation amount is overestimated It is not evident a consistent attitude for the forecasted precipitation driven by a certain initial and boundary condition model Perturbations of the convective schemes are more important than the perturbations of the particular parameters for turbulent and length scales used It seems that the members with Kain-Fritsch convective scheme have better resolution but worse reliability compared to the members with Tiedtke convective scheme 10th COSMO General Meeting

10th COSMO General Meeting

MSLP By Month 10th COSMO General Meeting

2m Temperature By Member Underestimation of Temperature. No particular effect of the scaling parameters is evident. Tiedtke1 Kain-Fritsch1 Tiedtke2 Tiedtke3 10th COSMO General Meeting

Reliability Diagrams 24hr No skill Climatology – No resolution 10th COSMO General Meeting

Reliability Diagrams 72hr No skill Climatology – No resolution 10th COSMO General Meeting