COSMO General Meeting – Moscow Sept 2010 Some results from operational verification in Italy Angela Celozzi - Federico Grazzini Massimo Milelli - Elena Oberto Adriano Raspanti - Maria Stefania Tesini

COSMO General Meeting – Moscow Sept 2010 Verification  CROSS MODEL COSMOME vs ECMWF  CROSS MODEL COSMOI7 vs ECMWF  CROSS MODEL COSMOME vs COSMOIT  COSMOME –Upper Air  COSMOI7 – Upper Air (OBS and Analysis)  CONDITIONAL VERIFICATIONS  Long Term and Seasonal Precipitation

COSMO General Meeting – Moscow Sept 2010 Verification  CROSS MODEL COSMOME vs ECMWF  CROSS MODEL ECMWF vs COSMOI7  CROSS MODEL COSMOME vs COSMOIT  COSMOME –Upper Air  COSMOI7 – Upper Air  CONDITIONAL VERIFICATIONS  Long Term and Seasonal Precipitation

COSMO General Meeting – Moscow Sept 2010 COSMOME vs ECMWF Temperature SON JJA MAM DJF

COSMO General Meeting – Moscow Sept 2010 COSMOME vs ECMWF Dew Point Temperature SON JJA MAM DJF

COSMO General Meeting – Moscow Sept 2010 COSMOME vs ECMWF Mean Sea Level Pressure SON JJA MAM DJF

COSMO General Meeting – Moscow Sept 2010 COSMOME vs ECMWF Total Cloud Cover SON JJA MAM DJF

COSMO General Meeting – Moscow Sept 2010 COSMOME vs ECMWF Wind Speed SON JJA MAM DJF

COSMO General Meeting – Moscow Sept 2010 Verification  CROSS MODEL COSMOME vs ECMWF  CROSS MODEL ECMWF vs COSMOI7  CROSS MODEL COSMOME vs COSMOIT  COSMOME –Upper Air  COSMOI7 – Upper Air  CONDITIONAL VERIFICATIONS  Long Term and Seasonal Precipitation

COSMO General Meeting – Moscow Sept 2010 COSMOI7 vs ECMWF Dew Point Temperature SON JJA MAM DJF

COSMO General Meeting – Moscow Sept 2010 COSMOI7 vs ECMWF Mean Sea Level Pressure SON JJA MAM DJF

COSMO General Meeting – Moscow Sept 2010 COSMOI7 vs ECMWF Wind Speed SON JJA MAM DJF

COSMO General Meeting – Moscow Sept 2010 COSMOI7 vs ECMWF Total Cloud Cover SON JJA MAM DJF

COSMO General Meeting – Moscow Sept 2010 Verification  CROSS MODEL COSMOME vs ECMWF  CROSS MODEL ECMWF vs COSMOI7  CROSS MODEL COSMOME vs COSMOIT  COSMOME –Upper Air  COSMOI7 – Upper Air  CONDITIONAL VERIFICATIONS  Long Term and Seasonal Precipitation

COSMO General Meeting – Moscow Sept 2010 Temperature COSMOME vs COSMOIT SON JJA MAM DJF

COSMO General Meeting – Moscow Sept 2010 Mean Sea Level Pressure COSMOME vs COSMOIT SON JJA MAM DJF

COSMO General Meeting – Moscow Sept 2010 Dew PointTemperature COSMOME vs COSMOIT SON JJA MAM DJF

COSMO General Meeting – Moscow Sept 2010 Total Cloud Cover COSMOME vs COSMOIT SON JJA MAM DJF

COSMO General Meeting – Moscow Sept 2010 Wind Speed COSMOME vs COSMOIT SON JJA MAM DJF

COSMO General Meeting – Moscow Sept 2010 Conclusion COSMO - ME generally better than IFS, except MSLP COSMO – I7 better or almost the same than IFS Comparison COSMO-ME and COSMO-IT shows improvements for High-Res.

COSMO General Meeting – Moscow Sept 2010 Verification  CROSS MODEL COSMOME vs COSMOIT  CROSS MODEL COSMOME vs ECMWF  CROSS MODEL ECMWF vs COSMOI7  COSMOME – Upper Air  COSMOI7 – Upper Air  CONDITIONAL VERIFICATIONS  Long Term and Seasonal Precipitation

COSMO General Meeting – Moscow Sept 2010 COSMOME –Upper Air Temperature

COSMO General Meeting – Moscow Sept 2010 COSMOME –Upper Air Wind Speed

COSMO General Meeting – Moscow Sept 2010 Verification  CROSS MODEL COSMOME vs COSMOIT  CROSS MODEL COSMOME vs ECMWF  CROSS MODEL ECMWF vs COSMOI7  COSMOME –UpperAir  COSMOI7 – Upper Air  CONDITIONAL VERIFICATIONS  Long Term and Seasonal Precipitation

COSMO General Meeting – Moscow Sept 2010 COSMOI7 –Upper Air Temperature

COSMO General Meeting – Moscow Sept 2010 COSMOI7 –Upper Air Wind Speed

COSMO General Meeting – Moscow Sept 2010 COSMO I7COSMO BackUpECMWF MSL FC+48 – Each model is verified against its own analysis Shaded contouring every 0.5 hPa, starting from 0.5. Red and blue lines represent Positive/Negative bias, every 0.5 hPa Spatial distribution of mean absolute error (MAE), computed over MAM 2010 UPPER-AIR (against analysis)

COSMO General Meeting – Moscow Sept 2010 Forecast Step MAE growth with forecast step, computed over COSMOI7 domain Spring 2010 (MAM 2010) – All models and analyses are interpolated on A regular grid at 0.25 * 0.25 deg of h-resolution. Everyone against its own analysis. MSL UPPER-AIR (against analysis)

COSMO General Meeting – Moscow Sept 2010 Spatial distribution of mean absolute error (MAE), computed over MAM 2010 T850 FC+48 – Each model is verified against its own analysis Shaded contouring every 0.5 C°, starting from 0.5. Red and blue lines represent Positive/Negative bias, every 0.5 C° COSMO I7 COSMO BackUpECMWF UPPER-AIR (against analysis)

COSMO General Meeting – Moscow Sept 2010 MAE growth with forecast step, computed over COSMOI7 domain Spring 2010 (MAM 2010) – All models and analyses are interpolated on A regular grid at 0.25 * 0.25 deg of h-resolution. Everyone against its own analysis. Z 700 hPa UPPER-AIR (against analysis)

COSMO General Meeting – Moscow Sept 2010 Conclusion COSMO – ME and COSMO-I7 have a general good result in upper air Verification COSMO-ME seems better, but improvement from MAM for COSMO-I7 (bug in AOF file until march)

COSMO General Meeting – Moscow Sept 2010 Verification  CROSS MODEL COSMOME vs COSMOIT  CROSS MODEL COSMOME vs ECMWF  CROSS MODEL ECMWF vs COSMOI7  COSMOME –Upper Air  COSMOI7 – Upper Air  CONDITIONAL VERIFICATIONS  Long Term and Seasonal Precipitation

COSMO General Meeting – Moscow Sept 2010 Conditional Verification Temp – TCC obs <=35% Worse behaviour for all the seasons Compare to no condition model SON MAM DJF

COSMO General Meeting – Moscow Sept 2010 Conditional Verification Temp – TCC obs >=75% SON MAM DJF Better behaviour for all the seasons Compare to no condition model

COSMO General Meeting – Moscow Sept 2010 Conditional Verification MSLP – MSLP >=mean SON MAM DJF

COSMO General Meeting – Moscow Sept 2010 Conditional Verification Tdew – Wind Speed (Obs) <=2 m/s SON MAM DJF Almost indifferent to the condition in obs space

COSMO General Meeting – Moscow Sept 2010 Conditional Verification Tdew – Wind Speed (fcs) <=2 m/s SON MAM DJF Worse behaviour for all the seasons In fcs space

COSMO General Meeting – Moscow Sept 2010 Conditional Verification Temp – Prec +06 <= 0,5 Temp – Prec +06 <=10

COSMO General Meeting – Moscow Sept 2010 Conditional Verification Temp – MSLP >=mean SON MAM DJF Worse behaviour in DJF for RMSE Similar fo other seasons

COSMO General Meeting – Moscow Sept 2010 Conditional Verification Temp – MSLP <=mean SON MAM DJF General better behaviour for all the Seasons compare to NC

COSMO General Meeting – Moscow Sept 2010 Conclusion Comparison between NC and Cond verification seems effective in most of the cases A standard set of Conditions should be decided by WG5 and produce on regular basis

COSMO General Meeting – Moscow Sept 2010 Verification  CROSS MODEL COSMOME vs COSMOIT  CROSS MODEL COSMOME vs ECMWF  CROSS MODEL ECMWF vs COSMOI7  COSMOME –Upper Air  COSMOI7 – Upper Air  CONDITIONAL VERIFICATIONS  Long Term and Seasonal Precipitation

COSMO General Meeting – Moscow Sept 2010 Seasonal trend - low thresholds All the versions present a seasonal cycle with an overestimation during summertime (except COSMO-7 and I2) COSMO-7 and I2 underestimate Overestimation error decreases in D+2 (spin-up effect vanished) QPF verification of the 4 model versions at 7 km res. (COSMO-I7, COSMO-7, COSMO-EU, COSMO-ME) with the 2 model versions at 2.8 km res. (COSMO- I2, COSMO-IT) Dataset: high resolution network of rain gauges coming from COSMO dataset and Civil Protection Department  1300 stations Method: 24h/6h averaged cumulated precipitation value over 90 meteo-hydrological basins

COSMO General Meeting – Moscow Sept 2010 Very light improvement trend Seasonal error cycle: lower ets during winter and summertime no significant differences between D+1 and D+2 Last winter (very snowy particularly in Northern Italy): low ets value (D+1 and D+2)  model error or lack of representativeness of the rain gauges over the plain during snowfall ? Seasonal trend - low thresholds

COSMO General Meeting – Moscow Sept 2010 Very light improvement trend Seasonal cycle with more false alarms in summertime (particularly for I7) no significant differences between D+1 and D+2 Seasonal trend - low thresholds

COSMO General Meeting – Moscow Sept 2010 Quite stationary during last seasons I2 has very low values during summer no significant differences between D+1 and D+2 Seasonal trend - low thresholds

COSMO General Meeting – Moscow Sept 2010 Driving model comparison: ECMWF/COSMO- ME/COSMO-IT, low thresholds ECMWF tendency to forecast low rainfall amounts  big overestimation, big false alarms, very low ets, quite good pod Better prediction for COSMO-models (no strong differences between ME and IT)

COSMO General Meeting – Moscow Sept 2010 ECMWF tendency to forecast low rainfall amounts  big overestimation, big false alarms, very low ets, quite good pod Better prediction for COSMO-models BUT bad performance during summertime Driving model comparison: ECMWF/COSMO- ME/COSMO-IT, low thresholds

COSMO General Meeting – Moscow Sept 2010 Slight bias reduction during latest seasons Last winter: all the versions overestimate (probably due to lack of representativeness of the rain gauges over the plain during snowfall) Strong COSMO-7 underestimation BUT slight improvement during latest seasons Seasonal trend - high thresholds

COSMO General Meeting – Moscow Sept 2010 Low values during summertime In general, quite stationary error since son2008 up to now All the versions present a jump around son2008: ets increases from up to (cosmo-I7: son2008 introduction of 4.3 version with new T2m diagnostic) Skill decreases with forecast time Seasonal trend - high thresholds

COSMO General Meeting – Moscow Sept 2010 Seasonal trend - high thresholds Slight far reduction during last two years BUT high values during summer2009 and winter 2010 (probably due to lack of representativeness of the rain gauges over the plain during snowfall) Small far increase last spring

COSMO General Meeting – Moscow Sept 2010 Low probability of detection during intense convective events It is noticeable the improvement since son2008 BUT a subsequent worsening during 2009 and the first half of 2010 Skill decreases with forecast time Seasonal trend - high thresholds

COSMO General Meeting – Moscow Sept 2010 ECMWF difficulty to forecast high rainfall amounts  bias around 1 BUT big false alarms, very low ets and pod Better prediction for COSMO-models Driving model comparison: ECMWF/COSMO-ME/COSMO-IT, high thresholds

COSMO General Meeting – Moscow Sept 2010 ECMWF difficulty to forecast high rainfall amounts  bias around 1 BUT big false alarms, very low ets and pod Better prediction for COSMO-models Driving model comparison: ECMWF/COSMO- ME/COSMO-IT, high thresholds

COSMO General Meeting – Moscow Sept 2010 Diurnal cycle - low thresholds Little initial spin-up (especially for I7 and I2) No strong performance differences among the versions Slight diurnal cycle Slight worsening with forecast time

COSMO General Meeting – Moscow Sept 2010 Diurnal cycle - high thresholds Little initial spin-up vanished with threshold increasing No strong performance differences among the versions except COSMO-7 underestimation Pronounced worsening with forecast time

COSMO General Meeting – Moscow Sept 2010 COSMO-7COSMO-I7COSMO-ME COSMO-EU COSMO-I2COSMO-IT Bias, 10mm/24h Systematic overestimation over Alpine areas, especially in the western part and in Veneto/Trentino-Alto Adige (incorrect representation of flow interaction with alpine chain during westerlies and north-easterlies ?) COSMO-7 underestimates especially in southern Italy (border of the domain ?) COSMO-I7 overestimates the Adriatic areas (especially during north-easterly flow  forecasters experience) COSMO-I2 underestimates, COSMO-IT overestimates

COSMO General Meeting – Moscow Sept 2010 Far, 10mm/24h COSMO-7COSMO-I7COSMO-ME COSMO-EU COSMO-I2COSMO-IT More false alarms over Sardinia and Alpine areas, especially in the western part and in Veneto/Trentino-Alto Adige (incorrect representation of flow interaction with alpine chain during westerlies and north-easterlies ?) More false alarms for COSMO-I7 (and the other ones) over the Adriatic areas (especially during north-easterly flow  forecasters experience)

COSMO General Meeting – Moscow Sept 2010 Relative error %, SON 2009 COSMO-7COSMO-I7COSMO-ME COSMO-EU COSMO-I2COSMO-IT Too precipitation amount over Alpine areas, especially in the western part and in Veneto/Trentino-Alto Adige (incorrect representation of flow interaction with alpine chain during westerlies and north-easterlies ?) Few QPF for COSMO-7, COSMO-I2 and COSMO-IT Quite good QPF for COSMO-I7, COSMO-ME and COSMO-EU

COSMO General Meeting – Moscow Sept 2010 Relative error %, DJF 2010 COSMO-7COSMO-I7COSMO-ME COSMO-EU COSMO-I2COSMO-IT Few precipitation amount (all the versions except COSMO-I2) in the Padana Plain: very snowy winter also in plain areas where there are no heated rain gauges  lack of representativeness Few QPF for COSMO-7 in southern Italy

COSMO General Meeting – Moscow Sept 2010 Relative error %, MAM 2010 COSMO-7COSMO-I7COSMO-ME COSMO-EU COSMO-I2COSMO-IT Too precipitation amount over Alpine areas, especially in the western part and in Veneto/Trentino-Alto Adige (incorrect representation of flow interaction with alpine chain during westerlies and north-easterlies ?) Few QPF for COSMO-7, COSMO-I2 and COSMO-IT Quite good QPF for COSMO-I7, COSMO-ME and COSMO-EU