1. The COSMO-LEPS system: present status and outlook Andrea Montani, Chiara Marsigli and Tiziana Paccagnella ARPA-SIM Hydrometeorological service of Emilia-Romagna, Italy
X General COSMO meeting
Cracow,15-19 September 2008
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

2. Present status and recent updates Possible changes and future plans
Outline
Present status and recent updates
Possible changes and future plans
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

3. The new COSMO-LEPS suite @ ECMWF before December 2007
3 levels
hPa
16 Representative Members driving the 16 COSMO-model integrations (weighted according to the cluster populations)
employing either Tiedtke or Kain-Fristch convection scheme (randomly choosen)
4 variables
Z U V Q
d-1 d
d+1
d+2
d+3
d+4
d+5
Cluster Analysis and RM identification
Cluster Analysis and RM identification
older EPS 00
younger EPS
2 time steps 12
clustering period
European area
Complete Linkage
suite running as a ”time-critical application” managed by ARPA-SIM;
Δx ~ 10 km; 40 ML;
COSM0-LM 3.20 since Nov06;
fc length: 132h;
Computer time (4.3 million BU for 2007) provided by the COSMO partners which are ECMWF member states.
COSMO-LEPS Integration Domain
COSMO-LEPS clustering area
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

4. Recent updates (1)
Changes to the operational suite on 1 December 2007:
new model version (COSMO-LM 4.0);
new numerics (Runge-Kutta scheme);
use of multi-layer soil model;
new perturbations introduced relative to the maximal turbulent length scale (tur_len) and to the length scale of thermal surface patterns (pat_len)  to increase spread and maintain skill.
Old system and new system ran in parallel for 52 days (October and November 2007);
Larger T2m spread in new system for all forecast ranges;
Comparable skill (in terms of T2m root-mean-square error) of the two systems.
Regarding the 5-RM ensembles results would seem to suggest that the use of just two EPS in the super-ensemble can be a good compromise, permitting to decrease the percentage of outliers significantly but leading only to a small decrease of the skill.
Regarding the impact of the ensemble size, the difference between each 5-member ensemble and the correspondent 10-member ensemble is remarkableThe impact of doubling the ensemble size is almost the same for every configuration and is larger than impact of changing the number of EPSs on which the Cluster Analysis is performed (2 or 3).
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

5. Recent updates (2)
Since 15/6/2008, JRC has got access to COSMO-LEPS data for the use in the experimental European Flood Alert System (EFAS)
In the framework PREVIEW Project (WP: Medium-Range Plain Flood), COSMO-LEPS showed added value for flood forecasting, with respect to ECMWF EPS and varEPS.
 COSMO-LEPS will be used, with ECMWF EPS and GME DWD deterministic forecasts, as input to the flood forecasting system over the major European river basins.
success for COSMO both from a scientific and a visibility point of view.
Regarding the 5-RM ensembles results would seem to suggest that the use of just two EPS in the super-ensemble can be a good compromise, permitting to decrease the percentage of outliers significantly but leading only to a small decrease of the skill.
Regarding the impact of the ensemble size, the difference between each 5-member ensemble and the correspondent 10-member ensemble is remarkableThe impact of doubling the ensemble size is almost the same for every configuration and is larger than impact of changing the number of EPSs on which the Cluster Analysis is performed (2 or 3).
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

6. COSMO-LEPS suite @ ECMWF since December 2007
16 Representative Members driving the 16 COSMO-model integrations (weighted according to the cluster populations)
employing either Tiedtke or Kain-Fristch convection scheme (randomly choosen) +
perturbations in turbulence scheme
3 levels
hPa
4 variables
Z U V Q
d-1 d
d+1
d+2
d+3
d+4
d+5
Cluster Analysis and RM identification
Cluster Analysis and RM identification
older EPS 00
younger EPS
2 time steps 12
clustering period
European area
Complete Linkage
suite runs as a ”time-critical application” managed by ARPA-SIM;
Δx ~ 10 km; 40 ML; fc+ 132h;
COSM0 v4.0 (with RK + multi-layer);
computer time (5.8 million BU for 2008) provided by the COSMO partners which are ECMWF member states.
COSMO-LEPS Integration Domain
COSMO-LEPS clustering area
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

7. Operational set-up Core products: Additional products:
16 perturbed COSMO-model runs (ICs and 3-hourly BCs from 16 EPS members) to generate, “via weights”, probabilistic output: start at 12UTC; t = 132h;
Additional products:
1 deterministic run (ICs and 3-hourly BCs from the high-resolution deterministic ECMWF forecast) to “join” deterministic and probabilistic approaches: start at 12UTC; t = 132h;
1 hindcast (or proxy) run (ICs and 3-hourly BCs from ECMWF analyses) to “downscale” ECMWF information: start at 00UTC; t = 36h.
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

8. but … A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

9. Semi-diurnal cycle in COSMO-LEPS scores
RPSS score for 12-hour cumulated precipitation … the higher the better …
Performance of the system assessed for the last 5 summers (JJA) and the last 5 autumns (SON).
RPSS
Evident 12-hour cycle in RPSS scores (the same holds for BSS, while less evident for ROC area scores).
Better performance of the system for “night-time” precipitation, that is for rainfall predicted between 18Z and 6Z (ranges 30-42h, 54-66h, …).
The amplitude of the cycle is somewhat reduced throughout the years (in summers) and with increasing forecast range.
Bad performance of the system in Summer 2006.
Regarding the 5-RM ensembles results would seem to suggest that the use of just two EPS in the super-ensemble can be a good compromise, permitting to decrease the percentage of outliers significantly but leading only to a small decrease of the skill.
Regarding the impact of the ensemble size, the difference between each 5-member ensemble and the correspondent 10-member ensemble is remarkableThe impact of doubling the ensemble size is almost the same for every configuration and is larger than impact of changing the number of EPSs on which the Cluster Analysis is performed (2 or 3).
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

10. Score dependence on the domain size (1)
Verification of COSMO-LEPS against synop reports over the MAP D-PHASE area (~ 470 stations; MAPDOM) and the full domain (~ 1500 stations; fulldom):
 different statistics of the verification samples;
 up to now, performance of the system over the 2 domains assessed only for 6 months (March-August 2007).
 difficult to draw general conclusions
Regarding the 5-RM ensembles results would seem to suggest that the use of just two EPS in the super-ensemble can be a good compromise, permitting to decrease the percentage of outliers significantly but leading only to a small decrease of the skill.
Regarding the impact of the ensemble size, the difference between each 5-member ensemble and the correspondent 10-member ensemble is remarkableThe impact of doubling the ensemble size is almost the same for every configuration and is larger than impact of changing the number of EPSs on which the Cluster Analysis is performed (2 or 3).
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

11. Score dependence on the domain size (2)
RPSS score… the higher the better… (and positive).
ROC area… the higher the better… (and above 0.6).
Outliers percentage … the lower the better.
RPSS
Smoother transitions from month to month in “fulldom” scores.
Slightly better performance of COSMO-LEPS over the MAPDOM, but the signal varies from month to month.
Higher predictability with orographic forcing?
 Need to check individual regions and/or to stratify for type of stations.
OUTL
ROC
Regarding the 5-RM ensembles results would seem to suggest that the use of just two EPS in the super-ensemble can be a good compromise, permitting to decrease the percentage of outliers significantly but leading only to a small decrease of the skill.
Regarding the impact of the ensemble size, the difference between each 5-member ensemble and the correspondent 10-member ensemble is remarkableThe impact of doubling the ensemble size is almost the same for every configuration and is larger than impact of changing the number of EPSs on which the Cluster Analysis is performed (2 or 3).
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

12. Overestimation of Td2m and soil moisture (1)
Verification period: MAM07 and MAM08.
Obs: synop reports (about 470 stations x day).
Region: 43-50N, 2-18E (MAP D-PHASE area).
Larger bias and larger rmse in MAM08 rather than in MAM07 for COSMO-LEPS deterministic run (in 2007, no multi-layer soil model).
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

13. Overestimation of Td2m and soil moisture (2)
from Andrè Walser
COSMO-LEPS members have higher soil moisture than COSMO-7 run at Meteoswiss (case of 20/7), especially in the Alps.
For the future, do not simply interpolate from ECMWF IFS to COSMO model with the multi-layer soil model.
generate “home-made” soil moisture analysis (from COSMO-EU, from a proxy-run, …)
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

14. Model explosion for COSMO-LEPS
every now and then, “Floating-point exceptions” when the model writes output files over small subdomains …
Temporarily patch (which works, anyway!):
the failed task is resubmitted with half time-step.
A more sophisticated investigation of model explosions will be carried out in the near future.
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

15. Feasibility study of COSMOLEPS at 7 km (cleps_7)
Motivations:
Provide a more detailed description of mesoscale processes by incresing the horizontal resolution.
Do not lose a “reasonable advantage” against ECMWF EPS, which will go to x=25 km during 2009.
“Keep the pace” with deterministic model (x~ 2-3 km): if the gap in resolutions between deterministic and probabilistic systems is too large, the two systems go for different solutions (that is, they forecast different weather!).
from 10 to 7 km (plus small domain extensions) does not seem a lot
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

16. but … A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

17. COSMO-LEPS at 7 km (cleps_7): the answer to forecasters’ dream?
Present system
x = 10 km
z = 40 ML
t = 90 s
ngp = 306x258x40 =
fcst range = 132h
cost = 640 BU x run
elapsed time = 45 min
New system
x = 7 km
z = 40 ML
t = 72 s
ngp = 510x405x40 =
fcst range = 132h
cost = 1925 BU x run
elapsed time = 138 min
… cleps_7 is about 3 times more expensive than the present configuration
new computer at ECMWF being installed
Computer resources for each ECMWF member state will increase by a factor of 5 (five)  and ….
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

18. The dream is possible COSMO-LEPS 10 km COSMO-LEPS 7 km
the grid of cleps_7 would be almost identical to that of COSMO-EU, this making easier and cleaner the use of initial fields provided by DWD (e.g. soil moisture analysis).
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

19. Future plans (2008 and 2009) tackle the soil moisture deficiency:
use the soil moisture analysis fields provided by DWD,
use the fields provided by the “proxy-run”,
run cleps_7 for ~ 40 days in autumn 2008 and assess the impact;
within TIGGE-LAM, develop coding of COSMO-LEPS output files in GRIB2 format;
migration to the new machine at ECMWF;
use a better snow analysis (possibly provided by DWD or Meteoswiss);
extend the cluster analysis so as to consider not only ECMWF EPS, but also UKMO MOGREPS as global ensemble providing ic’s and bc’s (first tests);
implementation of cleps_7;
gaining from COSMO-SREPS experience, introduce more model perturbations;
test COSMO-LEPS nested on the under-development ECMWF EDA during MAP D‑PHASE period;
support calibration and verification.
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

20. Thank you !
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

21. Main results
COSMO-LEPS system runs on a daily basis since November 2002 (6 “failures” in almost 6 years of activity) and it has become a “member-state time-critical application” at ECMWF ( ECMWF operators involved in the suite monitoring).
COSMO-LEPS products used in EC Projects (e.g. PREVIEW), scientific experiments (e.g. COPS, MAP D-PHASE, EFAS) and met-ops rooms across COSMO community.
Time series scores cannot easily disentangle improvements related to COSMO-LEPS itself from those due to better boundaries by ECMWF EPS.
Nevertheless, positive trends can be identified:
increase in ROC area scores and reduction in outliers percentages;
positive impact of increasing the population from 5 to 10 members (June 2004);
some deficiency in the skill of the system were identified after the system upgrades occurred on Feb 2006 (from 10 to 16 members; from 32 to 40 ML + EPS upgrade!!!), but scores are encouraging throughout 2007 and, to a certain extent, 2008.
2 more features:
marked semi-diurnal cycle in COSMO-LEPS scores (better skill for “night-time” forecasts);
better scores over the Alpine area rather than over the full domain (to be confirmed).
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

22. Average values (boxes 0.5 x 0.5)
MAM06
Average values (boxes 0.5 x 0.5)
tp > 1mm/24h
tp > 5mm/24h
As regards AVERAGE precipitation above these two threshols, the 3 systems have similar performance.
COSMO-LEPS
16-MEMBER EPS
51-MEMBER EPS
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

23. ENSEMBLE SIZE REDUCTION IMPACT EVALUATED ON CASE STUDIES (1)
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

24. 5 RMs 10 RMs All 51 20 mm 100 mm 2003082512 Friuli (+72-+96)
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008
20 mm
100 mm

25. ENSEMBLE SIZE REDUCTION IMPACT EVALUATED ON CASE STUDIES (2)
Observed precipitation between UTC and UTC
Piedmont case
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

26. 5 RMs 10 RMs All 51 20 mm 150 mm 2002111212 Piedmont
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008
20 mm
150 mm

27. COSMO-LEPS Real time products
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

28. Why Limited Area Ensemble Prediction?
Global Ensemble Prediction Systems
have become extremely important tools to tackle the problem of predictions beyond day 2
are usually run at a coarser resolution with respect to deterministic global predictions → skill in forecasting intense and localised events is currently still limited.
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

29. Why Limited Area Ensemble Prediction? (2)
As regards high resolution deterministic forecast in the short range, where limited-area models play the major role, a “satisfactory” QPF is still one of the major challenges. The same can be said for other local parameters.
This is due, among other reasons, to the inherently low degree of predictability typical of severe and localised events.
Probabilistic/Ensemble approach is so required also for the short range at higher resolution
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

30. From Global EPS to LAM EPS
In the Limited-area ensemble systems, tailored for the short range, perturbations must be already “active” during the first hours of integration
The characteristic of the LAM ensemble are strongly dependent by the lateral boundaries forcing.
Due to the “regional” application of these Limited Area Ensembles, methodologies can be different in different geographycal regions.
A pratical consideration:
Global EPS ~ Big Centres
Limited Area EPS ~ (also) Relatively Small Centres
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

31. OBJECTIVE VERIFICATION OF COSMO-LEPS
In the last period the verification package is being developed keeping into account two measure of precipitation:
the cumulative volume of water deployed over a specific region
the rainfall peaks which occur within this region
COSMO observations
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

32. Verification grid OBS MASK CLEPS EPS
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

33. Average values boxes 0.5x0.5 deg
3 sis
Average values boxes 0.5x0.5 deg
tp > 1mm/24h
tp > 5mm/24h
COSMO-LEPS
16-MEMBER EPS
51-MEMBER EPS
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

34. tp > 1mm/24h tp > 5mm/24h tp > 10mm/24h COSMO-LEPS
NOCC=2671
tp > 1mm/24h
ave 0.5
NOCC=1195
tp > 5mm/24h
NOCC=610
tp > 10mm/24h
COSMO-LEPS
16-MEMBER EPS
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

35. Maximum values boxes 0.5x0.5 deg
3 sis
Maximum values boxes 0.5x0.5 deg
tp > 1mm/24h
tp > 5mm/24h
COSMO-LEPS
16-MEMBER EPS
51-MEMBER EPS
tp > 10mm/24h
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

36. Average values boxes 0.5x0.5 deg
NOCC=227
tp > 20mm/24h
COSMO-LEPS
16-MEMBER EPS
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

37. COSMO-LEPS weighting procedure maximum values (boxes 0.5x0.5 deg)
tp > 1mm/24h
COSMO-LEPS NW
COSMO-LEPS W
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

38. The COSMO-LEPS suite @ ECMWF November 2002 – May 2004
3 levels
hPa
4 variables
Z U V Q
d-1 d
d+1
d+2
d+3
d+4
d+5
5 Representative Members
Driving the 5
COSMO-model integrations
oldest EPS
oldest EPS 12
Cluster Analysis and RM identification
Cluster Analysis and RM identification
middle EPS 00
youngest EPS
2 time steps 12
clustering period
European area
Complete Linkage
COSMO-LEPS clustering area
COSMO-LEPS Integration Domain
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

39. The COSMO-LEPS suite @ ECMWF June 2004 – January 2006
3 levels
hPa
4 variables
Z U V Q
d-1 d
d+1
d+2
d+3
d+4
d+5
10 Representative Members
driving the 10
COSMO-model integrations
employing either Tiedtke or Kain-Fristch scheme randomly choosen
Cluster Analysis and RM identification
Cluster Analysis and RM identification
middle EPS 00
youngest EPS
2 time steps 12
clustering period
European area
Complete Linkage
Suite running in real time at ECMWF managed by ARPA-SIM;
Δx ~ 10 km
Fc length: 120h
COSMO-LEPS Integration Domain
COSMO-LEPS clustering area
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

40. COSMO-LEPS (developed at ARPA-SIM)
What is it?
It is a Limited-area Ensemble Prediction System (LEPS), based on COSMO-model and implemented within COSMO (COnsortium for Small-scale MOdelling, which includes Germany, Greece, Italy, Poland, Romania, Switzerland).
Why?
Because the horizontal resolution of global-model ensemble systems is limited by computer time constraints and does not allow a detailed description of mesoscale and orographic-related processes.
The forecast of heavy precipitation events can still be inaccurate (in terms of both locations and intensity) after the short range.
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

41. COSMO-LEPS project
combine the advantages of global-model ensembles with the high-resolution details gained by the LAMs, so as to identify the possible occurrence of intense and localised weather events (heavy rainfall, strong winds, temperature anomalies, snowfall, …)
generation of COSMO-LEPS in order to improve the Late-Short (48hr) to Early-Medium (132hr) range forecast of the so-called “severe weather events”.
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

42. COSMO-LEPS methodology
Dim 2
Possible evolution scenarios
Dim 1
Initial conditions
Cluster members chosen as representative members (RMs)
ensemble size reduction
Dim 2
Initial conditions
Dim 1
LAM integrations driven by
RMs
LAM scenario
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

43. Time series of Brier Skill Score
BSS is written as 1-BS/BSref. Sample climate is the reference system. Useful forecast systems if BSS > 0.
BS measures the mean squared difference between forecast and observation in probability space.
Equivalent to MSE for deterministic forecast.
BSS
Jun04: 5m  10m
Feb06: 10m16m; 32ML  40 ML
fc step: 30-42h
improvement of performance detectable for all thresholds along the years;
still problems with high thresholds, but good trend in 2007.
Regarding the 5-RM ensembles results would seem to suggest that the use of just two EPS in the super-ensemble can be a good compromise, permitting to decrease the percentage of outliers significantly but leading only to a small decrease of the skill.
Regarding the impact of the ensemble size, the difference between each 5-member ensemble and the correspondent 10-member ensemble is remarkableThe impact of doubling the ensemble size is almost the same for every configuration and is larger than impact of changing the number of EPSs on which the Cluster Analysis is performed (2 or 3).
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

44. Time series of ROC area for COSMO-LEPS (2)
Area under the curve in the FAR vs HIT diagram.
Valuable forecast systems have ROC area values > 0.6
ROC
Jun04: 5m  10m
Feb06: 10m  16m; 32ML  40 ML
fc step: 78-90h
the positive impact of increasing the ensemble size in 2004 is evident for all thresholds and for different fcst ranges.
poor performance of the system in Spring and Summer 2006, despite system upgrades.
Regarding the 5-RM ensembles results would seem to suggest that the use of just two EPS in the super-ensemble can be a good compromise, permitting to decrease the percentage of outliers significantly but leading only to a small decrease of the skill.
Regarding the impact of the ensemble size, the difference between each 5-member ensemble and the correspondent 10-member ensemble is remarkableThe impact of doubling the ensemble size is almost the same for every configuration and is larger than impact of changing the number of EPSs on which the Cluster Analysis is performed (2 or 3).
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

45. Time series of ROC area ROC
Area under the curve in the HIT rate vs FAR diagram.
Valuable forecast systems have ROC area values > 0.6.
Jun04: 5m  10m
Feb06: 10m  16m; 32ML  40 ML
fc step: 30-42h
ROC
fc step: 78-90h
Jun04: 5m  10m
Feb06: 10m  16m; 32ML  40 ML
the positive impact of increasing the ensemble size in 2004 is evident for all thresholds and for different fcst ranges.
poor performance of the system in Spring and Summer 2006 (both particularly dry), despite upgrades.
Regarding the 5-RM ensembles results would seem to suggest that the use of just two EPS in the super-ensemble can be a good compromise, permitting to decrease the percentage of outliers significantly but leading only to a small decrease of the skill.
Regarding the impact of the ensemble size, the difference between each 5-member ensemble and the correspondent 10-member ensemble is remarkableThe impact of doubling the ensemble size is almost the same for every configuration and is larger than impact of changing the number of EPSs on which the Cluster Analysis is performed (2 or 3).
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

46. Outliers: time series + seasonal scores
How many times the analysis is out of the forecast interval spanned by the ensemble members.
… the lower the better …
Performance of the system assessed as time series and for 5 different Summers (JJA).
Jun04: 5m  10m
Feb06: 10m  16m; 32ML  40 ML
OUTL
Evident seasonal cycle (more outliers in winter), but overall reduction of outliers in the years.
Reduction of outliers from one Summer to the other, related to the increase of ensemble size (more evident for the 5 to 10 increase).
Need to take into account the different statistics for each season (JJA 2003 less rainy than the others).
Regarding the 5-RM ensembles results would seem to suggest that the use of just two EPS in the super-ensemble can be a good compromise, permitting to decrease the percentage of outliers significantly but leading only to a small decrease of the skill.
Regarding the impact of the ensemble size, the difference between each 5-member ensemble and the correspondent 10-member ensemble is remarkableThe impact of doubling the ensemble size is almost the same for every configuration and is larger than impact of changing the number of EPSs on which the Cluster Analysis is performed (2 or 3).
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

47. Outliers: time series + seasonal scores
How many times the analysis is out of the forecast interval spanned by the ensemble members.
… the lower the better …
Performance of the system assessed for 4 different Autumns.
OUTL
Jun04: 5m  10m
Feb06: 10m  16m; 32ML  40 ML
Continuous reduction of outliers from one year to the other, also related to the increase of ensemble size (more evident for the 5 to 10 increase, rather than from 10 to 16).
Need to take into account the different statistics for each season (SON 2006 less rainy than the others).
Regarding the 5-RM ensembles results would seem to suggest that the use of just two EPS in the super-ensemble can be a good compromise, permitting to decrease the percentage of outliers significantly but leading only to a small decrease of the skill.
Regarding the impact of the ensemble size, the difference between each 5-member ensemble and the correspondent 10-member ensemble is remarkableThe impact of doubling the ensemble size is almost the same for every configuration and is larger than impact of changing the number of EPSs on which the Cluster Analysis is performed (2 or 3).
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

48. Semi-diurnal cycle in COSMO-LEPS scores
BSS score … the higher the better …
Performance of the system assessed for 5 different Summers (JJA).
BSS
Evident 12-hour cycle in BSS scores (the same holds for RPSS, while less evident for ROC area scores).
Better performance of the system for “night-time” precipitation, that is for rainfall predicted between 18Z and 6Z (ranges 30-42h, 54-66h, …).
The amplitude of the cycle is somewhat reduced throughout the years and with increasing forecast range.
The bad performance in Summer 2006 is confirmed.
Regarding the 5-RM ensembles results would seem to suggest that the use of just two EPS in the super-ensemble can be a good compromise, permitting to decrease the percentage of outliers significantly but leading only to a small decrease of the skill.
Regarding the impact of the ensemble size, the difference between each 5-member ensemble and the correspondent 10-member ensemble is remarkableThe impact of doubling the ensemble size is almost the same for every configuration and is larger than impact of changing the number of EPSs on which the Cluster Analysis is performed (2 or 3).
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008

49. Main results
COSMO-LEPS system runs on a daily basis since November 2002 (6 “failures” in almost 5 years of activity) and it has become a “member-state time-critical application” at ECMWF ( ECMWF operators involved in the suite monitoring).
COSMO-LEPS products used in EC Projects (e.g. PREVIEW) , field campaigns (e.g. COPS, MAP D-PHASE) and met-ops rooms across COSMO community.
Time series scores cannot easily disentangle improvements related to COSMO-LEPS itself from those due to better boundaries by ECMWF EPS.
Nevertheless, positive trends can be identified:
increase in ROC area scores and reduction in outliers percentages;
positive impact of increasing the population from 5 to 10 members (June 2004);
although some deficiency in the skill of the system were identified after the system upgrades occurred on February 2006 (from 10 to 16 members; from 32 to 40 model levels + EPS upgrade!!!), scores are encouraging throughout 2007.
2 more features:
marked semi-diurnal cycle in COSMO-LEPS scores (better skill for “night-time” forecasts);
better scores over the Alpine area rather than over the full domain (to be confirmed).
A.Montani; The COSMO-LEPS system: present status and outlook
COSMO meeting, Cracow, September 2008