1. Slide 1 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 1 TIGGE-LAM: Archiving at ECMWF Manuel Fuentes Data and Services Section ECMWF

2. Slide 2 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 2 The TIGGE-Global database Three archive centres: CMA, NCAR and ECMWF Ten data providers: -ECMWF, JMA (Japan), UK Met Office (UK), CMA (China), NCEP (USA), MSC (Canada), Météo-France (France), BOM (Australia), KMA (Korea), CPTEC (Brazil) Exchanges using UNIDATA LDM, HTTP and FTP Operational since 1st of October 2006 174 TB, growing by ~ 1.5 TB/week -1.5 million fields/day

3. Slide 3 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 3 Archive Centre Data Providers NCAR NCEP CMC UKMO ECMWF MeteoFrance JMA KMA CMA BoM CPTEC TIGGE Archive Centres and Data Providers

4. Slide 4 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 4 Strong governance Precise definition of: -Which products: list of parameters, levels, steps, units,… -Which format: GRIB2 -Which transport protocol: UNIDATAs LDM -Which naming convention: WMO file name convention Only exception: the grid and resolution -Choice of the data provider. Data provider to provide interpolation to regular lat/lon -Best possible model output Many tools and examples: -Sample dataset available -Various GRIB2 tools, tigge_check validator, … -Scripts that implement exchange protocol Web site with documentation, sample data set, tools, news….

5. Slide 5 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 5 Using SMS to handle TIGGE flow

6. Slide 6 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 6 Quality assurance: homogeneity Homogeneity is paramount for TIGGE to succeed -The more consistent the archive the easier it will be to develop applications There are three aspects to homogeneity: -Common terminology (parameter names, file names,…) -Common data format (format, units, …) -Definition of an agreed list of products (Parameters, Steps, levels, …) What is not homogeneous: -Resolution -Base time (although most provider have a run at 12 UTC) -Forecast length -Number of ensemble

7. Slide 7 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 7 QA: Checking for homogeneity E.g. Cloud-cover: instantaneous or six hourly?

8. Slide 8 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 8 QA: Completeness The objective is to have 100% complete datasets at the Archive Centres Completeness may not be achieved for two reasons: -The transfer of the data to the Archive Centre fails -Operational activities at a data provider are interrupted and back filling past runs is impractical Incomplete datasets are often very difficult to use Most of the current tools (e.g. epsgrams) used for ensemble forecasts assume a fixed number of members from day to day -These tools will have to be adapted

9. Slide 9 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 9 QA: Checking completeness

10. Slide 10 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 10 Access to TIGGE data Data is available to Research and Education after 48 hours -Self registration by agreeing to the terms and conditions Portals at CMA, NCAR and ECMWF ECMWF portal offers: -Access to offline data -Aggregation along any axis (date, level, parameter, origin, ensemble, …) -Provision of multi-model data on a single grid (regridding to any lat/lon grid) -Sub-area selection -Reduces volumes to be downloaded by many order of magnitude -Batch access via the portal

11. Slide 11 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 11 TIGGE portal at ECMWF

12. Slide 12 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 12 TIGGE portal at ECMWF (using Ajax)

13. Slide 13 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 13 TIGGE Portal: grid selection

14. Slide 14 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 14 TIGGE Portal: area selection

15. Slide 15 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 15 TIGGE Portal: use SMS for scheduling users requests Monitoring Per user limits Global limits

16. Slide 16 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 16 TIGGE Portal: direct access to MARS Most of the data is on tape: access to long timeseries Even finer control of resource usage

17. Slide 17 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 17 TIGGE Portal: Batch access

18. Slide 18 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 18 TIGGE Portal: Active users

19. Slide 19 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 19 TIGGE Portal: Users by country

20. Slide 20 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 20 TIGGE Portal: Data volumes

21. Slide 21 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 21 TIGGE: Lessons learnt Strong commitment required: -During development: encoding, units -Day to day operations: failed jobs need to be re-run Robust suites and scripts -Problems with transmission or data corruption -Servers may not available (retry) -Keep data for at least 3 days Technical contacts point

22. Slide 22 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 22 TIGGE-LAM Archiving at ECMWF Re-use TIGGE-Global technology TIGGE-LAM Archiving meeting in September 2008 -Un-rotated lat/lon, 0.1 0, virtually on (0,0) -HP parameters, addition of LSP, ORO and LSM -Specific name for each LAM EPS configuration -Data format: GRIB2

23. Slide 23 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 23 TIGGE-LAM: Suite names for Europe LAM EPS SystemSuite Name at ECMWFInstitute -Consortium MOGREPSMOGREPS-MO- EUAUK Met Office - UK SREPSAEMet-SREPS-MM-EUATAEMET- Spain HIRLAM SRNWP PEPSSRNWP-PEPS COSMO-LEPSCOSMOLEPS-ARPASIMC-EUARPA-SIM - Italy COSMO NORLAMEPS Met.NO ALADIN LAEFALADIN-LAEFZAMG / Austria COSMO DE EPSCOSMO-DE EPSDWD - Germany SRNWP COSMO-SREPSCOSMO-SREPS-BO-EUARPA-SIM - Italy COSMO GLAMEPS DNMI/Univ Oslo âNorway HIRLAM ALADIN PEARCE Meteo-France / France DMI - HIRLAM DMI OMSZ ALADIN EPSOMSZ- ALADIN-EPS Hungary,Czech Republic, Croatia

24. Slide 24 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 24 TIGGE-LAM: Data format Encoding in GRIB2 -Support for all HP parameters Possibly add TIGGE-LAM GRIB2 local extension -Differentiate between different systems -Preserve Data Providers own local extensions For Data Providers that cannot provide GRIB2, ECMWF can help convert from GRIB1 using grib_api Examples from COSMO/LEPS and SREPS Send sample data to: tigge-technical@ecmwf.int

25. Slide 25 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 25 TIGGE-LAM: Data Transmission & Archiving No need for LDM -FTP, push of pull or other Filenames can be agreed between Data Provider and ECMWF -Date & time required to avoid overlapping ECMWF will archive data -Consistency checks performed after archiving

26. Slide 26 TIGGE-LAM Workshop Bologna 19-21 Jan 2009 26 Conclusion TIGGE-Global has been running for more than 2 years -Robust infrastructure in place Basis to ensure TIGGE-LAM success Thank you!