Slide 1 TIGGE-LAM Workshop Bologna Jan TIGGE-LAM: Archiving at ECMWF Manuel Fuentes Data and Services Section ECMWF

Slide 2 TIGGE-LAM Workshop Bologna Jan The TIGGE-Global database Three archive centres: CMA, NCAR and ECMWF Ten data providers: -ECMWF, JMA (Japan), UK Met Office (UK), CMA (China), NCEP (USA), MSC (Canada), Météo-France (France), BOM (Australia), KMA (Korea), CPTEC (Brazil) Exchanges using UNIDATA LDM, HTTP and FTP Operational since 1st of October TB, growing by ~ 1.5 TB/week -1.5 million fields/day

Slide 3 TIGGE-LAM Workshop Bologna Jan Archive Centre Data Providers NCAR NCEP CMC UKMO ECMWF MeteoFrance JMA KMA CMA BoM CPTEC TIGGE Archive Centres and Data Providers

Slide 4 TIGGE-LAM Workshop Bologna Jan Strong governance Precise definition of: -Which products: list of parameters, levels, steps, units,… -Which format: GRIB2 -Which transport protocol: UNIDATAs LDM -Which naming convention: WMO file name convention Only exception: the grid and resolution -Choice of the data provider. Data provider to provide interpolation to regular lat/lon -Best possible model output Many tools and examples: -Sample dataset available -Various GRIB2 tools, tigge_check validator, … -Scripts that implement exchange protocol Web site with documentation, sample data set, tools, news….

Slide 5 TIGGE-LAM Workshop Bologna Jan Using SMS to handle TIGGE flow

Slide 6 TIGGE-LAM Workshop Bologna Jan Quality assurance: homogeneity Homogeneity is paramount for TIGGE to succeed -The more consistent the archive the easier it will be to develop applications There are three aspects to homogeneity: -Common terminology (parameter names, file names,…) -Common data format (format, units, …) -Definition of an agreed list of products (Parameters, Steps, levels, …) What is not homogeneous: -Resolution -Base time (although most provider have a run at 12 UTC) -Forecast length -Number of ensemble

Slide 7 TIGGE-LAM Workshop Bologna Jan QA: Checking for homogeneity E.g. Cloud-cover: instantaneous or six hourly?

Slide 8 TIGGE-LAM Workshop Bologna Jan QA: Completeness The objective is to have 100% complete datasets at the Archive Centres Completeness may not be achieved for two reasons: -The transfer of the data to the Archive Centre fails -Operational activities at a data provider are interrupted and back filling past runs is impractical Incomplete datasets are often very difficult to use Most of the current tools (e.g. epsgrams) used for ensemble forecasts assume a fixed number of members from day to day -These tools will have to be adapted

Slide 9 TIGGE-LAM Workshop Bologna Jan QA: Checking completeness

Slide 10 TIGGE-LAM Workshop Bologna Jan Access to TIGGE data Data is available to Research and Education after 48 hours -Self registration by agreeing to the terms and conditions Portals at CMA, NCAR and ECMWF ECMWF portal offers: -Access to offline data -Aggregation along any axis (date, level, parameter, origin, ensemble, …) -Provision of multi-model data on a single grid (regridding to any lat/lon grid) -Sub-area selection -Reduces volumes to be downloaded by many order of magnitude -Batch access via the portal

Slide 11 TIGGE-LAM Workshop Bologna Jan TIGGE portal at ECMWF

Slide 12 TIGGE-LAM Workshop Bologna Jan TIGGE portal at ECMWF (using Ajax)

Slide 13 TIGGE-LAM Workshop Bologna Jan TIGGE Portal: grid selection

Slide 14 TIGGE-LAM Workshop Bologna Jan TIGGE Portal: area selection

Slide 15 TIGGE-LAM Workshop Bologna Jan TIGGE Portal: use SMS for scheduling users requests Monitoring Per user limits Global limits

Slide 16 TIGGE-LAM Workshop Bologna Jan TIGGE Portal: direct access to MARS Most of the data is on tape: access to long timeseries Even finer control of resource usage

Slide 17 TIGGE-LAM Workshop Bologna Jan TIGGE Portal: Batch access

Slide 18 TIGGE-LAM Workshop Bologna Jan TIGGE Portal: Active users

Slide 19 TIGGE-LAM Workshop Bologna Jan TIGGE Portal: Users by country

Slide 20 TIGGE-LAM Workshop Bologna Jan TIGGE Portal: Data volumes

Slide 21 TIGGE-LAM Workshop Bologna Jan TIGGE: Lessons learnt Strong commitment required: -During development: encoding, units -Day to day operations: failed jobs need to be re-run Robust suites and scripts -Problems with transmission or data corruption -Servers may not available (retry) -Keep data for at least 3 days Technical contacts point

Slide 22 TIGGE-LAM Workshop Bologna Jan TIGGE-LAM Archiving at ECMWF Re-use TIGGE-Global technology TIGGE-LAM Archiving meeting in September Un-rotated lat/lon, 0.1 0, virtually on (0,0) -HP parameters, addition of LSP, ORO and LSM -Specific name for each LAM EPS configuration -Data format: GRIB2

Slide 23 TIGGE-LAM Workshop Bologna Jan TIGGE-LAM: Suite names for Europe LAM EPS SystemSuite Name at ECMWFInstitute -Consortium MOGREPSMOGREPS-MO- EUAUK Met Office - UK SREPSAEMet-SREPS-MM-EUATAEMET- Spain HIRLAM SRNWP PEPSSRNWP-PEPS COSMO-LEPSCOSMOLEPS-ARPASIMC-EUARPA-SIM - Italy COSMO NORLAMEPS Met.NO ALADIN LAEFALADIN-LAEFZAMG / Austria COSMO DE EPSCOSMO-DE EPSDWD - Germany SRNWP COSMO-SREPSCOSMO-SREPS-BO-EUARPA-SIM - Italy COSMO GLAMEPS DNMI/Univ Oslo âNorway HIRLAM ALADIN PEARCE Meteo-France / France DMI - HIRLAM DMI OMSZ ALADIN EPSOMSZ- ALADIN-EPS Hungary,Czech Republic, Croatia

Slide 24 TIGGE-LAM Workshop Bologna Jan TIGGE-LAM: Data format Encoding in GRIB2 -Support for all HP parameters Possibly add TIGGE-LAM GRIB2 local extension -Differentiate between different systems -Preserve Data Providers own local extensions For Data Providers that cannot provide GRIB2, ECMWF can help convert from GRIB1 using grib_api Examples from COSMO/LEPS and SREPS Send sample data to:

Slide 25 TIGGE-LAM Workshop Bologna Jan TIGGE-LAM: Data Transmission & Archiving No need for LDM -FTP, push of pull or other Filenames can be agreed between Data Provider and ECMWF -Date & time required to avoid overlapping ECMWF will archive data -Consistency checks performed after archiving

Slide 26 TIGGE-LAM Workshop Bologna Jan Conclusion TIGGE-Global has been running for more than 2 years -Robust infrastructure in place Basis to ensure TIGGE-LAM success Thank you!