1. Accessing EURO4M and UERRA data in ECMWF MARS archive Richard Mládek, Manuel Fuentes (ECMWF)

2. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse ECMWF MARS MARS - (Meteorological Archive and Retrieval System) GRIB, BUFR and ODB (in future NetCDF) contains about 100 PB of data very easy access to the Archives via a pseudo-meteorological language managed archive => the data has to follow a certain structure, based on archiving and retrieval patterns (needs to know how the data is going to be produced/used before deciding how to store it) https://software.ecmwf.int/wiki/display/UDOC/MARS+user+documentatio n

3. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse ECMWF GRIB-API --> grib_ls uerra-sample.grib2 uerra-sample.grib2 edition centre date dataType gridType stepRange typeOfLevel level shortName packingType 2 eswi 20100101 an lambert 0 hybrid 50 t grid_simple 2 eswi 20100101 an lambert 0 heightAboveGround 100 t grid_simple 2 eswi 20100101 an lambert 0 isobaricInhPa 500 t grid_simple 2 eswi 20100101 an lambert 0 heightAboveGround 2 2t grid_simple 4 of 4 grib messages in uerra-sample.grib2 4 of 4 total grib messages in 1 files The ECMWF GRIB API is an application program interface accessible from C, FORTRAN and Python programs developed for encoding and decoding WMO FM-92 GRIB edition 1 and edition 2 messages. A useful set of command line tools is also provided to give quick access to GRIB messages. Manuals & installation package: https://software.ecmwf.int/wiki/display/GRIB https://software.ecmwf.int/wiki/display/GRIB

4. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse EURO4M testbed sample datasets archived  data in original GRIB1 format archived as “it is“ (no common definitions of the parameters) UERRA datasets not ready yet  Data in WMO compliant GRIB2 format (UERRA common definitions for all parameters will be compulsory) EURO4M / UERRA datasets differences

5. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse EURO4M testbed Datasets: COSMO (DWD) HIRLAM (SMHI) MESCAN (MF) UM/4DVAR (MO) By the end of 2015 four models archived for common period 2008-9 Only 15 selected surface parameters (if available) Much more data from MO because it was archived already in 2012 during EURO4M Data is varying among centres (different parameters, starting times, levels..) => potential complications for users Data is available only to users with ECMWF accout via standard MARS retrieval tools

6. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse Centre UM/4DVAR (MO)HIRLAM (SMHI) MESCAN (MF)COSMO (DWD) Domain size (projection) 480*384 (rot.latlon) 326*341 (rot.latlon) 1080*1000 (Lamb.conf) 848*824 (rot.latlon) Typesfcan / fc an Times3, 9, 15, 21h0, 6, 12, 18h 0, 1, 2...23h Steps(fc only)0, 1, 2...27h6, 12…36h6h Levelsml(71) / pl(20) / sfcsfc Parameters7(ml)/5(pl)/51(sfc)5(an) /15(fc)3(an) / 4(fc)16 (fluxes are not accumulated!) ml..model levels pl..pressure levels sfc..surface level an..analysis fc..forecast rot.latlon..Rotated Latitude/Longitude grid Lamb.conf..Lambert Conformal EURO4M testbed

7. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse EURO4M testbed (fc) Centre UM/4DVAR (MO) HIRLAM (SMHI) MESCAN (MF) Parameters type=fc level=sfc o 1.5m specific humidity o 1.5m temperature over land o 10 metre wind gust in the last 24 hours o 10m U wind over land o 10m V wind over land o Clear-sky (II) down surface sw flux o Clear-sky (II) up surface sw flux o Convective precipitation (water) o Convective rainfall rate o Convective snow o Convective snowfall rate o Fraction of sea- ice in sea o High cloud cover o Land-sea mask o Large scale rainfall rate o Large scale precipitation o Large scale snow o Large scale snowfall rate o Latent heat flux o Long wave radiation flux o Low cloud cover o Maximum temperature at 1.5m since previous post- processing o Mean sea level pressure o Medium cloud cover o Minimum temperature at 1.5m since previous post- processing o Net long-wave radiation flux (surface) o Net short-wave radiation flux (surface) o Orography o Precipitation rate o Relative humidity at 1.5m o Sensible heat flux o Short wave radiation flux at surface o Short wave radiation flux at top of atmosphere o Soil temperature layer 1 o Soil temperature layer 2 o Soil temperature layer 3 o Soil temperature layer 4 o Surface pressure o Temperature o Total Cloud Cover o Total Precipitation o Total cloud amount - random overlap o Total cloud amount in lw radiation o Total column water vapour o Very low cloud amount o Visibility at 1.5m o Volumetric soil water layer 1 o Volumetric soil water layer 2 o Volumetric soil water layer 3 o Volumetric soil water layer 4 o 10 metre U wind component o 10 metre V wind component o 2 metre temperature o Convective precipitation (water) o Convective snow o Global radiation flux o Large scale precipitation o Large scale snow o Long wave radiation flux o Net long-wave radiation flux (surface) o Net short-wave radiation flux (surface) o Relative humidity o Snow depth, cold snow o Total Cloud Cover o Total precipitation o 10 metre U wind component o 10 metre V wind component o Surface solar radiation downwards o Surface thermal radiation downwards

8. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse EURO4M testbed (an) Centre HIRLAM (SMHI)MESCAN (MF) COSMO (DWD) Parameters type=an level=sfc o 10 metre U wind component o 10 metre V wind component o 2 metre temperature o Relative humidity o Total Cloud Cover o 2 metre temperature o Relative humidity o Total precipitation o 10 metre U wind component o 10 metre V wind component o 2 metre temperature o 2m Relative Humidity o Convective rain rate o Convective snowfall rate water equivalent o Downward direct short wave radiation flux at surface (mean over forecast time) Initialisation o Geometric Height of the earth surface above sea level o Large scale rain rate o Large scale snowfall rate water equivalent o Max 2m Temperature (i) Initialisation o Min 2m Temperature (i) Initialisation o Net short wave radiation flux (at the surface) o Snow depth o Total Cloud Cover o Total Precipitation

9. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse 2 main MARS key words identifying EURO4M datasets: 1)class = rm 2)expver = 1, 2, 3, 4 1 => UM/4DVAR (MO) 2 => HIRLAM (SMHI) 3 => MESCAN (MF) 4 => COSMO (DWD) Data volume (13.4 GB): UM/4DVAR (MO): 7.8 GB HIRLAM (SMHI): 4.1 GB MESCAN (MF): 952 MB COSMO (DWD): 525 MB EURO4M in MARS

10. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse 1) via MARS batch requests run on unix command line (traditional the most common way in the past) 2) via web MARS catalogue (smaller samples up to 1 month) 3) via ECMWF WebAPI (an alternative way for downloading MARS data in a programmatic way via Web) => the recommended way => the same mars batch requests as for (1) can be used MARS access

11. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse MARS access via WebAPI ECMWF WebAPI is a set of services developed by ECMWF to allow users from the outside to access some internal features and data of the centre. So far 2 services: 1) Access MARS (the most general; account at ECMWF required) 2) Access ECMWF public datasets (TIGGE, ERA40, S2S, UERRA…) EURO4M

12. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse Accessing EURO4M (via WebAPI) Follow 3 easy steps: 1) Install ECMWF WebAPI (unix & Python based) follow instructions at https://software.ecmwf.int/wiki/display/WEBAPI/Access+MARS & download the “mars” sample script 2) Check EURO4M data availability browse MARS web catalogue at http://apps.ecmwf.int/mars-catalogue/?class=rm or write MARS list request (mars_list.batch) and run it:mars mars_list.batch 3) Get the data Write MARS retrieval request (mars.batch) and run it:mars mars.batch

13. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse Accessing EURO4M (2. Check EURO4M data availability via MARS web catalogue) 3-4.2.2016UERRA Workshop Toulouse13 Check for availability View MARS request Estimate download size Retrieve.. Retrieve now Active request Finished! Download the data

14. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse Checking EURO4M data availability 2. write MARS list request (mars_list.batch) and run it using WebAPI: mars mars_list.batch list, class = rm, date = 2008-01-01, expver = 2, type = fc, levtype = all, param = all, time = all, stream = oper, output = tree, tar = euro4m.hirlam.2008-01-01.tree Example of mars_list.batch: class=rm, expver=2, levtype=sfc, stream=oper, type=fc, date=2008-01-01, time=00:00:00/06:00:00/12:00:00/18:00:00, step=12/18/24/30/36/6, param=11.1/111.1/112.1/115.1/117.1/138.1/33.1/34. 1/52.1/61.1/62.1/63.1/71.1/78.1/79.1 Output: euro4m.hirlam.2008-01-01.txt

15. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse Retrieving EURO4M 3. write MARS retrieval request (mars.batch) and run it using WebAPI: mars mars.batch retrieve, class = rm, date = 2008-01-01, expver = 2, type = fc, param = all, time = all, stream = oper, tar = euro4m-hirlam.grib2 Example of mars.batch: Output: GRIB2 file euro4m-hirlam.grib2

16. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse UERRA at ECMWF https://software.ecmwf.int/wiki/display/UER => Watch the News page!

17. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse UERRA datasets 8 datasets coming from 5 models and 4 centres The different models have slightly different domains

18. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse UERRA parameters The final list of parameters: 8 parameters on 3 types of vertical levels (model, pressure and height) 43 surface parameters (3 static fields) Still some open topics (work on common WMO compliant GRIB2 definitions ; skin versus surface temperature; any additional products (e.g.probabilistic) planned?) Latest version of UERRA parameters available at https://software.ecmwf.int/wiki/display/UER/Parameters

19. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse Accessing future UERRA The same methods as for EURO4M can be used + UERRA will be part of ECMWF public datasets (like ERA40, TIGGE, S2S etc.): Open access to all users after accepting data policy Additional dedicated data web portal (for checking data availability & limited data retrievals) Simplified WebAPI service “Access ECMWF public datasets” for bigger data requests (not so powerful as “Access MARS”; e.g. MARS list option is not available)

20. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse Example of TIGGE data portal

21. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse Retrieving TIGGE data 3. write MARS retrieval request (mars.batch) and run it using WebAPI: mars mars.batch dataset = tigge, step = 24/to/120/by/24, number = all, levtype = sfc, date = 20071001/to/20071003, time = 00/12, origin = all, type = pf, param = tp, area = 70/-130/30/-60, grid = 2/2, target = tigge.grib Example of mars.batch for public dataset: Output: tigge.grib

22. 3-4. 2. 2016UERRA 2nd User Workshop - Toulouse Links UERRA at ECMWF: https://software.ecmwf.int/wiki/display/UERhttps://software.ecmwf.int/wiki/display/UER ECMWF Web API tutorial: https://software.ecmwf.int/wiki/display/WEBAPIhttps://software.ecmwf.int/wiki/display/WEBAPI ECMWF GRIB-API: https://software.ecmwf.int/wiki/display/GRIBhttps://software.ecmwf.int/wiki/display/GRIB MARS web catalogue: http://apps.ecmwf.int/mars-cataloguehttp://apps.ecmwf.int/mars-catalogue MARS documentation: https://software.ecmwf.int/wiki/display/UDOC/MARS+user+documentation https://software.ecmwf.int/wiki/display/UDOC/MARS+user+documentation