1. Data Storage System, Model Output and Analysis Tools PRECIS Workshop Tanzania Meteorological Agency 29 th June – 3 rd July 2015

2. © Crown copyright Met Office In Linux, variables are identified with a preceding $. So $x, $TEMP, $David, $R2D2, $EXETER could all be variable names in Linux. Variables can hold text or numbers, including file names and directories. $HOME is a predefined Linux variable which refers to the main working directory of the user’s account. In Linux, this is usually specified by /home/[username] – e.g. $HOME=/home/precis The text file setvars is created at installation. It contains a long list of variable declarations which tell PRECIS where input and output data lives. Find setvars in $HOME/setvars (e.g. /home/precis/setvars) Structure

3. © Crown copyright Met Office PRECIS write its output data to the directory set in setvars as $ARCHIVEDIR (/home/precis/archive by default) Each experiment is uniquely identified by a five character RUNID. Within $ARCHIVEDIR, there are subfolders according to the RUNID. Examples: /home/precis/archive/abcab/ /home/precis/archive/exete/ /data/archive/jedaa (if ARCHIVEDIR=/data/archive) Location of the data

4. © Crown copyright Met Office ARCHIVEDIR in setvars

5. © Crown copyright Met Office Inside $ARCHIVEDIR :

6. The STASH code Each variable type (temperature, precipitation, etc.) from the model is written to a separate directory defined by that variable’s 5-digit ‘STASH’ code $ARCHIVEDIR/ RUNID /03236/ 03236 = Mean temperature at 1.5m $ARCHIVEDIR/ RUNID /08208/ 08208 = Soil Moisture Content $ARCHIVEDIR/ RUNID /05216/ 05216 = Mean total precipitation For a full list, see Appendix C of the PRECIS technical manual (www.precisrcm.com/tech_man.pdf)www.precisrcm.com/tech_man.pdf

7. © Crown copyright Met Office Inside $ARCHIVEDIR/RUNID:

8. Files follow this general pattern: [RUNID]a.[file type][Date].[STASH code].[format] Example: abdaba.pji9140.03236.pp All file types are listed in Appendix D of the technical manual Output file naming convention

9. © Crown copyright Met Office Inside $ARCHIVEDIR/RUNID/03236

10. The PRECIS models output data in the PP binary data format. This is the Met Office’s own format. Conversion tools are provided to convert PP data to CF compliant NetCDF (standard) GRIB (for GrADs and Ferret) ArcInfo ascii (e.g. for GIS/spreadsheets) See Section 6.2 of the technical manual for further discussion. Output Data format

11. Analysis Utilities To use the output data in any way, you will need to make use of some kind of data analysis software utility. In general, you need to process your data and/or visualise it. Two tools are used during the PRECIS training workshop: CDO and NCL, details of which follow. Other analysis tools exist, such as the Met Office’s IRIS system.

12. Analysis Utilities (cont) The utilities GrADs, Ferret and CDAT are included on the PRECIS installation software, but no training in the use of these is given nor will any technical support in their use be provided!!! There is a suite of data analysis tools for PP data which are installed as part of PRECIS and are fully supported by the PRECIS team. You will use some of these tools during the workshop.

13. Details of output data conventions

14. Reduces number of characters used in setting date First digit indicates the decade:[0-9a-z] Actually number of decades since 1800. E.g.: 0  1800-1810 a  1900-1910 g  1960-1970 r  2070-2080 Second digit indicate year in decade[0-9] Third digit represents month:[1-9a-c] Fourth digit is the day of the month:[1-9a-v] Fifth digit is the hour of the day:[0-9a-o] The UM Date Stamp

15. r4c10 0:00z 1 st December 2074 h93ab 11:00z 10 th March 1979 j3jun June 1993 l56ud 13:00 30 th June 2015 Some examples

16. PRECIS outputs monthly and seasonal means by default You can also output daily mean and hourly mean output, but beware filling up the hard drive! GCMs use either a 360 day calendar (30 days per month) or the standard Gregorian calendar. PRECIS keeps the calendar of the driving GCM. With 360 day calendar, you have control over defining the seasons (e.g. If you want three four-month seasons per year, or two six month seasons, etc) Temporal Frequency of Output

17. Data analysis and visualisation utilities useful for PRECIS

18. CDO and NCL what are they? CDO and NCL are tools for data analysis and visualisation. Different tools are useful for different activities e.g. CDO tools for analysis and NCL for visualisation. In the worksheets these tools are used together to process, analyse and visualise PRECIS output.

19. CDO – Climate Data Operators Developed by the Max Planck Institute (Germany) for use with GRIB 1 / GRIB2, NetCDF3 / NetCDF4 and other data formats. Designed specifically for climate and NWP data analysis, there are more than 600 operators available. Can be run on Linux, Windows, MacOS and others.

20. CDO examples Calculate the JJA mean: cdo selmon,6/8 infile.nc jja.nc cdo timmean jja.nc jja.mean.nc or, pipe the operators together using ‘-’ cdo timmean –selmon,6/8 infile.nc jja.mean.nc Re-grid data onto a target grid: cdo griddes target_grid.nc > mygrid cdo remapbil.mygrid file.nc regridded_file.nc

21. Free to download and documentation and support forums can be found at https://code.zmaw.de/projects/cdohttps://code.zmaw.de/projects/cdo CDO – Climate Data Operators

22. NCL – NCAR Command Language (1) Developed by NCAR (National Center for Atmospheric Research, USA ) for use with GRIB 1/2, netCDF 3/4 and ascii data. Free interpreted language designed for scientific data processing and visualization. Can be run on Linux, Windows, MacOS and others. Can be typed directly into the NCL command line (very time consuming), or scripted. Produces plots of publishable quality.

23. NCL – NCAR Command Language (2) Extensive documentation and hundreds of example scripts and plots at http://www.ncl.ucar.edu/index.shtmlhttp://www.ncl.ucar.edu/index.shtml

24. NCL – NCAR Command Language (3)

25. © Crown copyright Met Office Xconv – for quick looks The xconv graphical utility (written by Jeff Cole at Reading University, UK) allows for quick visualisation of PP and NetCDF format files. xconv is limited in that it: Only creates box plots Does not display map outlines Allows no control over scale or colours used in plots Therefore, xconv is not a good tool for creating plots for reports and published works. It is best used as a tool to take a “quick look” at data.

26. © Crown copyright Met Office Xconv Main Xconv Window

27. © Crown copyright Met Office Xconv A sample plot of Soil Moisture Content with Xconv

28. © Crown copyright Met Office Panoply Panoply is a netCDF, HDF and GRIB data viewer written by NASA. Panoply is written in Java. It runs on Windows, MacOS or Linux. Panoply plots data from NetCDF, HDF and GRIB data sets. You can: Slice and plot specific latitude-longitude, latitude-vertical, or time- latitude arrays from larger multidimensional variables. Overlay continent outlines or masks on lon-lat plots Change the scale and colour table Save plots to disk GIF, JPEG, PNG or TIFF bitmap images or as PDF or PostScript graphics files.

29. © Crown copyright Met Office Panoply The main window

30. © Crown copyright Met Office A sample plot of precipitation using Panoply

31. Using tools together The worksheets combine these tools to post process and analysis PRECIS data. 1.Use PRECIS tools to do initial post processing 2.Convert the data to NetCDF format 3.Use CDO to analyse the NetCDF data. 4.Use NCL to visualise (and process) the NetCDF data.

32. Questions

33. Models with the Gregorian calendar (ERA) have only monthly, and 3-month seasonal available. The Base Date for climate meaning sets the reference date for the beginning of means. The ‘infinite time series’ of all climate means will coincide with this date. Example: If the base date is set to 01 Apr 1960, four month seasonal means will be apr/may/jun/jul, aug/sep/oct/nov, and dec/jan/feb/mar. Annual means will be represented by 01 Apr – 30 Mar. Take the initial spin-up year into consideration when choosing the base date. “Climate Mean” Output (reference)

34. For models with a 360 day calendar (such as those driving the HadAM3P GCM) up to four climate meaning periods may be set These allow for long timescale means to be calculated within the model (rather than by the user) Choice of means is determined by a comma separated list representing shortest to longest Meaning periods are nested, and each one is specified by how many multiples of the previous (shorter) period it is. “Climate Mean” Output (reference)

35. Example 1: 30,3,4,10 30 : 30 day (monthly) means 3 : 30 * 3 = 90 day (seasonal) means 4 : 90 * 4 = 360 day (annual) means 10 : 360 * 10 = 10 year (decadal) means

36. “Climate Mean” Output (reference) Example 2: 30,4,3,5 30: 30 day (monthly) means 4 : 30 * 4 = 120 day (4-month seasonal) means 3 : 120 * 3 = 360 day (annual) means 5 : 360 * 5 = 5 year (decadal) means