NCAS Computational Modelling Service (CMS) Group providing services to the UK academic modelling community Output of UM Diagnostics Directly in CF NetCDF; an end to costly data conversions Jeff Cole, Grenville Lister, NCAS-CMS, Dept. of Meteorology, University of Reading

NCAS Computational Modelling Service (CMS) Group providing services to the UK academic modelling community Typical workflow Run the model to create 64-bit fields-file output Convert fields files to 32-bit pp (delete fields files) Convert pp data to CF NetCDF (all of it, ad hoc, delete pp files?) Analyze, visualize, process NetCDF data Problems? Low volumes of data - probably not Large volumes of data – greater potential for inefficiency

NCAS Computational Modelling Service (CMS) Group providing services to the UK academic modelling community Cascade 80TB of pp data to NetCDF conversion rate to NetCDF 10hr/TB on the HECToR NERC post processor 800hrs converting pp-NetCDF (doesn’t include fields-file – pp conversion) Model data production 10 wall clock hrs/TB Leeds experience Faster conversions - 4hr/TB “As far as I'm concerned the main issue is the amount of extra disk space the conversion takes up. It essentially doubles what is needed because I am never 100% sure the conversion has worked properly so I never delete the original pp files.”

NCAS Computational Modelling Service (CMS) Group providing services to the UK academic modelling community Objectives Directly write UM STASH output in NetCDF format Implement the whole UMUI STASH panel Convert PP file metadata into CF metadata Support standard and mean PP files Support NetCDF versions 3 and 4, including data compression Support multiple UM versions

NCAS Computational Modelling Service (CMS) Group providing services to the UK academic modelling community Coding decisions All NetCDF library calls contained in one f90 module Minimize changes to existing UM code All model metadata is written when NetCDF file is first opened Use STASH domain and time profile names as vertical and time coordinate names Standard names and units are taken from file STASH_to_CF.txt, produced by David Hassell

NCAS Computational Modelling Service (CMS) Group providing services to the UK academic modelling community Writing metadata Subroutine ncfile_init (called from initial) Loop1: loop over Fortran units if PP file is to be output on this unit Open NetCDF file Write global attributes Loop2: loop over all possible STASH codes if this STASH code is to be written to a PP file Write horizontal dimension information (ncfile_write_horiz_dim) Write vertical dimension information (ncfile_write_vert_dim) Write time dimension information (ncfile_write_time_dim) Create NetCDF variable Save NetCDF variable id, to be used when writing data Write NetCDF variable attributes

NCAS Computational Modelling Service (CMS) Group providing services to the UK academic modelling community Writing metadata Loop3: loop over Fortran units if PP file is to be output on this unit Put NetCDF file into data mode Write horizontal dimension data (ncfile_write_horiz_var) Write vertical dimension data (ncfile_write_vert_var) Write time dimension data (ncfile_write_time_var)

NCAS Computational Modelling Service (CMS) Group providing services to the UK academic modelling community Atmosphere initial ncfile_init stwork ncfile_write_data netcdf_mod

NCAS Computational Modelling Service (CMS) Group providing services to the UK academic modelling community NetCDF timings HadGEM3A, vn8.2, N96L85, 3 day run, 118 diagnostics, 9 PP files

NCAS Computational Modelling Service (CMS) Group providing services to the UK academic modelling community

To Do Finish implementing CF attributes (bounds, cell_methods) More work needed on time dimension Complete STASH panel options (meaning, grid subsection, timeseries) Add support for mean PP files Put in code so PP output works when files are re-initialised and for CRUNs Add support for writing NetCDF data in I/O Server code Add UMUI panel to select NetCDF options and code to read NetCDF namelist variables Add support for NetCDF4 data compression Port code to other UM versions, including adding support for the old ocean model Comprehensive testing