TIGGE, GRIB to NetCDF converter Doug Schuster (NCAR/ECMWF)

Topic Outline Introduction Introduction Ensemble NetCDF File Structure Ensemble NetCDF File Structure Converter Highlights Converter Highlights Challenges Challenges Future Plans Future Plans

GRIB to NetCDF Conversion t, EGRR, 1 t (1,2,3,4) d (1,2,3,4) Metadata t, ECMF, 2 t, EGRR, 2 t, ECMF, 1 d, EGRR, 1 d, EGRR, 2 d, ECMF, 1 d, ECMF, 2 Gather metadata and message locations Create NetCDF file structure Populate NetCDF parameter arrays (1,2,3,4) represents ensemble member id (Realization) GRIB FileNetCDF File

Ensemble NetCDF File Structure NetCDF File format NetCDF File format –Based on available CF conventions –File organization built according to Doblas-Reyes (ENSEMBLES project) proposed NetCDF file structure –Provides grid/ensemble specific metadata for each member Data Provider Data Provider Forecast type (perturbed, control, deterministic) Forecast type (perturbed, control, deterministic) –Allows for multiple combinations of initialization times and forecast periods within one file. Pairs of initialization and forecast step Pairs of initialization and forecast step

Ensemble NetCDF File Structure NetCDF Parameter structure (5 dimensions): NetCDF Parameter structure (5 dimensions): –Reftime –Realization (Ensemble member id) –Level –Latitude –Longitude “Coordinate” variables are use to describe: “Coordinate” variables are use to describe: –Realization Provides metadata associated with each ensemble grid. Provides metadata associated with each ensemble grid. –Reftime Allows for multiple initialization times and forecast periods to be contained within one file Allows for multiple initialization times and forecast periods to be contained within one file

Ensemble NetCDF File Structure Coordinate variables for Realization Coordinate variables for Realization –Provide ensemble member metadata Institution Institution Forecast Type Forecast Type Realization 1234 Param(Realization) Institution(Realization)ECMFECMFEGRREGRR ForecastType(Realization)cfpfcfpf Grid 1Grid 2Grid 3Grid 4 -At fixed Reftime and Level

Ensemble NetCDF File Structure Coordinate variable for Reftime Coordinate variable for Reftime –Provide Forecast Times for each Initialization Leadtime Leadtime Reftime ,12z ,00z ,00z Param(Reftime) Leadtime(Reftime) Grid 1Grid 2Grid 3 -At fixed Realization and Level

Ensemble NetCDF File Structure CDL Examples (reftime and coordinate variable) CDL Examples (reftime and coordinate variable) int reftime(reftime) ; reftime:data_type = "long" ; reftime:data_type = "long" ; reftime:units = "hours since :00:00" ; reftime:units = "hours since :00:00" ; reftime:standard_name = "forecast_reference_time" ; reftime:standard_name = "forecast_reference_time" ; reftime:long_name = "Time of model initialization" ; reftime:long_name = "Time of model initialization" ; int leadtime(reftime) ; int leadtime(reftime) ; leadtime:data_type = "int" ; leadtime:data_type = "int" ; leadtime:units = "hours" ; leadtime:units = "hours" ; leadtime:standard_name = "forecast_period" ; leadtime:standard_name = "forecast_period" ; leadtime:long_name = "hours since forecast_reference_time" ; leadtime:long_name = "hours since forecast_reference_time" ;

Ensemble NetCDF File Structure CDL Examples (realization and coordinate variables) CDL Examples (realization and coordinate variables) int realization(realization) ; realization:data_type="int" realization:data_type="int" realization:units = "1" ; realization:units = "1" ; realization:standard_name = "realization" ; realization:standard_name = "realization" ; realization:long_name = "Number of the simulation in the realization:long_name = "Number of the simulation in the ensemble" ; ensemble" ; char forecast_type(realization, string2) ; forecast_type:data_type = "char" ; forecast_type:data_type = "char" ; forecast_type:standard_name = "forecast_type" ; forecast_type:standard_name = "forecast_type" ; forecast_type:long_name = ”Forecast type" ; forecast_type:long_name = ”Forecast type" ; char institution(realization, string4) ; institution:data_type = "char" ; institution:data_type = "char" ; institution:standard_name = "institution" ; institution:standard_name = "institution" ; institution:long_name = "Institution responsible for the institution:long_name = "Institution responsible for the forecast system" ; forecast system" ;

Ensemble NetCDF File Structure CDL Examples (Ensemble parameter example) CDL Examples (Ensemble parameter example) short t(reftime, realization, level, latitude, longitude) ; t:data_type = "short" ; t:data_type = "short" ; t:units = "K" ; t:units = "K" ; t:standard_name = "air_temperature" ; t:standard_name = "air_temperature" ; t:coordinates = "leadtime institution forecast_type" ; t:coordinates = "leadtime institution forecast_type" ; t:scale_factor = ; t:scale_factor = ; t:add_offset = ; t:add_offset = ; t:_FillValue = s ; t:_FillValue = s ;

Conversion Tool Highlights Tool User options Tool User options –Pack to 16 bit shorts. –Select grids based on forecast type Perturbed Perturbed Control Control Deterministic Deterministic Control and Deterministic Control and Deterministic All 3 forecast types All 3 forecast types Large file support with NetCDF or later. Large file support with NetCDF or later.

Conversion Tool Highlights Tool Limitations Tool Limitations –First version –TIGGE data –Regular Lat/Lon grids –Requires Multiple Outside Libraries NetCDF NetCDF GRIB API GRIB API Jasper (JPEG 2000) Jasper (JPEG 2000) –Hard Coded Cell Methods –Slow

Conversion Tool Highlights Tool Performance Tool Performance –GRIB-2 Simple Packing to NetCDF 32 BIT GRIB-2 size x ~2 GRIB-2 size x ~2 –GRIB-2 Simple Packing to NetCDF 16 BIT Similar size Similar size –GRIB-2 JPEG 2000 to NetCDF 32 BIT GRIB-2 size x ~8 GRIB-2 size x ~8 –GRIB-2 JPEG 2000 to NetCDF 16 BIT GRIB-2 size x ~4 GRIB-2 size x ~4

Conversion Tool Highlights ECMWF sl params, 2x2 Degree Global Grid, simple packing, converted on 3.4 GHz processor Linux Box

Challenges Parameter names Parameter names –Map available GRIB-2 names to CF names. Product Discipline, Parameter Category, Parameter Number Product Discipline, Parameter Category, Parameter Number –Map dynamically generated names and GRIB-2 encoding to CF parameter names Abstract Definitions Abstract Definitions Not a simple Discipline/Category/Number combination. Not a simple Discipline/Category/Number combination. –Require statistical processing and level information. Will generally be CF name + Cell method Will generally be CF name + Cell method –Get all centers to use this style of parameter encoding. NCEP only uses the “Discipline/Category/Number” parameter encoding scheme. NCEP only uses the “Discipline/Category/Number” parameter encoding scheme. –GRIB API doesn’t recognize hard coded parameters that include statistical processing (e.g. maximum temperature).

Challenges User Tools User Tools –Compatibility with NetCDF ensemble file structure. Deal with the ensemble dimension. Deal with the ensemble dimension. Properly ingest, compute valid times. Properly ingest, compute valid times. Read Coordinate variables to provide ensemble member metadata. Read Coordinate variables to provide ensemble member metadata. NetCDF or later required for large file support. NetCDF or later required for large file support.

Future Plans Implement on TIGGE portals to provide option for NetCDF format data download. Implement on TIGGE portals to provide option for NetCDF format data download. Generalize converter tool to encompass all types of GRIB data. Generalize converter tool to encompass all types of GRIB data. Add GRIB API “Definitions” files for CF parameter names, and cell methods. Add GRIB API “Definitions” files for CF parameter names, and cell methods. Support Additional GRID types Support Additional GRID types Explore NetCDF 4 Explore NetCDF 4 –API with greater functionality and flexibility. Add user options. Add user options.