1. www.hdfgroup.org The HDF Group Introduction to netCDF-4 Elena Pourmal The HDF Group 110/17/2015

2. www.hdfgroup.org Overview 10/17/20152 What is netCDF? Data model and formats Ecosystem

3. www.hdfgroup.org WHAT IS NETCDF? 310/17/2015

4. www.hdfgroup.org Background 10/17/20154 Developed and maintained by Unidata http://www.unidata.ucar.edu/ http://www.unidata.ucar.edu/ Mission: provide data services, tools, and community leadership to advance Earth system science, enhance educational opportunities, and broaden participation Funded by NSF through UCAR Open source software

5. www.hdfgroup.org What is netCDF? 10/17/20155 netCDF = network Common Data Form Data model Scientific data (especially suitable for gridded data) Widely used in ocean and atmospheric science, climate and weather modeling Other disciplines: molecular dynamics, fusion research, medical imaging File FORMAT Portable, self-describing, etc. Not just one file format (netcdf, HDF4, HDF5, OPeNDAP, etc.) Application programming interfaces (APIs) C, Java, C++, Fortran Python, Ruby, Perl, MATLAB, IDL

6. www.hdfgroup.org History of netCDF 10/17/20156 1988 – first release of netCDF 1991 – netCDF 2.0 1994 – HDFv 3.3 reads netCDF files 2004 – netCDF 3.6 supports 64-bit offsets 2008 – netCDF-4 based on HDF5 2010 and later – parallel, OPeNDAP, HDF4

7. www.hdfgroup.org NETCDF DATA MODEL AND FORMATS 710/17/2015

8. www.hdfgroup.org netCDF classic data model 10/17/20158 Variables Name, shape, type N-dim arrays Dimension Name, length Attributes Name, type, value

9. www.hdfgroup.org netCDF classic data model Data model elements: Variables Name, shape, type N-dim arrays Dimensions Name, length Can be shared Attributes Name, type, value Model limitations: Atomic types only One extendible dimension Flat structure Model advantages: Widely adopted Best practices and conventions 10/17/20159

10. www.hdfgroup.org netCDF classic format Pros: Simple (header with metadata, raw data) Suitable for parallel access (pnetcdf from Argonne) Aside: could easily support SWMR Cons: No support for compression No extensibility in multiple dimension XDR-based (big-endian) Costly to add more variables 10/17/201510

11. www.hdfgroup.org netCDF enhanced data model Data model elements: Variables Dimensions Attributes Groups Model limitations: Complex Harder to use and not widely adopted Emerging best practices and conventions Model advantages: Rich collection of datatypes Hierarchical data organization 10/17/201511

12. www.hdfgroup.org Example of ncdump output netcdf OMI-Aura_L2-example { dimensions: PRESSURE = 18 ; DATETIME = 2 ; independent_465 = 465 ; independent_22 = 22 ; independent_7 = 7 ; independent_2 = 2 ; variables: float PRESSURE(PRESSURE) ; PRESSURE:MissingValue = -999.f ; PRESSURE:standard_name = "air_pressure" ; PRESSURE:VAR_NAME = "PRESSURE" ; PRESSURE:VAR_DESCRIPTION = "Pressure at retrieval layer" ; PRESSURE:units = "hPa" ; PRESSURE:valid_range = -Infinityf, Infinityf ; PRESSURE:VAR_DEPEND = "PRESSURE" ; short APrioriCovarianceMatrix(DATETIME, independent_465) ; APrioriCovarianceMatrix:MissingValue = -32767s ; APrioriCovarianceMatrix:UniqueFieldDefinition = "OMI-Specific" ; APrioriCovarianceMatrix:Offset = 0. ; 10/17/201512

13. www.hdfgroup.org Example of h5dump output /PRESSURE Dataset {18/18} Attribute: CLASS scalar Type: 16-byte null-terminated ASCII string Data: "DIMENSION_SCALE" Attribute: MissingValue {1} Type: native float Data: -999 Attribute: NAME scalar Type: 9-byte null-terminated ASCII string Data: "PRESSURE" Attribute: REFERENCE_LIST {9} Type: struct { "dataset" +0 object reference "dimension" +8 native int } 16 bytes Data: (0) {DATASET-1:14192, 1}, {DATASET-1:14192, 2}, {DATASET-1:30064, 1}, {DATASET-1:32126, 1}, {DATASET-1:32472, 1}, {DATASET-1:34237, 1}, (6) {DATASET-1:49807, 1}, {DATASET-1:55542, 1}, {DATASET-1:62105, 1} 10/17/201513

14. www.hdfgroup.org netCDF-4 - netCDF extended format and library netCDF-4 uses HDF5 as a storage layer Pros: Compression Chunking storage Parallel I/O Extensibility (more objects can be added easily) No limitation on number objects Cons: Complex as HDF5 Clunky support for dimensions Not easy to tune for performance since some HDF5 parameters are hidden 10/17/201514

15. www.hdfgroup.org netCDF-4 Architecture 10/17/201515 HDF5 Library netCDF-4Library netCDF-3 Interface netCDF-3 applications netCDF-3 applications netCDF-4 applications netCDF-4 applications HDF5 applications HDF5 applications netCDF files netCDF files netCDF-4 HDF5 files HDF5 files

16. www.hdfgroup.org NETCDF ECOSYSTEM Real Strength 1610/17/2015

17. www.hdfgroup.org Ecosystem 10/17/201517

18. www.hdfgroup.org Power of CF conventions 10/17/201518 NPP data visualized with IDV Without CF metadata With CF metadata and dimensions

19. www.hdfgroup.org10/17/2015 19

20. www.hdfgroup.org Questions 10/17/201520 ? Thank you!