netCDF An Effective Way to Store and Retrieve Scientific Datasets Jianwei Li 02/11/2002

Motivation Related scientific datasets in a single file. –Define a file format –Present the multi-dimensional arrays in a self- describing way Common data access through a simple interface –Typical ways of access to multi-dimensional arrays –Platform-independent / Network-transparent Here comes netCDF!

Outline Today Overview of netCDF netCDF File Format netCDF library and Programming Model Advantages & Disadvantages Future Plans for NetCDF

What Is netCDF? Self-describing. Portable. Direct-access. Appendable. Sharable. NetCDF (network Common Data Form) is an interface for array-oriented data access. It defines a machine-independent file format for representing multi-dimensional arrays with ancillary data, and provide support for creation, access, and sharing of array-oriented data. The netCDF software is implemented as I/O libraries for C, FORTRAN, C++, and Perl and other language for which a netCDF library is available.

Why use netCDF? Facilitate the use of common datasets by distinct applications. Permit datasets to be transported between or shared by dissimilar computers transparently, i.e., without translation. Reduce the programming effort usually spent interpreting formats. Reduce errors arising from misinterpreting data and ancillary data. Facilitate using output from one application as input to another. Establish an interface standard which simplifies the inclusion of new software into existing system.

Platforms that netCDF runs on? AIX HPUX-9.05 IRIX, IRIX64 MSDOS (gcc, f2c, GNU make) Linux OSF OpenVMS OS/2 SUNOS ULTRIX UNICOS Windows NT Platforms that range from CRAYs to personal computers and include most UNIX-based workstations.

netCDF dataset Components Dimensions name, length –Fixed dimension –UNLIMITED dimension Variables: named arrays name, type, shape, attributes –Fixed sized variables: array of fixed dimensions –Record variables: array with its most-significant dimension UNLIMITED –Coordinate variables: 1-D array with the same name as its dimension Attributes name, type, values, length –Variable attributes –Global attributes

An Example netCDF File

netCDF File Format Header –Number of record variables –Dimension list –Global attribute list –Variable list Data (row-major) –Fixed-sized data data for each variable is stored contiguously –Record data a variable number of fixed-size records, each of which contains one record for each of the record variables in order. Both use extended XDR

Data access in netCDF dataset access to all elements (whole array); access to individual elements, specified with an index vector; access to array sections, specified with an index vector, and count vector; access to subsampled array sections, specified with an index vector, count vector, and stride vector; access to mapped array sections, specified with an index vector, count vector, stride vector, and an index mapping vector (map the indices of an element in an external subarray to the offset in internal buffer). Direct Access! Efficient for small subsets out of large datasets Forms of access:

Dataset Access Example Start[] = {2, 1}, Count[] = {3, 2}, Stride[] = {3, 2}, Imap[] = {1, 3}

netCDF Library & Programming Model Modes: definition mode, data mode IDs: dataset ID, dimension ID, variable ID, attribute number Create & WriteRead by name Read sequencially Add dim,var,att

Example library functions

netCDF data types and type conversion Different library functions for each data type Automatic conversion to or from external type Conversion from one numeric type to another netCDF Data TypeC API MnemonicDiscription byteNC_BYTE 8-bit characters for text data. charNC_CHAR 8-bit signed or unsigned integers short NC_SHORT16-bit signed integers int NC_INT32-bit signed integers float NC_FLOAT32-bit IEEE floating-point double NC_DOUBLE64-bit IEEE floating-point

Other Data Types? Data Structure –No direct support –Can build some data structure by grouping arrays or by using data in one array as pointers into another array. Inefficient, and may not be self-describing! Character String –not a primitive netCDF external data type –Charater-string attributes: well represented by NC_CHAR values and its length. –Charater-string variables: use a character-position dimension as the most quickly varying dimension for the variable (dim-length = the max string length)

Share Data Between Read and Write Support for multiple read and single write Use NC_SHARE mode to create or open dataset to use unbuffered I/O Or call nc_sync() to force writing changes back to disk Header changes in define mode are synchronized to disk only when nc_enddef() is called, reader must call nc_sync() to bring header up-to-date in its memory

Advantages Simple and useful libraries for array- oriented data access The dataset is self-describing Portable and easy to share Flexible and efficient data access Mechanism to support extendable arrays

Disadvantages No support for multiple concurrent write Limited number of external numeric data types: 8-, 16-, 32-bit integers, or 32- or 64-bit floating-point numbers. No support for nested data structures such as trees, nested arrays, or other recursive structures Only one unlimited dimension. Multiple variables sharing an unlimited dimension, must all grow together. Redefine(structure changes) that requires more file space is accomplished by copying the dataset.

Future plans for netCDF Parallel write support Support for bit data type and transparent data packing for bit variables Support for efficient structure changes Support for nested arrays and other data structure