NetCDF Ed Hartnett Unidata/UCAR

Slides:



Advertisements
Similar presentations
Introduction to the BinX Library eDIKT project team Ted Wen Robert Carroll
Advertisements

DataTools Models Data, models and tools: Dealing with any complex hydraulic engineering problem invariable use is made of: data, models and tools.
1 Projection Indexes in HDF5 Rishi Rakesh Sinha The HDF Group.
Recent Work in Progress
A Unified Data Model and Programming Interface for Working with Scientific Data Doug Lindholm Laboratory for Atmospheric and Space Physics University of.
The NCAR Command Language (NCL) and the NetCDF Data Format Research Tools Presentation Matthew Janiga 10/30/2012.
A Common Data Model In the Middle Tier Enabling Data Access in Workflows … HDF/HDF-EOS Workshop XIV September 29, 2010 Doug Lindholm Laboratory for Atmospheric.
Streaming NetCDF John Caron July What does NetCDF do for you? Data Storage: machine-, OS-, compiler-independent Standard API (Application Programming.
The Future of NetCDF Russ Rew UCAR Unidata Program Center Acknowledgments: John Caron, Ed Hartnett, NASA’s Earth Science Technology Office, National Science.
NetCDF An Effective Way to Store and Retrieve Scientific Datasets Jianwei Li 02/11/2002.
Introduction to NetCDF Ernesto Munoz. Outline Overview of NetCDF Overview of NetCDF NetCDF file information NetCDF file information CDL utilities: ncdump,
NetCDF 3.6: What’s New Russ Rew Unidata Program Center University Corporation for Atmospheric Research
Introduction to NetCDF Russ Rew, UCAR Unidata ICTP Advanced School on High Performance and Grid Computing 13 April 2011.
Status of netCDF-3, netCDF-4, and CF Conventions Russ Rew Community Standards for Unstructured Grids Workshop, Boulder
Show of Hands... How many traveled to be here? University/Gov't/Industry How many use netCDF? Primary programming language for netCDF? Other data formats.
Developing a NetCDF-4 Interface to HDF5 Data
Introduction to NetCDF4 MuQun Yang The HDF Group 11/6/2007HDF and HDF-EOS Workshop XI, Landover, MD.
Netcdf course Intro CF convention Netcdf excercises.
NetCDF-4 The Marriage of Two Data Formats Ed Hartnett, Unidata June, 2004.
NetCDF and HDF5 Ed Hartnett, Unidata/UCAR, Unidata Mission: To provide the data services, tools, and cyberinfrastructure leadership that advance.
Developing a NetCDF-4 Interface to HDF5 Data Russ Rew (PI), UCAR Unidata Mike Folk (Co-PI), NCSA/UIUC Ed Hartnett, UCAR Unidata Quincey Kozial, NCSA/UIUC.
1 High level view of HDF5 Data structures and library HDF Summit Boeing Seattle September 19, 2006.
NetCDF for Developers and Data Providers Russ Rew, UCAR Unidata ICTP Advanced School on High Performance and Grid Computing 14 April 2011.
HDF5 A new file format & software for high performance scientific data management.
A Metadata Based Approach For Supporting Subsetting Queries Over Parallel HDF5 Datasets Vignesh Santhanagopalan Graduate Student Department Of CSE.
Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.
1 Overview of HDF5 HDF Summit Boeing Seattle The HDF Group (THG) September 19, 2006.
An Introduction to MINC John G. Sled. What is MINC? A medical image file format based on NetCDF A core set tools and libraries for image processing A.
NetCDF-4 and Parallel I/O GSFC, Nov 20,2008 Ed Hartnett.
NetCDF for High Performance Computing Introduction to NetCDF What is netCDF? NetCDF Data Models How we think of data. NetCDF Software Libraries Using.
Mid-Course Review: NetCDF in the Current Proposal Period Russ Rew
The future of MINC Robert D. Vincent
N P O E S S I N T E G R A T E D P R O G R A M O F F I C E NPP/ NPOESS Product Data Format Richard E. Ullman NOAA/NESDIS/IPO NASA/GSFC/NPP Algorithm Division.
Page 1 Status of HDF-EOS, Related Software, and Tools Abe Taaheri, Raytheon IIS HDF & HDF-EOS Workshp XIII Riverdale, MD November 4, 2009.
The netCDF-4 data model and format Russ Rew, UCAR Unidata NetCDF Workshop 25 October 2012.
SciDAC All Hands Meeting, March 2-3, 2005 Northwestern University PIs:Alok Choudhary, Wei-keng Liao Graduate Students:Avery Ching, Kenin Coloma, Jianwei.
Integrating netCDF and OPeNDAP (The DrNO Project) Dr. Dennis Heimbigner Unidata Go-ESSP Workshop Seattle, WA, Sept
Project 4 : SciDAC All Hands Meeting, September 11-13, 2002 A. Choudhary, W. LiaoW. Gropp, R. Ross, R. Thakur Northwestern UniversityArgonne National Lab.
Accessing Remote Datasets using the DAP protocol through the netCDF interface. Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009.
Advanced Utilities Extending ncgen to support the netCDF-4 Data Model Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009.
A High performance I/O Module: the HDF5 WRF I/O module Muqun Yang, Robert E. McGrath, Mike Folk National Center for Supercomputing Applications University.
NetCDF Data Model Issues Russ Rew, UCAR Unidata NetCDF 2010 Workshop
Page 1 TOOLKIT / HDF-EOS Status and Development Abe Taaheri, Raytheon IIS Aura DSWG meeting October 2007.
HDF Hierarchical Data Format Nancy Yeager Mike Folk NCSA University of Illinois at Urbana-Champaign, USA
Core Java Introduction Byju Veedu Ness Technologies httpdownload.oracle.com/javase/tutorial/getStarted/intro/definition.html.
The HDF Group Introduction to netCDF-4 Elena Pourmal The HDF Group 110/17/2015.
NetCDF-4: Software Implementing an Enhanced Data Model for the Geosciences Russ Rew, Ed Hartnett, and John Caron UCAR Unidata Program, Boulder
NetCDF and Scientific Data Durability Russ Rew, UCAR Unidata ESIP Federation Summer Meeting
Advances in the NetCDF Data Model, Format, and Software Russ Rew Coauthors: John Caron, Ed Hartnett, Dennis Heimbigner UCAR Unidata December 2010.
CF 2.0 Coming Soon? (Climate and Forecast Conventions for netCDF) Ethan Davis ESO Developing Standards - ESIP Summer Mtg 14 July 2015.
Developing Conventions for netCDF-4 Russ Rew, UCAR Unidata June 11, 2007 GO-ESSP.
Update on Unidata Technologies for Data Access Russ Rew
NcBrowse: A Graphical netCDF File Browser Donald Denbo NOAA-PMEL/UW-JISAO
Utilities for netCDF-4 Dr. Dennis Heimbigner Unidata Advanced netCDF Workshop July 25, 2011.
Unidata Infrastructure for Data Services Russ Rew GO-ESSP Workshop, LLNL
Libcf – A CF Convention Library for NetCDF Ed Hartnett Unidata Program Center Boulder Colorado June 11, 2007.
NetCDF Data Model Details Russ Rew, UCAR Unidata NetCDF 2009 Workshop
Other Projects Relevant (and Not So Relevant) to the SODA Ideal: NetCDF, HDF, OLE/COM/DCOM, OpenDoc, Zope Sheila Denn INLS April 16, 2001.
NetCDF-Java version 2.2 Common Data Model John Caron Unidata/UCAR Dec 10, 2004.
DAP+NETCDF Using the netCDF-4 Data Model
Moving from HDF4 to HDF5/netCDF-4
SRNWP Interoperability Workshop
NetCDF 3.6: What’s New Russ Rew
Plans for an Enhanced NetCDF-4 Interface to HDF5 Data
Unidata Advanced netCDF Workshop
Status for Endeavor 6: Improved Scientific Data Access Infrastructure
Libcf – A CF Convention Library for NetCDF
HDF-EOS Workshop XXI / The 2018 ESIP Summer Meeting
NCL variable based on a netCDF variable model
OPeNDAP/Hyrax Interfaces
Presentation transcript:

NetCDF Ed Hartnett Unidata/UCAR

Unidata Unidata - helps universities acquire, display, and analyze Earth-system data. UCAR – University Corporation for Atmospheric Research - a nonprofit consortium of 66 universities.

SDSC Presentation, July 2005 Intro to NetCDF Classic Intro to NetCDF-4

What is NetCDF? A conceptual data model for scientific data. A set of APIs in C, F77, F90, Java, etc. to create and manipulate data files. Some portable binary formats. Useful for storing arrays of data and accompanying metadata.

History of NetCDF netCDF 2.0 released netCDF developed at Unidata netCDF 3.0 released netCDF released netCDF 4.0 beta released

Getting netCDF Download latest release from the netCDF web page: Builds and installs on most platforms with no configuration necessary. For a list platforms netCDF versions have built on, and the output of building and testing netCDF, see the web site.

NetCDF Portability NetCDF is tested on a wide variety of platforms, including Linux, AIX, SunOS, MacOS, IRIX, OSF1, Cygwin, and Windows. We test with native compilers when we can get them. 64-bit builds are supported with some configuration effort.

What Comes with NetCDF NetCDF comes with 4 language APIs: C, C++, Fortran 77, and Fortran 90. Tools ncgen and ncdump. Tests. Documentation.

NetCDF Java API The netCDF Java API is entirely separate from the C API. You don’t need to install the C API for the Java API to work. Java API contains many exciting features, such as remote access and more advanced coordinate systems.

Tools to work with NetCDF Data The netCDF core library provides basic data access. ncgen and ncdump provide some helpful command line functionality. Many additional tools are available, see:

CDL – Common Data Language Grammar defined for displaying information about netCDF files. Can be used to create files without programming. Can be used to create reading program in Fortran or C. Used by ncgen/ncdump utilities.

Example of CDL netcdf foo { // example netCDF specification in CDL dimensions: lat = 10, lon = 5, time = unlimited; variables: int lat(lat), lon(lon), time(time); float z(time,lat,lon), t(time,lat,lon); double p(time,lat,lon); int rh(time,lat,lon); lat:units = "degrees_north"; lon:units = "degrees_east"; data: lat = 0, 10, 20, 30, 40, 50, 60, 70, 80, 90; lon = -140, -118, -96, -84, -52; }

Software Architecture of NetCDF-3 V3 C API V2 C API V2 C tests V3 C testsF77 API F77 testsF90 API C++ APIncdumpncgen Fortran, C++ and V2 APIs are all built on the C API. Other language APIs (perl, python, MatLab, etc.) use the C API.

NetCDF Documentation Unidata distributes a NetCDF Users Guide which describes the data model in detail. A language-specific guide is provided for C, C++, Fortran 77, and Fortran 90 users. All documentation can be found at:

NetCDF Jargon “Variable” – a multi-dimensional array of data, of any of 6 types (char, byte, short, int, float, or double). “Dimension” – information about an axis: it’s name and length. “Attribute” – a 1D array of metadata.

More NetCDF Jargon “Coordinate Variable” – a 1D variable with the same name as a dimension, which stores values for each dimension value. “Unlimited Dimension” – a dimension which has no maximum size. Data can always be extended along the unlimited dimension.

The NetCDF Classic Data Model The netCDF Classic Data Model contains dimensions, variables, and attributes. At most one dimension may be unlimited. The Classic Data Model is embodied by netCDF versions 1 through NetCDF is moving towards a new, richer data model: the Common Data Model.

NetCDF Example Suppose a user wants to store temperature and pressure values on a 2D latitude/longitude grid. In addition to the data, the user wants to store information about the lat/lon grid. The user may have additional data to store, for example the units of the data values.

NetCDF Model Example temperature pressure Units: C Units: mb VariablesDimensions latitude longitude Attributes latitude longitude Coordinate Variables

Important NetCDF Functions nc_create and nc_open to create and open files. nc_enddef, nc_close. nc_def_dim, nc_def_var, nc_put_att_*, to define dimensions, variables, and attributes. nc_inq, nc_inq_var, nc_inq_dim, nc_get_att_* to learn about dims, vars, and atts. nc_put_vara_*, nc_get_vara_* to write and read data.

C Functions to Define Metadata /* Create the file. */ if ((retval = nc_create(FILE_NAME, NC_CLOBBER, &ncid))) return retval; /* Define the dimensions. */ if ((retval = nc_def_dim(ncid, LAT_NAME, LAT_LEN, &lat_dimid))) return retval; if ((retval = nc_def_dim(ncid, LON_NAME, LON_LEN, &lon_dimid))) return retval; /* Define the variables. */ dimids[0] = lat_dimid; dimids[1] = lon_dimid; if ((retval = nc_def_var(ncid, PRES_NAME, NC_FLOAT, NDIMS, dimids, &pres_varid))) return retval if ((retval = nc_def_var(ncid, TEMP_NAME, NC_FLOAT, NDIMS, dimids, &temp_varid))) return retval; /* End define mode. */ if ((retval = nc_enddef(ncid))) return retval;

C Functions to Write Data /* Write the data. */ if ((retval = nc_put_var_float(ncid, pres_varid, pres_out))) return retval; if ((retval = nc_put_var_float(ncid, temp_varid, temp_out))) return retval; /* Close the file. */ if ((retval = nc_close(ncid))) return retval;

C Example – Getting Data /* Open the file. */ if ((retval = nc_open(FILE_NAME, 0, &ncid))) return retval; /* Read the data. */ if ((retval = nc_get_var_float(ncid, 0, pres_in))) return retval; if ((retval = nc_get_var_float(ncid, 1, temp_in))) return retval; /* Do something useful with the data… */ /* Close the file. */ if ((retval = nc_close(ncid))) return retval;

Data Reading and Writing Functions There are 5 ways to read/write data of each type. var1 – reads/writes a single value. var – reads/writes entire variable at once. vara – reads/writes an array subset. vars – reads/writes an array by slices. varm – reads/writes a mapped array. Ex.: nc_put_vars_short writes shorts by slices.

Attributes Attributes are 1-D arrays of any of the 6 netCDF types. Read/write them with functions like: nc_get_att_float and nc_put_att_int. Attributes may be attached to a variable, or may be global to the file.

NetCDF File Formats Starting with 3.6.0, netCDF supports two binary data formats. NetCDF Classic Format is the format that has been in use for netCDF files from the beginning. NetCDF 64-bit Offset Format was introduced in and allows much larger files. Use classic format unless you need the large files.

NetCDF-3 Summary NetCDF is a software library and some binary data formats, useful for scientific data, developed at Unidata. NetCDF organizes data into variables, with dimensions and attributes. NetCDF has proven to be reliable, simple to use, and very popular.

Why Add to NetCDF-3? Increasingly complex data sets call for greater organization. Size limits, unthinkably huge in 1988, are routinely reached in Parallel I/O is required for advanced Earth science applications. Interoperability with HDF5.

NetCDF-4 NetCDF-4 aims to provide the netCDF API as a front end for HDF5. Funded by NASA, executed at Unidata and NCSA. Includes reliable netCDF-3 code, and is fully backward compatible.

NetCDF-4 Organizations Unidata/UCARUnidata NCSA – The National Center for Supercomputing Applications University of Illinois at Urbana-ChampaignThe National Center for Supercomputing Applications University of Illinois at Urbana-Champaign NASA – NetCDF-4 was funded by NASA award number AIST

New Features of NetCDF-4 Multiple unlimited dimensions. Groups to organize data. New types, including compound types and variable length arrays. Parallel I/O.

The Common Data Model NetCDF-4, scheduled for beta-release this Summer, will conform to the Common Data Model. Developed by John Caron at Unidata, with the cooperation of HDF, OpenDAP, netCDF, and other software teams, CDM unites different models into a common framework. CDM is a superset of the NetCDF Classic Data Model

The NetCDF-4 Data Model NetCDF-4 implements the Common Data Model. Adds groups, each group can contain variables, attributes and dimensions, and groups. Dimensions are scoped so that variables in different groups can share dimensions. Compound types allow users to define new types, comprised of other atomic or user-defined types. New integer and string types.

Software Architecture of NetCDF-4 V4 C API V2 C API V2 C tests V3 C testsF77 API F77 testsF90 API C++ APIncdumpncgen V3 C APIHDF5

NetCDF-4 Release Status Latest alpha release includes all netCDF-4 features – depends on latest HDF5 development snapshot. Beta release – due out in August, replaces artificial netCDF-4 constructs, and depends on a yet-to-be-released version of HDF5. Promotion from beta to full release will happen sometime in 2006.

Building NetCDF-4 NetCDF-4 requires that HDF5 version be installed. This is not released yet. The latest HDF5 development release works with the latest netCDF alpha release. To build netCDF-4, specify –enable- netcdf-4 at configure.

When to Use NetCDF-4 Format The new netCDF-4 features (groups, new types, parallel I/O) are only available for netCDF-4 format files. When you need HDF5 files. When portability is less important, until netCDF-4 becomes widespread.

Versions and Formats Classic Format 64-Bit Offset Format NetCDF-4 Format netCDF 2.0 released netCDF developed by Glenn Davis netCDF 3.0 released netCDF released netCDF 4.0 beta released

NetCDF-4 Feature Review Multiple unlimited dimensions. How to use groups. Using compound types. Other new types. Variable length arrays. Parallel I/O. HDF5 Interoperability.

Multiple Unlimited Dimensions Unlimited dimensions are automatically expanded as new data are written. NetCDF-4 allows multiple unlimited dimensions.

Working with Groups Define a group, then use it as a container for the classic data model. Groups can be used to organize sets of data.

Model_Run_1a rhlat lontemp units history Model_Run_1 rhlat lontemp units history Model_Run_2 rhlat lontemp units history An Example of Groups

New Functions to Use Groups Open/create returns ncid of root group. Create a new group with nc_def_grp. nc_def_grp(int parent_ncid, char *name, int *new_ncid); Learn about groups with nc_inq_grps. nc_inq_grps(int ncid, int *numgrps, int *ncids);

C Example Using Groups if (nc_create(FILE_NAME, NC_NETCDF4, &ncid)) ERR; if (nc_def_grp(ncid, DYNASTY, &tudor_id)) ERR; if (nc_def_dim(tudor_id, DIM1_NAME, NC_UNLIMITED, &dimid)) ERR; if (nc_def_grp(tudor_id, HENRY_VII, &henry_vii_id)) ERR; if (nc_def_var(henry_vii_id, VAR1_NAME, NC_INT, 1, &dimid, &varid)) ERR; if (nc_put_vara_int(henry_vii_id, varid, start, count, data_out)) ERR; if (nc_close(ncid)) ERR;

Create Complex Types Like C structs, compound types can be assembled into a user defined type. Compound types can be nested – that is, they can contain other compound types. New functions are needed to create new types. V2 API functions are used to read/write complex types.

C Example of Compound Types /* Create a file with a compound type. Write a little data. */ if (nc_create(FILE_NAME, NC_NETCDF4, &ncid)) ERR; if (nc_def_compound(ncid, sizeof(struct s1), SVC_REC, &typeid)) ERR; if (nc_insert_compound(ncid, typeid, BATTLES_WITH_KLINGONS, HOFFSET(struct s1, i1), NC_INT)) ERR; if (nc_insert_compound(ncid, typeid, DATES_WITH_ALIENS, HOFFSET(struct s1, i2), NC_INT)) ERR; if (nc_def_dim(ncid, STARDATE, DIM_LEN, &dimid)) ERR; if (nc_def_var(ncid, SERVICE_RECORD, typeid, 1, dimids, &varid)) ERR; if (nc_put_var(ncid, varid, data)) ERR; if (nc_close(ncid)) ERR;

New Ints, Opaque, String Types Opaque types are bit-blobs of fixed size. String types allow multi-dimensional arrays of strings. New integer types: UBYTE, USHORT, UINT, UINT64, INT64.

Variable Length Arrays Variable length arrays allow the efficient storage of arrays of variable size. For example: an array of soundings of different number of elements.

Parallel I/O with NetCDF-4 Must use configure option –enable-parallel when building netCDF. Depends on HDF5 parallel features, which require MPI. Must create or open file with nc_create_par or nc_open_par. All metadata operations are collective. Adding a new record is collective. Variable reads/writes are independent by default, but can be changed to do collective operations.

HDF5 Interoperability NetCDF-4 can interoperate with HDF5 with a SUBSET of HDF5 features. Will not work with HDF5 files that have looping groups, references, and types not found in netCDF-4. HDF5 file must use new dimension scale API to store shared dimension info. If a HDF5 follows the Common Data Model, NetCDF-4 can interoperate on the same files.

Future Plans for NetCDF NetCDF 4.0 release in Beta for next major version of netCDF in Summer, Full compatibility with Common Data Model. Remote access, including remote subsetting of data. XML-based representation of netCDF metadata. Full Fortran 90 support, but limited F77 support.

For Further Information netCDF mailing list: Ed: netCDF web site: