Libcf – A CF Convention Library for NetCDF

Slides:



Advertisements
Similar presentations
1 NASA CEOP Status & Demo CEOS WGISS-25 Sanya, China February 27, 2008 Yonsook Enloe.
Advertisements

Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1.
10 th Argo data management 2009 Toulouse Argo format and CF compatibility OceanOBS09 Strengthen and enhance the international framework under GCOS, GOOS,
Preparing CMOR for CMIP6 and other WCRP Projects
The Future of NetCDF Russ Rew UCAR Unidata Program Center Acknowledgments: John Caron, Ed Hartnett, NASA’s Earth Science Technology Office, National Science.
NetCDF-4 Interoperability with HDF4 and HDF5 Ed Hartnett Unidata, 8/4/9.
McIDAS-V McIDAS-V The 5 th Generation of McIDAS by Tom Whittaker Space Science and Engineering Center University of Wisconsin-Madison USA with contributions.
NetCDF Ed Hartnett Unidata/UCAR
Obtaining MISR Data and Information Jeff Walter Atmospheric Science Data Center April 17, 2009.
Status of netCDF-3, netCDF-4, and CF Conventions Russ Rew Community Standards for Unstructured Grids Workshop, Boulder
Show of Hands... How many traveled to be here? University/Gov't/Industry How many use netCDF? Primary programming language for netCDF? Other data formats.
Developing a NetCDF-4 Interface to HDF5 Data
Introduction to NetCDF4 MuQun Yang The HDF Group 11/6/2007HDF and HDF-EOS Workshop XI, Landover, MD.
Netcdf course Intro CF convention Netcdf excercises.
NetCDF-4 The Marriage of Two Data Formats Ed Hartnett, Unidata June, 2004.
NetCDF and HDF5 Ed Hartnett, Unidata/UCAR, Unidata Mission: To provide the data services, tools, and cyberinfrastructure leadership that advance.
Important ESDIS 2009 tasks review Kent Yang, Mike Folk The HDF Group April 1st, /1/20151Annual briefing to ESDIS.
Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.
NetCDF-4 and Parallel I/O GSFC, Nov 20,2008 Ed Hartnett.
NetCDF for High Performance Computing Introduction to NetCDF What is netCDF? NetCDF Data Models How we think of data. NetCDF Software Libraries Using.
Mid-Course Review: NetCDF in the Current Proposal Period Russ Rew
The netCDF-4 data model and format Russ Rew, UCAR Unidata NetCDF Workshop 25 October 2012.
Integrating netCDF and OPeNDAP (The DrNO Project) Dr. Dennis Heimbigner Unidata Go-ESSP Workshop Seattle, WA, Sept
Accessing Remote Datasets using the DAP protocol through the netCDF interface. Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009.
Advanced Utilities Extending ncgen to support the netCDF-4 Data Model Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009.
HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, Annual briefing to ESDIS10/31/2015.
The HDF Group HDF/HDF-EOS Workshop XV1 Tools to Improve the Usability of NASA HDF Data Kent Yang and Joe Lee The HDF Group April 17, 2012.
NetCDF Data Model Issues Russ Rew, UCAR Unidata NetCDF 2010 Workshop
University of San Diego Scripps Institution of Oceanography UDDC Adoption and Examples OceanSITES DMT Meeting November 2011.
NetCDF file generated from ASDC CERES SSF Subsetter ATMOSPHERIC SCIENCE DATA CENTER Conversion of Archived HDF Satellite Level 2 Swath Data Products to.
NPOESS Enhanced Description Tool - “ned” Richard E. Ullman NASA/GSFC/NPP NOAA/NESDIS/IPO Data / Information Architecture Algorithm / System Engineering.
Data formats and requirements in CMIP6: the climate-prediction case Pierre-Antoine Bretonnière EC-Earth meeting, Reading, May 2015.
The HDF Group Data Interoperability The HDF Group Staff Sep , 2010HDF/HDF-EOS Workshop XIV1.
The HDF Group Introduction to netCDF-4 Elena Pourmal The HDF Group 110/17/2015.
The HDF Group HDF Group Support for NPP/JPSS Mike Folk, Elena Pourmal, Larry Knox, Albert Cheng The HDF Group DEWG Meeting June 19, 2012.
NetCDF-4: Software Implementing an Enhanced Data Model for the Geosciences Russ Rew, Ed Hartnett, and John Caron UCAR Unidata Program, Boulder
NetCDF and Scientific Data Durability Russ Rew, UCAR Unidata ESIP Federation Summer Meeting
Data File Formats: netCDF by Tom Whittaker University of Wisconsin-Madison SSEC/CIMSS 2009 MUG Meeting June, 2009.
The HDF Group New Elements and Lessons Learned for New Mission HDF5 Products Ideas for new mission HDF5 data products 1July 8, 2013 Larry.
1 09/2003 Processing Library Update CF Checker – Script made available as a web based form on the BADC site -
Unidata Technologies Relevant to GO-ESSP: An Update Russ Rew
CF 2.0 Coming Soon? (Climate and Forecast Conventions for netCDF) Ethan Davis ESO Developing Standards - ESIP Summer Mtg 14 July 2015.
Developing Conventions for netCDF-4 Russ Rew, UCAR Unidata June 11, 2007 GO-ESSP.
Development of a CF Conventions API Russ Rew GO-ESSP Workshop, LLNL
Update on Unidata Technologies for Data Access Russ Rew
NcBrowse: A Graphical netCDF File Browser Donald Denbo NOAA-PMEL/UW-JISAO
Update on the GHRSST Users Manual GDS2.0 rev04.4 Book Captains: Chris Jeffery 1 and Jorge Vazquez Content Brief overview of the GHRSST project Background.
Utilities for netCDF-4 Dr. Dennis Heimbigner Unidata Advanced netCDF Workshop July 25, 2011.
Unidata Infrastructure for Data Services Russ Rew GO-ESSP Workshop, LLNL
Libcf – A CF Convention Library for NetCDF Ed Hartnett Unidata Program Center Boulder Colorado June 11, 2007.
NetCDF Data Model Details Russ Rew, UCAR Unidata NetCDF 2009 Workshop
Other Projects Relevant (and Not So Relevant) to the SODA Ideal: NetCDF, HDF, OLE/COM/DCOM, OpenDoc, Zope Sheila Denn INLS April 16, 2001.
1 April 14, Starting New Open Source Software Projects William Cohen NCSU CSC 591W April 14, 2008.
Adding CF Attributes to an HDF5 File
DAP+NETCDF Using the netCDF-4 Data Model
Data repositories CHFP at CIMA is the WGSIP data repository: OPeNDAP, web interface, NetCDF3, CHFP convention. Common complaint: Too many repositories.
Moving from HDF4 to HDF5/netCDF-4
NetCDF 3.6: What’s New Russ Rew
Plans for an Enhanced NetCDF-4 Interface to HDF5 Data
Efficiently serving HDF5 via OPeNDAP
Requirements for GSICS Plotting Tool to support VIS/NIR products
Extending the NetCDF Supported Data Formats using a Dispatch Layer
Masaya Takahashi (JMA) and Sebastien Wagner (EUMETSAT)
Updating GSICS Plotting Tool to support VISNIR products
Moving applications to HDF
Requirements on GSICS Plotting Tool to support VISNIR products
Status for Endeavor 6: Improved Scientific Data Access Infrastructure
GSICS Baseline Review: Product meta-data and structures
Masaya Takahashi Japan Meteorological Agency
NCL variable based on a netCDF variable model
Presentation transcript:

Libcf – A CF Convention Library for NetCDF Ed Hartnett Unidata Program Center Boulder Colorado June 11, 2007

Unidata Mission : To provide data, tools, and community leadership for enhanced Earth-system education and research.

NetCDF/libcf Development in C/C++/Fortran at Unidata NetCDF 3.6.2 released March, 2007 NetCDF 4.0 beta1 released April 2007 Libcf alpha4 released November 2006

The NetCDF Classic Model The netCDF classic data model: files, dimensions, variables, attributes, 6 data types. The expanded netCDF-4 data model adds: groups, additional atomic and user-defined data types, additional unlimited dimensions. Create netCDF-4/HDF5 files which conform to the classic model with a flag at create time.

Goals for libcf Make using CF Convetions easier for readers and writers of data. Reduce duplication of effort. Ensure CF conforming files. Provide a geo-science API for netCDF. Provide some of the advanced features developed for netCDF-Java. Work with both netCDF-3 and netCDF-4/HDF5 files.

Limitations of libcf Handles classic model netCDF only. Lightly tested. Doesn't handle indirect coordinates. Does not check standard name table. Alpha release - Functions in the API may change a little.

Current Status of libcf Available in alpha release from libcf web page at Unidata: http://www.unidata.ucar.edu/software/libcf/ Nightly build/test cycle. C and F77 APIs are provided. Seeking feedback and test cases from CF users.

Other Software Efforts Climate Model Output Rewritter (CMOR). CFIO from NASA/GMAO. Brian Ermold (ACRF Development) DOD project (just discovered).

Libcf and NetCDF-4

File Level Functions Mark file as CF-1.0 Does file follow CF- 1.0 Write or read CF recommended global attributes (title, history, institution, source, etc.)‏ nccf_def_convention nccf_inq_convention nccf_def_file nccf_add_history

Variable Functions Define or read CF recommended variable attributes (units, fill_value, institution, etc.)‏ nccf_def_var nc_inq_var nccf_def_var_missing nc_inq_var_missing nccf_def_notes nccf_inq_notes

Lat/Lon Functions Write and read simple latitude and longitude coordinate variables. Currently handles simple situations only. nc_def_latitude nc_inq_latitude nc_def_longitude nc_inq_longitude

Vertical Dimensions, First Try A vertical coordinate API, done like the lat/long API, proves more complex. int nccf_def_lvl(int ncid, const char *name, size_t len, nc_type xtype, const char *units, int positive_up, const char *standard_name, const char *formula_terms, int cdm_axis_type, int *lvl_dimidp, int *lvl_varidp); int nccf_inq_lvl(int ncid, char *name, size_t *lenp, nc_type *xtypep, size_t *ft_lenp, char *formula_terms, int *positive_upp, int *lvl_dimidp, int *lvl_varidp);

Vertical Dimensions, Second Try Functions to handle the vertical coordinates in appendix D of the CF 1.0 document. NetCDF define mode considerations make the situation more complex. nccf_def_lvl_sigma/nccf_def_ft_sigma nccf_inq_sigma Similarly for atmosphere_ln_pressure_coordinate, atmosphere_hybrid_height_coordinate, atmosphere_sleve_coordinate, etc.

Time Dimension Write and read time coordinate variable information. nc_def_time nc_inq_time

Geographic Subsetting Recent addition to libcf. Does not yet handle complex cases. int nccf_get_vara(int ncid, int varid, float *lat_range, int *nlat, float *lon_range, int *nlon, int lvl_index, int timestep, void *data);

An Example if (nc_create(FILE_NAME_SIMPLE_SIGMA, 0, &ncid)) ERR; if (nccf_def_file(ncid, TITLE, HISTORY)) ERR; /* Create coordinates. */ if (nccf_def_latitude(ncid, NLATS, NC_INT, &lat_did, &lat_vid)) ERR; if (nccf_def_longitude(ncid, NLONS, NC_INT, &lon_did, &lon_vid)) ERR; if (nccf_def_lvl_sigma(ncid, "sigma", NC_FLOAT, NLVLS, &lvl_did, &lvl_vid)) ERR; if (nccf_def_time(ncid, "time", NC_UNLIMITED, NC_FLOAT, TIME_UNITS, "time", &time_did, &time_vid)) ERR; <snip> /* Save the ps and ptop info in formula terms attribute. */ if (nccf_def_ft_sigma(ncid, lvl_vid, ps_vid, ptop_vid)) ERR;

Website and Documentation

Daily Snapshots and Testing

Future of libcf Better geographic subsetting. More testing on more sample files. Continued integration and borrowing of functionality from other CF efforts. More convenience functions. Optional standard name table checking. Optional UDUNITS checking. CF Conventions 2.0 and beyond. F90 and C++ APIs.

How to Contribute Send CF example data files to ed@unidata.ucar.edu Download and try the libcf library. Let us know of any additional parallel efforts.

Annual NetCDF Bike Rides Celebrate your favorite data format with a bike ride! Wednesday afternoon, rain or shine. Route to be determined – bike paths if possible!