Libcf – A CF Convention Library for NetCDF Ed Hartnett Unidata Program Center Boulder Colorado June 11, 2007.

Slides:



Advertisements
Similar presentations
Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1.
Advertisements

10 th Argo data management 2009 Toulouse Argo format and CF compatibility OceanOBS09 Strengthen and enhance the international framework under GCOS, GOOS,
Preparing CMOR for CMIP6 and other WCRP Projects
The Future of NetCDF Russ Rew UCAR Unidata Program Center Acknowledgments: John Caron, Ed Hartnett, NASA’s Earth Science Technology Office, National Science.
NetCDF-4 Interoperability with HDF4 and HDF5 Ed Hartnett Unidata, 8/4/9.
NetCDF Ed Hartnett Unidata/UCAR
Introduction to NetCDF Russ Rew, UCAR Unidata ICTP Advanced School on High Performance and Grid Computing 13 April 2011.
Obtaining MISR Data and Information Jeff Walter Atmospheric Science Data Center April 17, 2009.
Status of netCDF-3, netCDF-4, and CF Conventions Russ Rew Community Standards for Unstructured Grids Workshop, Boulder
Show of Hands... How many traveled to be here? University/Gov't/Industry How many use netCDF? Primary programming language for netCDF? Other data formats.
Developing a NetCDF-4 Interface to HDF5 Data
Introduction to NetCDF4 MuQun Yang The HDF Group 11/6/2007HDF and HDF-EOS Workshop XI, Landover, MD.
Netcdf course Intro CF convention Netcdf excercises.
NetCDF-4 The Marriage of Two Data Formats Ed Hartnett, Unidata June, 2004.
NetCDF and HDF5 Ed Hartnett, Unidata/UCAR, Unidata Mission: To provide the data services, tools, and cyberinfrastructure leadership that advance.
Developing a NetCDF-4 Interface to HDF5 Data Russ Rew (PI), UCAR Unidata Mike Folk (Co-PI), NCSA/UIUC Ed Hartnett, UCAR Unidata Quincey Kozial, NCSA/UIUC.
Unidata’s TDS Workshop TDS Overview – Part II October 2012.
Important ESDIS 2009 tasks review Kent Yang, Mike Folk The HDF Group April 1st, /1/20151Annual briefing to ESDIS.
Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.
NetCDF-4 and Parallel I/O GSFC, Nov 20,2008 Ed Hartnett.
NetCDF for High Performance Computing Introduction to NetCDF What is netCDF? NetCDF Data Models How we think of data. NetCDF Software Libraries Using.
Mid-Course Review: NetCDF in the Current Proposal Period Russ Rew
07:44:46Service Oriented Cyberinfrastructure Lab, Introduction to BOINC By: Andrew J Younge
The netCDF-4 data model and format Russ Rew, UCAR Unidata NetCDF Workshop 25 October 2012.
An Internet of Things: People, Processes, and Products in the Spotfire Cloud Library Dr. Brand Niemann Director and Senior Data Scientist/Data Journalist.
Integrating netCDF and OPeNDAP (The DrNO Project) Dr. Dennis Heimbigner Unidata Go-ESSP Workshop Seattle, WA, Sept
Accessing Remote Datasets using the DAP protocol through the netCDF interface. Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009.
Advanced Utilities Extending ncgen to support the netCDF-4 Data Model Dr. Dennis Heimbigner Unidata netCDF Workshop August 3-4, 2009.
HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, Annual briefing to ESDIS10/31/2015.
Using and modifying plan constraints in Constable Jim Blythe and Yolanda Gil Temple project USC Information Sciences Institute
NetCDF Data Model Issues Russ Rew, UCAR Unidata NetCDF 2010 Workshop
University of San Diego Scripps Institution of Oceanography UDDC Adoption and Examples OceanSITES DMT Meeting November 2011.
NetCDF file generated from ASDC CERES SSF Subsetter ATMOSPHERIC SCIENCE DATA CENTER Conversion of Archived HDF Satellite Level 2 Swath Data Products to.
The CF Conventions: Options for Sustained Support Involving Unidata Russ Rew Unidata Policy Committee May 12, 2008.
Data formats and requirements in CMIP6: the climate-prediction case Pierre-Antoine Bretonnière EC-Earth meeting, Reading, May 2015.
The HDF Group Data Interoperability The HDF Group Staff Sep , 2010HDF/HDF-EOS Workshop XIV1.
ELandings Data Extract Mechanisms. Data Extract Options:
The HDF Group Introduction to netCDF-4 Elena Pourmal The HDF Group 110/17/2015.
NetCDF-4: Software Implementing an Enhanced Data Model for the Geosciences Russ Rew, Ed Hartnett, and John Caron UCAR Unidata Program, Boulder
NetCDF and Scientific Data Durability Russ Rew, UCAR Unidata ESIP Federation Summer Meeting
00/XXXX 1 Data Processing in PRISM Introduction. COCO (CDMS Overloaded for CF Objects) What is it. Why is COCO written in Python. Implementation Data Operations.
Data File Formats: netCDF by Tom Whittaker University of Wisconsin-Madison SSEC/CIMSS 2009 MUG Meeting June, 2009.
1 09/2003 Processing Library Update CF Checker – Script made available as a web based form on the BADC site -
Unidata Technologies Relevant to GO-ESSP: An Update Russ Rew
CF 2.0 Coming Soon? (Climate and Forecast Conventions for netCDF) Ethan Davis ESO Developing Standards - ESIP Summer Mtg 14 July 2015.
Developing Conventions for netCDF-4 Russ Rew, UCAR Unidata June 11, 2007 GO-ESSP.
1 2.5 DISTRIBUTED DATA INTEGRATION WTF-CEOP (WGISS Test Facility for CEOP) May 2007 Yonsook Enloe (NASA/SGT) Chris Lynnes (NASA)
Development of a CF Conventions API Russ Rew GO-ESSP Workshop, LLNL
Update on Unidata Technologies for Data Access Russ Rew
NcBrowse: A Graphical netCDF File Browser Donald Denbo NOAA-PMEL/UW-JISAO
Update on the GHRSST Users Manual GDS2.0 rev04.4 Book Captains: Chris Jeffery 1 and Jorge Vazquez Content Brief overview of the GHRSST project Background.
Utilities for netCDF-4 Dr. Dennis Heimbigner Unidata Advanced netCDF Workshop July 25, 2011.
Unidata Infrastructure for Data Services Russ Rew GO-ESSP Workshop, LLNL
NetCDF Data Model Details Russ Rew, UCAR Unidata NetCDF 2009 Workshop
I/O server work at ICHEC Alastair McKinstry IS-ENES workshop, 2013.
Lecture 1 Page 1 CS 111 Summer 2013 Important OS Properties For real operating systems built and used by real people Differs depending on who you are talking.
Other Projects Relevant (and Not So Relevant) to the SODA Ideal: NetCDF, HDF, OLE/COM/DCOM, OpenDoc, Zope Sheila Denn INLS April 16, 2001.
DAP+NETCDF Using the netCDF-4 Data Model
Moving from HDF4 to HDF5/netCDF-4
NetCDF 3.6: What’s New Russ Rew
Plans for an Enhanced NetCDF-4 Interface to HDF5 Data
Requirements for GSICS Plotting Tool to support VIS/NIR products
Extending the NetCDF Supported Data Formats using a Dispatch Layer
Unidata Advanced netCDF Workshop
Updating GSICS Plotting Tool to support VISNIR products
Status for Endeavor 6: Improved Scientific Data Access Infrastructure
Libcf – A CF Convention Library for NetCDF
GSICS Baseline Review: Product meta-data and structures
Masaya Takahashi Japan Meteorological Agency
NCL variable based on a netCDF variable model
Presentation transcript:

Libcf – A CF Convention Library for NetCDF Ed Hartnett Unidata Program Center Boulder Colorado June 11, 2007

Unidata Mission : To provide data, tools, and community leadership for enhanced Earth-system education and research.

NetCDF/libcf Development in C/C++/Fortran at Unidata NetCDF released March, 2007 NetCDF 4.0 beta1 released April 2007 Libcf alpha4 released November 2006

The NetCDF Classic Model The netCDF classic data model: files, dimensions, variables, attributes, 6 data types. The expanded netCDF-4 data model adds: groups, additional atomic and user-defined data types, additional unlimited dimensions. Create netCDF-4/HDF5 files which conform to the classic model with a flag at create time.

Goals for libcf Make using CF Convetions easier for readers and writers of data. Reduce duplication of effort. Ensure CF conforming files. Provide a geo-science API for netCDF. Provide some of the advanced features developed for netCDF-Java. Work with both netCDF-3 and netCDF-4/HDF5 files.

Limitations of libcf Handles classic model netCDF only. Lightly tested. Doesn't handle indirect coordinates. Does not check standard name table. Alpha release - Functions in the API may change a little.

Current Status of libcf Available in alpha release from libcf web page at Unidata: Nightly build/test cycle. C and F77 APIs are provided. Seeking feedback and test cases from CF users.

Other Software Efforts Climate Model Output Rewritter (CMOR). CFIO from NASA/GMAO. Brian Ermold (ACRF Development) DOD project (just discovered).

Libcf and NetCDF-4

File Level Functions Mark file as CF-1.0 Does file follow CF- 1.0 Write or read CF recommended global attributes (title, history, institution, source, etc.)‏ nccf_def_convention nccf_inq_convention nccf_def_file nccf_add_history

Variable Functions Define or read CF recommended variable attributes (units, fill_value, institution, etc.)‏ nccf_def_var nc_inq_var nccf_def_var_missing nc_inq_var_missing nccf_def_notes nccf_inq_notes

Lat/Lon Functions Write and read simple latitude and longitude coordinate variables. Currently handles simple situations only. nc_def_latitude nc_inq_latitude nc_def_longitude nc_inq_longitude

Vertical Dimensions, First Try A vertical coordinate API, done like the lat/long API, proves more complex. int nccf_def_lvl(int ncid, const char *name, size_t len, nc_type xtype, const char *units, int positive_up, const char *standard_name, const char *formula_terms, int cdm_axis_type, int *lvl_dimidp, int *lvl_varidp); int nccf_inq_lvl(int ncid, char *name, size_t *lenp, nc_type *xtypep, size_t *ft_lenp, char *formula_terms, int *positive_upp, int *lvl_dimidp, int *lvl_varidp);

Vertical Dimensions, Second Try Functions to handle the vertical coordinates in appendix D of the CF 1.0 document. NetCDF define mode considerations make the situation more complex. nccf_def_lvl_sigma/nccf_def_ft_sigma nccf_inq_sigma Similarly for atmosphere_ln_pressure_coordinate, atmosphere_hybrid_height_coordinate, atmosphere_sleve_coordinate, etc.

Time Dimension Write and read time coordinate variable information. nc_def_time nc_inq_time

Geographic Subsetting Recent addition to libcf. Does not yet handle complex cases. int nccf_get_vara(int ncid, int varid, float *lat_range, int *nlat, float *lon_range, int *nlon, int lvl_index, int timestep, void *data);

An Example if (nc_create(FILE_NAME_SIMPLE_SIGMA, 0, &ncid)) ERR; if (nccf_def_file(ncid, TITLE, HISTORY)) ERR; /* Create coordinates. */ if (nccf_def_latitude(ncid, NLATS, NC_INT, &lat_did, &lat_vid)) ERR; if (nccf_def_longitude(ncid, NLONS, NC_INT, &lon_did, &lon_vid)) ERR; if (nccf_def_lvl_sigma(ncid, "sigma", NC_FLOAT, NLVLS, &lvl_did, &lvl_vid)) ERR; if (nccf_def_time(ncid, "time", NC_UNLIMITED, NC_FLOAT, TIME_UNITS, "time", &time_did, &time_vid)) ERR; /* Save the ps and ptop info in formula terms attribute. */ if (nccf_def_ft_sigma(ncid, lvl_vid, ps_vid, ptop_vid)) ERR;

Website and Documentation

Daily Snapshots and Testing

Future of libcf Better geographic subsetting. More testing on more sample files. Continued integration and borrowing of functionality from other CF efforts. More convenience functions. Optional standard name table checking. Optional UDUNITS checking. CF Conventions 2.0 and beyond. F90 and C++ APIs.

How to Contribute Send CF example data files to Download and try the libcf library. Let us know of any additional parallel efforts.

Annual NetCDF Bike Rides Celebrate your favorite data format with a bike ride! Wednesday afternoon, rain or shine. Route to be determined – bike paths if possible!