Plans for an Enhanced NetCDF-4 Interface to HDF5 Data

Slides:

Advertisements

Similar presentations

/2829 November 2007 WDF-Presentation V Common Wind Tunnel Data Format.

Advertisements

Streaming NetCDF John Caron July What does NetCDF do for you? Data Storage: machine-, OS-, compiler-independent Standard API (Application Programming.

The Future of NetCDF Russ Rew UCAR Unidata Program Center Acknowledgments: John Caron, Ed Hartnett, NASA’s Earth Science Technology Office, National Science.

University of Illinois at Urbana-ChampaignHDF 1McGrath/Yang 2/27/02 Transitioning from HDF4 to HDF5 Robert E. McGrath Kent Yang.

NetCDF Ed Hartnett Unidata/UCAR

1 CF Unleashed: Introduction to Cf/Radial Joe VanAndel National Center for Atmospheric Research 2013/1/8 The National Center for Atmospheric.

Status of netCDF-3, netCDF-4, and CF Conventions Russ Rew Community Standards for Unstructured Grids Workshop, Boulder

Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps Mike Folks, The HDF Group Ruth Duerr, NSIDC 1.

Developing a NetCDF-4 Interface to HDF5 Data

ElVis Developments for Simulation and Analysis Programs Tarun Pondicherry Summer 2006 Science Ed High School Intern Eliot Feibush, Mentor 8/16/2006.

Introduction to NetCDF4 MuQun Yang The HDF Group 11/6/2007HDF and HDF-EOS Workshop XI, Landover, MD.

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Transition from HDF4 to HDF5: Issues Robert E. McGrath NCSA University of Illinois.

NetCDF-4 The Marriage of Two Data Formats Ed Hartnett, Unidata June, 2004.

NetCDF and HDF5 Ed Hartnett, Unidata/UCAR, Unidata Mission: To provide the data services, tools, and cyberinfrastructure leadership that advance.

Developing a NetCDF-4 Interface to HDF5 Data Russ Rew (PI), UCAR Unidata Mike Folk (Co-PI), NCSA/UIUC Ed Hartnett, UCAR Unidata Quincey Kozial, NCSA/UIUC.

1 Russ Rew, Ed Hartnett, John Caron UCAR Unidata Program Center Mike Folk, Robert McGrath, Quincey Kozial NCSA and The HDF Group, Inc. Final Project Review,

NetCDF for Developers and Data Providers Russ Rew, UCAR Unidata ICTP Advanced School on High Performance and Grid Computing 14 April 2011.

HDF5 A new file format & software for high performance scientific data management.

Important ESDIS 2009 tasks review Kent Yang, Mike Folk The HDF Group April 1st, /1/20151Annual briefing to ESDIS.

NPP/ NPOESS Product Data Format Richard E. Ullman NASA/GSFC/NPP NOAA/NESDIS/IPOAlgorithm / System EngineeringData / Information Architecture

“A Library outranks any other one thing a community can do to benefit its people.” Andrew Carnegie Mary R. Marlino, Ed.D. DLESE Program Center Presentation.

Mid-Course Review: NetCDF in the Current Proposal Period Russ Rew

The future of MINC Robert D. Vincent

HDF Converting between HDF4 and HDF5 MuQun Yang, Robert E. McGrath, Mike Folk National Center for Supercomputing Applications University of Illinois,

N P O E S S I N T E G R A T E D P R O G R A M O F F I C E NPP/ NPOESS Product Data Format Richard E. Ullman NOAA/NESDIS/IPO NASA/GSFC/NPP Algorithm Division.

The netCDF-4 data model and format Russ Rew, UCAR Unidata NetCDF Workshop 25 October 2012.

SciDAC All Hands Meeting, March 2-3, 2005 Northwestern University PIs:Alok Choudhary, Wei-keng Liao Graduate Students:Avery Ching, Kenin Coloma, Jianwei.

Opportunities in Parallel I/O for Scientific Data Management Rajeev Thakur and Rob Ross Mathematics and Computer Science Division Argonne National Laboratory.

HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) 1HDF and HDF-EOS Workshop XII10/17/2008.

Integrating netCDF and OPeNDAP (The DrNO Project) Dr. Dennis Heimbigner Unidata Go-ESSP Workshop Seattle, WA, Sept

Database tools An SQL layer around CDF. Overview Current SPENVIS database Limitations SQL wrapped CDF Advantages Demonstration.

A High performance I/O Module: the HDF5 WRF I/O module Muqun Yang, Robert E. McGrath, Mike Folk National Center for Supercomputing Applications University.

NetCDF Data Model Issues Russ Rew, UCAR Unidata NetCDF 2010 Workshop

Presented by Jens Schwidder Tara D. Gibson James D. Myers Computing & Computational Sciences Directorate Oak Ridge National Laboratory Scientific Annotation.

CCGrid, 2012 Supporting User Defined Subsetting and Aggregation over Parallel NetCDF Datasets Yu Su and Gagan Agrawal Department of Computer Science and.

NPOESS Enhanced Description Tool - “ned” Richard E. Ullman NASA/GSFC/NPP NOAA/NESDIS/IPO Data / Information Architecture Algorithm / System Engineering.

HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) 1HDF and HDF-EOS Workshop XII, Aurora,

The HDF Group Data Interoperability The HDF Group Staff Sep , 2010HDF/HDF-EOS Workshop XIV1.

The HDF Group Introduction to netCDF-4 Elena Pourmal The HDF Group 110/17/2015.

Parallel I/O Performance Study and Optimizations with HDF5, A Scientific Data Package MuQun Yang, Christian Chilan, Albert Cheng, Quincey Koziol, Mike.

NetCDF-4: Software Implementing an Enhanced Data Model for the Geosciences Russ Rew, Ed Hartnett, and John Caron UCAR Unidata Program, Boulder

NetCDF and Scientific Data Durability Russ Rew, UCAR Unidata ESIP Federation Summer Meeting

Aura HDF-EOS File Format Guidelines: Overview and Status Cheryl Craig.

Data File Formats: netCDF by Tom Whittaker University of Wisconsin-Madison SSEC/CIMSS 2009 MUG Meeting June, 2009.

Advances in the NetCDF Data Model, Format, and Software Russ Rew Coauthors: John Caron, Ed Hartnett, Dennis Heimbigner UCAR Unidata December 2010.

ENEON first workshop Observing Europe: Networking the Earth Observation Networks in Europe September, Paris [Name of your network/organization] [Name.

SDM Center Parallel I/O Storage Efficient Access Team.

CF 2.0 Coming Soon? (Climate and Forecast Conventions for netCDF) Ethan Davis ESO Developing Standards - ESIP Summer Mtg 14 July 2015.

Parallel I/O Performance Study and Optimizations with HDF5, A Scientific Data Package Christian Chilan, Kent Yang, Albert Cheng, Quincey Koziol, Leon Arber.

Development of a CF Conventions API Russ Rew GO-ESSP Workshop, LLNL

NetCDF: Data Model, Programming Interfaces, Conventions and Format Adapted from Presentations by Russ Rew Unidata Program Center University Corporation.

Update on Unidata Technologies for Data Access Russ Rew

Unidata Infrastructure for Data Services Russ Rew GO-ESSP Workshop, LLNL

NetCDF Data Model Details Russ Rew, UCAR Unidata NetCDF 2009 Workshop

Other Projects Relevant (and Not So Relevant) to the SODA Ideal: NetCDF, HDF, OLE/COM/DCOM, OpenDoc, Zope Sheila Denn INLS April 16, 2001.

NetCDF-Java version 2.2 Common Data Model John Caron Unidata/UCAR Dec 10, 2004.

Adding CF Attributes to an HDF5 File

Transition from HDF4 to HDF5: Issues

Moving from HDF4 to HDF5/netCDF-4

NetCDF 3.6: What’s New Russ Rew

Extending the NetCDF Supported Data Formats using a Dispatch Layer

Tad Scheiblich RSI December 2, 2005

Unidata & NetCDF BoF Scientific File Formats

Tom Rink Tom Whittaker Paolo Antonelli Kevin Baggett.

A BRIEF INTRODUCTION TO UNIX OPERATING SYSTEM

Updating GSICS Plotting Tool to support VISNIR products

Status for Endeavor 6: Improved Scientific Data Access Infrastructure

Hierarchical Data Format (HDF) Status Update

Masaya Takahashi Japan Meteorological Agency

OPeNDAP’s Server4: Building a High Performance Data Server for the DAP

Presentation transcript:

Plans for an Enhanced NetCDF-4 Interface to HDF5 Data Russ Rew, UCAR Unidata Program Mike Folk, NCSA/UIUC Ed Hartnett, UCAR Unidata Program Robert E. McGrath, NCSA/UIUC

What is NetCDF? Similarities with HDF5: Differences from HDF5: A scientific data model A data access library and set of interfaces A format for storing and accessing data Differences from HDF5: Simpler, more widely used (currently) Less powerful, less performance-oriented Current version: netCDF-3 is supported by UCAR Unidata Program

NetCDF-4/HDF5 Goals Combine desirable characteristics of netCDF and HDF5, while taking advantage of their separate strengths: Widespread use and simplicity of netCDF Generality and performance of HDF5 Preserve format and API compatibility for netCDF users Demonstrate benefits of combination in advanced Earth science modeling efforts

NetCDF-4 Features Enabled by HDF5 Large file support Parallel I/O Multiple dynamic dimensions Packed data, compression New data types Dynamic schema modifications Other possibilities: groups, Unicode, user-defined types, better coordinate support, …

Backward Compatibility Two kinds of compatibility: API: Extension of netCDF-3 API Support existing netCDF-3 applications by recompiling/relinking Format: Access to existing netCDF-3 data Access to HDF5 files written through netCDF-4 interface Leverage C interface for other interfaces to Fortran90, C++, Perl, Ruby, … Java interface must be developed separately

NetCDF-4 Architecture HDF5 Library netCDF-4 Library netCDF-3 Interface applications HDF5 netCDF files HDF5 files Supports access to netCDF files and HDF5 files created through netCDF-4 interface

NCSA Contributions Overall design Dimension scales Parallel I/O To support shared dimensions Addition to HDF5 data model Parallel I/O Using Argonne/Northwestern work Challenging performance goal

Acknowledgments and URL The netCDF-4/HDF5 project is funded by the NASA AIST program Unidata is sponsored by the National Science Foundation URL: www.unidata.ucar.edu/packages/netcdf-4