University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 1 Transition from HDF4 to HDF5: Issues Robert E. McGrath NCSA University of Illinois.

Slides:



Advertisements
Similar presentations
DRIVER Digital Repository Infrastructure Vision of European Research Guidelines for Content Providers Presented by Martin Feijen, SURF [NL]
Advertisements

HDF-EOS 2/5 to netCDF Converter Bob Bane, Richard Ullman, Jingli Yang Data Usability Group NASA/Goddard Space Flight Center.
HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.)
1 Desktop Support for HDF and HDF EOS Ted Meyer President & CEO fortnersoftware LLC (703) ext.119.
Chapter 8: Introduction to High-level Language Programming Invitation to Computer Science, C++ Version, Third Edition.
Chapter 8: Introduction to High-level Language Programming Invitation to Computer Science, C++ Version, Third Edition.
University of Illinois at Urbana-ChampaignHDF Mike Folk HDF-EOS Workshop IV Sept , 2000 HDF Update HDF.
University of Illinois at Urbana-ChampaignHDF 1McGrath/Yang 2/27/02 Transitioning from HDF4 to HDF5 Robert E. McGrath Kent Yang.
© Siemens Product Lifecycle Management Software Inc. All rights reserved Siemens PLM Software Top Questions of I-deas users Learning NX Mark Lawry.
HDF 1 NCSA HDF XML Activities Robert E. McGrath Mike Folk National Center for Supercomputing Applications.
The HDF-EOS5 Tutorial Ray Milburn L3 Communciations, EER Systems Inc McCormick Drive, 170 Largo, MD USA
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps Mike Folks, The HDF Group Ruth Duerr, NSIDC 1.
Developing a NetCDF-4 Interface to HDF5 Data
EARTH SCIENCE MARKUP LANGUAGE “Define Once Use Anywhere” INFORMATION TECHNOLOGY AND SYSTEMS CENTER UNIVERSITY OF ALABAMA IN HUNTSVILLE.
Experiments with Remote Access to Scientific Data using HDF5 Kun Yan Robert E. McGrath NCSA May 15, 2002.
HDF-EOS Workshop VII, An XML Approach to HDF-EOS5 Files Jingli Yang 1, Bob Bane 1, Muhammad Rabi 1, Zhangshi Yin 1, Richard Ullman 1, Robert McGrath.
EOSDIS User survey follow-up Mike Folk, Kent Yang, Elena Pourmal The HDF Group Oct. 17, 2012 Annual HDF Briefing to ESDIS1.
DM_PPT_NP_v01 SESIP_0715_GH1 America runs on Excel and HDF5 Gerd Heber The HDF Group Champaign Illinois USA This work was supported by NASA/GSFC under.
HDF Windows Support MuQun Yang, Xuan Bai, Elena Pourmal, Barbara Jones, Pedro Vincent, Robert E. McGrath National Center for Supercomputing Applications.
Using KDF9 Fortran to implement APT IV in Miles Ellis.
1 HDF-EOS APIs, tools, etc. Larry Klein, Abe Taaheri, and Cid Praderas L-3 Communications Government Services, Inc. November 30, 2005.
NASA EOS DATA COMPRESSION WITH HDF5 SCALEOFFSET FILTER This work was funded by the NASA Earth Science Technology Office under NASA award AIST and.
Important ESDIS 2009 tasks review Kent Yang, Mike Folk The HDF Group April 1st, /1/20151Annual briefing to ESDIS.
UNIT - 1Topic - 3. Computer software is a program that tells a computer what to do. Computer software, or just software, is any set of machine-readable.
HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term.
Why do I want to know about HDF and HDF- EOS? Hierarchical Data Format for the Earth Observing System (HDF-EOS) is NASA's primary format for standard data.
PROVIDING REMOTE ACCESS TO MAP SET AND SERIES HOLDINGS USING DIGITAL INDEX MAPS AS A DISCOVERY TOOL By Paige G. Andrew Faculty Maps Cataloger Pennsylvania.
HDF Converting between HDF4 and HDF5 MuQun Yang, Robert E. McGrath, Mike Folk National Center for Supercomputing Applications University of Illinois,
EARTH SCIENCE MARKUP LANGUAGE Why do you need it? How can it help you? INFORMATION TECHNOLOGY AND SYSTEMS CENTER UNIVERSITY OF ALABAMA IN HUNTSVILLE.
1/14/200925th IIPS Conference 1 Challenges to Archive and Access NASA HDF-EOS Data in the long Term MuQun Yang (The HDF Group) Choonghwan Lee (The HDF.
The HDF Group ESIP Summer Meeting HDF Studio John Readey The HDF Group 1 July 8 – 11, 2014.
The HDF Group HDF5 Tools Updates Peter Cao, The HDF Group September 28-30, 20101HDF and HDF-EOS Workshop XIV.
HDF Dimension Scales in HDF5 HDF-EOS Workshop IX San Francisco, CA November 30 - December 2, 2005 Pedro Vicente Nunes THG/NCSA Champaign-Urbana, IL HDF.
1 HDF-EOS Development Current Status and Schedule Larry Klein, Shen Zhao, Abe Taaheri and Ray Milburn L-3 Communications Government Services, Inc. September.
The HDF Group November 3-5, 2009 HDF-OPeNDAP Project Update HDF/HDF-EOS Workshop XIII1 Joe Lee and Kent Yang The HDF Group James Gallagher.
Towards Long-Term Archiving of NASA HDF-EOS and HDF Data Data Maps and the Use of Mark-Up Language Ruth Duerr, Mike Folk, Muqun Yang, Chris Lynnes, Peter.
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps Ruth Duerr, NSIDC Christopher Lynnes, GES DISC The HDF Group Oct HDF and.
HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) 1HDF and HDF-EOS Workshop XII10/17/2008.
Integrating netCDF and OPeNDAP (The DrNO Project) Dr. Dennis Heimbigner Unidata Go-ESSP Workshop Seattle, WA, Sept
A High performance I/O Module: the HDF5 WRF I/O module Muqun Yang, Robert E. McGrath, Mike Folk National Center for Supercomputing Applications University.
The HDF Group HDF/HDF-EOS Workshop XV1 Tools to Improve the Usability of NASA HDF Data Kent Yang and Joe Lee The HDF Group April 17, 2012.
- 1 - HDF5, HDF-EOS and Geospatial Data Archives HDF and HDF-EOS Workshop VII September 24, 2003.
NetCDF file generated from ASDC CERES SSF Subsetter ATMOSPHERIC SCIENCE DATA CENTER Conversion of Archived HDF Satellite Level 2 Swath Data Products to.
HDF4 OPeNDAP Project Progress Report MuQun Yang and Hyo-Kyung Lee 1 HDF Developers' Meeting11/24/2015.
1 HDF Vendors/Software Developers Workshop HDF And HDF-EOS Tools R.Suresh NASA/GSFC/HSTX Ph: FAX:
HDF Windows Support MuQun Yang, Xuan Bai, Elena Pourmal, Barbara Jones, Pedro Vincent, Robert E. McGrath National Center for Supercomputing Applications.
1 HDF-EOS Workshop II HDF-EOS Workshop II Introduction Candace Carlisle Earth Science Data and Information Systems (ESDIS) Project.
HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) 1 HDF and HDF-EOS Workshop XII10/17/2008.
Using a Friendly OPeNDAP Client Library to Access HDF5 Data MuQun Yang and Hyo-Kyung Lee (The HDF Group) 1 25th IIPS Conference01/14/2009.
HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) 1HDF and HDF-EOS Workshop XII, Aurora,
The HDF Group Data Interoperability The HDF Group Staff Sep , 2010HDF/HDF-EOS Workshop XIV1.
GLAS Standard Data Products for Distribution by NSIDC Polar DAAC User Working Group PoDAG Meeting XVI February 2000 Presenters H. Jay Zwally, NASA/GSFC.
1 Status of HDF-EOS, Related Software and Tools. 2 TOOLKIT / HDF-EOS Support.
Implementation of a Relational Database as an Aid to Automatic Target Recognition Christopher C. Frost Computer Science Mentor: Steven Vanstone.
Robert Wolfe NASA Goddard Space Flight Center Code 614.5, Greenbelt, MD Robert Wolfe NASA Goddard Space Flight Center Code 614.5,
HDF and HDF-EOS Workshop VII September 24, 2003 HDF5, HDF-EOS and Geospatial Data Archives Don Keefer Illinois State Geological Survey Mike Folk Univ.
FITSIO, HDF4, NetCDF, PDB and HDF5 Performance Some Benchmarks Results Elena Pourmal Science Data Processing Workshop February 27, 2002.
HDF-EOS Workshop IV September 19-21, 2000 Richard E. Ullman ESDIS Information Architect NASA/ GSFC, Code 423.
11/8/2007HDF and HDF-EOS Workshop XI, Landover, MD1 Software to access HDF5 Datasets via OPeNDAP MuQun Yang, Hyo-Kyung Lee The HDF Group.
Scion Macros How to make macros for Scion The Fast and Easy Guide.
Vincenzo Innocente, CERN/EPUser Collections1 Grid Scenarios in CMS Vincenzo Innocente CERN/EP Simulation, Reconstruction and Analysis scenarios.
ECHO Technical Interchange Meeting 2013 Timothy Goff 1 Raytheon EED Program | ECHO Technical Interchange 2013.
HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) 1HDF and HDF-EOS Workshop XII, Aurora,
Emergent Information Technologies, Inc.
Transition from HDF4 to HDF5: Issues
Moving from HDF4 to HDF5/netCDF-4
Plans for an Enhanced NetCDF-4 Interface to HDF5 Data
Kent Yang, Mike Folk The HDF Group March 31, 2009
POOL persistency framework for LHC
COMPUTER SOFT WARE Software is a set of electronic instructions that tells the computer how to do certain tasks. A set of instructions is often called.
Presentation transcript:

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Transition from HDF4 to HDF5: Issues Robert E. McGrath NCSA University of Illinois at Urbana-Champaign

HDF 9/19/2000 McGrath 9/19/ Overall Goal A discussion of compatibility, conversion, and interoperability issues for HDF4 and HDF5, and HDFEOS. For more details see the white paper at:

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Conclusions (Preview) Transition from HDF4 to HDF5 is not necessarily transparent or effortless.

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Conclusions: HDF Transition from HDF4 to HDF5 is not transparent: in most cases source code will have to be written or rewritten. Many tools will support both HDF4 and HDF5. Default translation from HDF4 to HDF5 will be supported as a utility and library.

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Conclusions: HDF-EOS Transition from HDFEOS2 to HDFEOS3 is not transparent: in most cases source code will have to be written or rewritten Translation of ‘pure’ HDF-EOS will be supported Translation of ‘hybrid’ HDF-EOS files is very difficult. Not possible to support V.2 and V.3 in a single program

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ What Does Compatibility Mean?

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ What does “compatibility” mean? Ability to “do the same things” with new software –I.e., “compatibility” is relative to some purpose –Many dimensions and shades of compatibility –No golden solution sufficient for every use Assume HDF4 users want to continue doing what they are already doing, but move to HDF5

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Some Types of Compatibility

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Compatibility of HDF and HDF-EOS

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Interoperation My definition of “interoperation”: one program or tool may use HDF4 or HDF5 or both

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ NCSA HDF4 and HDF5

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Single Combined HDF Library dataset1 dataset2 Stored as HDF4 Objects image1 image2 Stored as HDF5 Objects HDF4 fileHDF5 file HDF 4 LibraryHDF 5 Library Interoperability: Any program may use either HDF4 or 5 or both: Not Supported

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Application Specific Library dataset1 dataset2 Stored as HDF4 Objects image1 image2 Stored as HDF5 Objects HDF4 fileHDF5 file HDF 4 LibraryHDF 5 Library Interoperability: Specific Application or tool may use either HDF4 or 5 or both: Up to Application and Tool Developers Application specific Library

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ HDF-EOS Version 2 and Version 3

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Grid1 Grid2 Stored as HDF4 Objects Grid3 Grid4 Stored as HDF5 Objects HDF4 fileHDF5 file HDF 4 LibraryHDF 5 Library HDFEOS Library Interoperability: Any program may using HDF-EOS Objects with either HDF4 or HDF5 or both: Not supported

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ HDFEOS V2 LibraryHDFEOS V3 Library Grid1 Grid2 Stored as HDF4 Objects Grid3 Grid4 Stored as HDF5 Objects HDF4 fileHDF5 file HDF 4 LibraryHDF 5 Library Interoperability: One application may use HDF-EOS Objects with either HDF4 or HDF5 or both: Not supported

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Translation Convert HDF4 objects (file) to HDF5 objects (file)

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Translation of HDF4 to HDF5 Stored as HDF4 Objects Stored as Analogous HDF5 Objects / HDF4 LibraryHDF5 Library HDF4 File HDF5 File Supported: NCSA mapping and h4toh5 tool and library

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Grid1 Grid2 Translation of HDF-EOS V2 to V3 HDFEOS Objects Stored as HDF4 Objects Grid1 Grid2 SAME HDFEOS Objects Stored as HDF5 Objects HDF4 file HDF5 file HDF4 Library / HDF5 Library HDF EOS V. 2.x Library HDF EOS V. 3 Library Supported

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ “Hybrid” HDF-EOS files HDF-EOS files may contain native HDF objects (images, arrays, etc.) as well as HDF-EOS objects (Grid, etc.) Almost all HDF-EOS files are ‘hybrids’. Very difficult to automatically translate hybrid files.

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ “Hybrid” HDF-EOS File NCSA HDF HDFEOS Science Application Code Other Components of HDFEOS Objects stored in HDF Objects Other HDF Objects Stored Directly Through HDF One HDF file, objects mixed together. Very difficult to tell which objects are part of HDF-EOS

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Conclusions Transition from HDF4 to HDF5 is not necessarily transparent or effortless.

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Conclusions: HDF Transition from HDF4 to HDF5 is not transparent: in most cases source code will have to be rewritten. Many tools will support both HDF4 and HDF5. Default translation from HDF4 to HDF5 will be supported as a utility and library.

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Conclusions: HDF-EOS Transition from HDFEOS2 to HDFEOS3 is not transparent: in most cases source code will have to be rewritten Translation of ‘pure’ HDF-EOS will be supported Translation of ‘hybrid’ HDF-EOS files is very difficult. Not possible to support V.2 and V.3 in a single program

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ For More Information NCSA White paper: “Transition From HDF4 to HDF5”: “Guidelines for Mapping HDF4 to HDF5 Objects”, h4toh5 tool: Soon