University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Transition from HDF4 to HDF5: Issues Robert E. McGrath NCSA University of Illinois at Urbana-Champaign

HDF 9/19/2000 McGrath 9/19/ Overall Goal A discussion of compatibility, conversion, and interoperability issues for HDF4 and HDF5, and HDFEOS. For more details see the white paper at:

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Conclusions (Preview) Transition from HDF4 to HDF5 is not necessarily transparent or effortless.

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Conclusions: HDF Transition from HDF4 to HDF5 is not transparent: in most cases source code will have to be written or rewritten. Many tools will support both HDF4 and HDF5. Default translation from HDF4 to HDF5 will be supported as a utility and library.

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Conclusions: HDF-EOS Transition from HDFEOS2 to HDFEOS3 is not transparent: in most cases source code will have to be written or rewritten Translation of ‘pure’ HDF-EOS will be supported Translation of ‘hybrid’ HDF-EOS files is very difficult. Not possible to support V.2 and V.3 in a single program

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ What Does Compatibility Mean?

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ What does “compatibility” mean? Ability to “do the same things” with new software –I.e., “compatibility” is relative to some purpose –Many dimensions and shades of compatibility –No golden solution sufficient for every use Assume HDF4 users want to continue doing what they are already doing, but move to HDF5

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Some Types of Compatibility

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Compatibility of HDF and HDF-EOS

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Interoperation My definition of “interoperation”: one program or tool may use HDF4 or HDF5 or both

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ NCSA HDF4 and HDF5

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Single Combined HDF Library dataset1 dataset2 Stored as HDF4 Objects image1 image2 Stored as HDF5 Objects HDF4 fileHDF5 file HDF 4 LibraryHDF 5 Library Interoperability: Any program may use either HDF4 or 5 or both: Not Supported

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Application Specific Library dataset1 dataset2 Stored as HDF4 Objects image1 image2 Stored as HDF5 Objects HDF4 fileHDF5 file HDF 4 LibraryHDF 5 Library Interoperability: Specific Application or tool may use either HDF4 or 5 or both: Up to Application and Tool Developers Application specific Library

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ HDF-EOS Version 2 and Version 3

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Grid1 Grid2 Stored as HDF4 Objects Grid3 Grid4 Stored as HDF5 Objects HDF4 fileHDF5 file HDF 4 LibraryHDF 5 Library HDFEOS Library Interoperability: Any program may using HDF-EOS Objects with either HDF4 or HDF5 or both: Not supported

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ HDFEOS V2 LibraryHDFEOS V3 Library Grid1 Grid2 Stored as HDF4 Objects Grid3 Grid4 Stored as HDF5 Objects HDF4 fileHDF5 file HDF 4 LibraryHDF 5 Library Interoperability: One application may use HDF-EOS Objects with either HDF4 or HDF5 or both: Not supported

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Translation Convert HDF4 objects (file) to HDF5 objects (file)

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Translation of HDF4 to HDF5 Stored as HDF4 Objects Stored as Analogous HDF5 Objects / HDF4 LibraryHDF5 Library HDF4 File HDF5 File Supported: NCSA mapping and h4toh5 tool and library

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Grid1 Grid2 Translation of HDF-EOS V2 to V3 HDFEOS Objects Stored as HDF4 Objects Grid1 Grid2 SAME HDFEOS Objects Stored as HDF5 Objects HDF4 file HDF5 file HDF4 Library / HDF5 Library HDF EOS V. 2.x Library HDF EOS V. 3 Library Supported

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ “Hybrid” HDF-EOS files HDF-EOS files may contain native HDF objects (images, arrays, etc.) as well as HDF-EOS objects (Grid, etc.) Almost all HDF-EOS files are ‘hybrids’. Very difficult to automatically translate hybrid files.

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ “Hybrid” HDF-EOS File NCSA HDF HDFEOS Science Application Code Other Components of HDFEOS Objects stored in HDF Objects Other HDF Objects Stored Directly Through HDF One HDF file, objects mixed together. Very difficult to tell which objects are part of HDF-EOS

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Conclusions Transition from HDF4 to HDF5 is not necessarily transparent or effortless.

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Conclusions: HDF Transition from HDF4 to HDF5 is not transparent: in most cases source code will have to be rewritten. Many tools will support both HDF4 and HDF5. Default translation from HDF4 to HDF5 will be supported as a utility and library.

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Conclusions: HDF-EOS Transition from HDFEOS2 to HDFEOS3 is not transparent: in most cases source code will have to be rewritten Translation of ‘pure’ HDF-EOS will be supported Translation of ‘hybrid’ HDF-EOS files is very difficult. Not possible to support V.2 and V.3 in a single program

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ For More Information NCSA White paper: “Transition From HDF4 to HDF5”: “Guidelines for Mapping HDF4 to HDF5 Objects”, h4toh5 tool: Soon