Transition from HDF4 to HDF5: Issues Robert E. McGrath NCSA University of Illinois at Urbana-Champaign 9/19/2000 McGrath 9/19/2000

For more details see the white paper at: Overall Goal A discussion of compatibility, conversion, and interoperability issues for HDF4 and HDF5, and HDFEOS. For more details see the white paper at: http://hdf.ncsa.uiuc.edu/HDF5/doc/TechNotes/TransitionIssues.pdf 9/19/2000 McGrath 9/19/2000

Conclusions (Preview) Transition from HDF4 to HDF5 is not necessarily transparent or effortless. 9/19/2000 McGrath 9/19/2000

Conclusions: HDF Transition from HDF4 to HDF5 is not transparent: in most cases source code will have to be written or rewritten. Many tools will support both HDF4 and HDF5. Default translation from HDF4 to HDF5 will be supported as a utility and library. 9/19/2000 McGrath 9/19/2000

Conclusions: HDF-EOS Transition from HDFEOS2 to HDFEOS3 is not transparent: in most cases source code will have to be written or rewritten Translation of ‘pure’ HDF-EOS will be supported Translation of ‘hybrid’ HDF-EOS files is very difficult. Not possible to support V.2 and V.3 in a single program 9/19/2000 McGrath 9/19/2000

What Does Compatibility Mean? 9/19/2000 McGrath 9/19/2000

What does “compatibility” mean? Ability to “do the same things” with new software I.e., “compatibility” is relative to some purpose Many dimensions and shades of compatibility No golden solution sufficient for every use Assume HDF4 users want to continue doing what they are already doing, but move to HDF5 9/19/2000 McGrath 9/19/2000

Some Types of Compatibility 9/19/2000 McGrath 9/19/2000

Compatibility of HDF and HDF-EOS 9/19/2000 McGrath 9/19/2000

Interoperation My definition of “interoperation”: one program or tool may use HDF4 or HDF5 or both 9/19/2000 McGrath 9/19/2000

NCSA HDF4 and HDF5 9/19/2000 McGrath 9/19/2000

Interoperability: Any program may use either HDF4 or 5 or both: Not Supported image2 image1 dataset1 dataset2 Single Combined HDF Library HDF 4 Library HDF 5 Library Stored as HDF4 Objects Stored as HDF5 Objects HDF4 file HDF5 file 9/19/2000 McGrath 9/19/2000

Application specific Library Interoperability: Specific Application or tool may use either HDF4 or 5 or both: Up to Application and Tool Developers image2 image1 dataset1 dataset2 Application specific Library Application Specific Library HDF 4 Library HDF 5 Library Stored as HDF4 Objects Stored as HDF5 Objects HDF4 file HDF5 file 9/19/2000 McGrath 9/19/2000

HDF-EOS Version 2 and Version 3 9/19/2000 McGrath 9/19/2000

Interoperability: Any program may using HDF-EOS Objects with either HDF4 or HDF5 or both: Not supported HDFEOS Library Grid1 Grid2 Grid3 Grid4 HDF 4 Library HDF 5 Library Stored as HDF4 Objects Stored as HDF5 Objects HDF4 file HDF5 file 9/19/2000 McGrath 9/19/2000

Interoperability: One application may use HDF-EOS Objects with either HDF4 or HDF5 or both: Not supported Grid1 Grid2 Grid3 Grid4 HDFEOS V2 Library HDFEOS V3 Library HDF 4 Library HDF 5 Library Stored as HDF4 Objects Stored as HDF5 Objects HDF4 file HDF5 file 9/19/2000 McGrath 9/19/2000

Translation Convert HDF4 objects (file) to HDF5 objects (file) 9/19/2000 McGrath 9/19/2000

Translation of HDF4 to HDF5 Supported: NCSA mapping and h4toh5 tool and library HDF4 Library HDF5 Library / Stored as HDF4 Objects Stored as Analogous HDF5 Objects HDF5 File HDF4 File 9/19/2000 McGrath 9/19/2000

Translation of HDF-EOS V2 to V3 Supported HDF EOS V. 2.x Library HDF EOS V. 3 Library Grid1 Grid2 Grid1 Grid2 SAME HDFEOS Objects HDFEOS Objects HDF4 Library HDF5 Library Stored as HDF4 Objects Stored as HDF5 Objects / HDF5 file HDF4 file 9/19/2000 McGrath 9/19/2000

“Hybrid” HDF-EOS files HDF-EOS files may contain native HDF objects (images, arrays, etc.) as well as HDF-EOS objects (Grid, etc.) Almost all HDF-EOS files are ‘hybrids’. Very difficult to automatically translate hybrid files. 9/19/2000 McGrath 9/19/2000

Science Application Code “Hybrid” HDF-EOS File Science Application Code HDFEOS Other NCSA HDF Components of HDFEOS Objects stored in HDF Objects Other HDF Objects Stored Directly Through HDF Very difficult to tell which objects are part of HDF-EOS One HDF file, objects mixed together. 9/19/2000 McGrath 9/19/2000

Conclusions Transition from HDF4 to HDF5 is not necessarily transparent or effortless. 9/19/2000 McGrath 9/19/2000

Conclusions: HDF Transition from HDF4 to HDF5 is not transparent: in most cases source code will have to be rewritten. Many tools will support both HDF4 and HDF5. Default translation from HDF4 to HDF5 will be supported as a utility and library. 9/19/2000 McGrath 9/19/2000

Conclusions: HDF-EOS Transition from HDFEOS2 to HDFEOS3 is not transparent: in most cases source code will have to be rewritten Translation of ‘pure’ HDF-EOS will be supported Translation of ‘hybrid’ HDF-EOS files is very difficult. Not possible to support V.2 and V.3 in a single program 9/19/2000 McGrath 9/19/2000

For More Information NCSA White paper: “Transition From HDF4 to HDF5”: http://hdf.ncsa.uiuc.edu/HDF5/doc/TechNotes/TransitionIssues.pdf “Guidelines for Mapping HDF4 to HDF5 Objects”, http://hdf.ncsa.uiuc.edu/HDF5/papers/H4-to-H5MappingGuidelines.pdf h4toh5 tool: Soon 9/19/2000 McGrath 9/19/2000