1. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 1 Transition from HDF4 to HDF5: Issues Robert E. McGrath NCSA University of Illinois at Urbana-Champaign

2. HDF 9/19/2000 McGrath 9/19/2000 2 Overall Goal A discussion of compatibility, conversion, and interoperability issues for HDF4 and HDF5, and HDFEOS. For more details see the white paper at: http://hdf.ncsa.uiuc.edu/HDF5/doc/TechNotes/TransitionIssues.pdf

3. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 3 Conclusions (Preview) Transition from HDF4 to HDF5 is not necessarily transparent or effortless.

4. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 4 Conclusions: HDF Transition from HDF4 to HDF5 is not transparent: in most cases source code will have to be written or rewritten. Many tools will support both HDF4 and HDF5. Default translation from HDF4 to HDF5 will be supported as a utility and library.

5. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 5 Conclusions: HDF-EOS Transition from HDFEOS2 to HDFEOS3 is not transparent: in most cases source code will have to be written or rewritten Translation of ‘pure’ HDF-EOS will be supported Translation of ‘hybrid’ HDF-EOS files is very difficult. Not possible to support V.2 and V.3 in a single program

6. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 6 What Does Compatibility Mean?

7. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 7 What does “compatibility” mean? Ability to “do the same things” with new software –I.e., “compatibility” is relative to some purpose –Many dimensions and shades of compatibility –No golden solution sufficient for every use Assume HDF4 users want to continue doing what they are already doing, but move to HDF5

8. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 8 Some Types of Compatibility

9. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 9 Compatibility of HDF and HDF-EOS

10. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 10Interoperation My definition of “interoperation”: one program or tool may use HDF4 or HDF5 or both

11. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 11 NCSA HDF4 and HDF5

12. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 12 Single Combined HDF Library dataset1 dataset2 Stored as HDF4 Objects image1 image2 Stored as HDF5 Objects HDF4 fileHDF5 file HDF 4 LibraryHDF 5 Library Interoperability: Any program may use either HDF4 or 5 or both: Not Supported

13. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 13 Application Specific Library dataset1 dataset2 Stored as HDF4 Objects image1 image2 Stored as HDF5 Objects HDF4 fileHDF5 file HDF 4 LibraryHDF 5 Library Interoperability: Specific Application or tool may use either HDF4 or 5 or both: Up to Application and Tool Developers Application specific Library

14. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 14 HDF-EOS Version 2 and Version 3

15. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 15 Grid1 Grid2 Stored as HDF4 Objects Grid3 Grid4 Stored as HDF5 Objects HDF4 fileHDF5 file HDF 4 LibraryHDF 5 Library HDFEOS Library Interoperability: Any program may using HDF-EOS Objects with either HDF4 or HDF5 or both: Not supported

16. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 16 HDFEOS V2 LibraryHDFEOS V3 Library Grid1 Grid2 Stored as HDF4 Objects Grid3 Grid4 Stored as HDF5 Objects HDF4 fileHDF5 file HDF 4 LibraryHDF 5 Library Interoperability: One application may use HDF-EOS Objects with either HDF4 or HDF5 or both: Not supported

17. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 17Translation Convert HDF4 objects (file) to HDF5 objects (file)

18. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 18 Translation of HDF4 to HDF5 Stored as HDF4 Objects Stored as Analogous HDF5 Objects / HDF4 LibraryHDF5 Library HDF4 File HDF5 File Supported: NCSA mapping and h4toh5 tool and library

19. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 19 Grid1 Grid2 Translation of HDF-EOS V2 to V3 HDFEOS Objects Stored as HDF4 Objects Grid1 Grid2 SAME HDFEOS Objects Stored as HDF5 Objects HDF4 file HDF5 file HDF4 Library / HDF5 Library HDF EOS V. 2.x Library HDF EOS V. 3 Library Supported

20. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 20 “Hybrid” HDF-EOS files HDF-EOS files may contain native HDF objects (images, arrays, etc.) as well as HDF-EOS objects (Grid, etc.) Almost all HDF-EOS files are ‘hybrids’. Very difficult to automatically translate hybrid files.

21. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 21 “Hybrid” HDF-EOS File NCSA HDF HDFEOS Science Application Code Other Components of HDFEOS Objects stored in HDF Objects Other HDF Objects Stored Directly Through HDF One HDF file, objects mixed together. Very difficult to tell which objects are part of HDF-EOS

22. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 22 Conclusions Transition from HDF4 to HDF5 is not necessarily transparent or effortless.

23. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 23 Conclusions: HDF Transition from HDF4 to HDF5 is not transparent: in most cases source code will have to be rewritten. Many tools will support both HDF4 and HDF5. Default translation from HDF4 to HDF5 will be supported as a utility and library.

24. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 24 Conclusions: HDF-EOS Transition from HDFEOS2 to HDFEOS3 is not transparent: in most cases source code will have to be rewritten Translation of ‘pure’ HDF-EOS will be supported Translation of ‘hybrid’ HDF-EOS files is very difficult. Not possible to support V.2 and V.3 in a single program

25. University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/2000 25 For More Information NCSA White paper: “Transition From HDF4 to HDF5”: http://hdf.ncsa.uiuc.edu/HDF5/doc/TechNotes/TransitionIssues.pdf “Guidelines for Mapping HDF4 to HDF5 Objects”, http://hdf.ncsa.uiuc.edu/HDF5/papers/H4-to-H5MappingGuidelines.pdf h4toh5 tool: Soon