1. Peter Cao The HDF Group xcao@hdfgroup.org November 28, 2006
HDF5 Tools Update
Peter Cao
The HDF Group
November 28, 2006
This report is based upon work supported in part by a Cooperative Agreement with NASA under NASA NNG05GC60A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration.
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

2. HDF and HDF-EOS Workshop X, Landover, MD
Outline
Overview of current tools
New tools with HDF5 1.8 release
A future tool: h5ub
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

3. HDF and HDF-EOS Workshop X, Landover, MD
HDF5 Command Line Tools
Readers
h5dump, h5diff, h5ls
new tools: h5check, h5stat
Writers
h5repack, h5repart, h5import, h5jam/h5unjam
new tool: h5copy
Converters
h4toh5, h5toh4, gif2h5, h52gif
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

4. HDF and HDF-EOS Workshop X, Landover, MD
Help Information
Located at bin/ with binary release
Use “-h” option for help
Online help at
For further help, bug report and feature request, send to
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

5. HDF and HDF-EOS Workshop X, Landover, MD
h5dump
Dumps the content of an HDF5 file to stdout and optionally to following types of files
ASCII text file
XML file
Binary file (new feature)
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

6. HDF and HDF-EOS Workshop X, Landover, MD
h5dump -H SDS.h5
HDF5 "SDS.h5" {
GROUP "/" {
GROUP "Floats" {
DATASET "FloatArray" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) } }
DATASET "IntArray" {
DATATYPE H5T_STD_I32LE
DATASPACE SIMPLE { ( 5, 6 ) / ( 5, 6 ) }
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

7. h5dump -d /Floats/FloatArray SDS.h5
HDF5 "SDS.h5" {
DATASET "/Floats/FloatArray" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) }
DATA {
(0,0): 0.01, 0.02, 0.03,
(1,0): 0.1, 0.2, 0.3,
(2,0): 1, 2, 3,
(3,0): 10, 20, 30 }
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

8. HDF and HDF-EOS Workshop X, Landover, MD
h5dump -x SDS.h5
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

9. HDF and HDF-EOS Workshop X, Landover, MD
h5dump Binary Output
-b F, --binary=F
The form of the binary output (F):
MEMORY -- for memory type
FILE -- for the disk file type
LE -- for pre-defined little endian type
BE for pre-defined big endian type
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

10. h5dump -d /IntArray -o out_le.bin -b LE SDS.h5
Dumps a 32-bit integer dataset, IntArray, from SDS.h5 to a little endian binary file out_le.bin
od --width=24 -t x4 out_le.bin
a b c d e f
e f
a b c d
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

11. HDF and HDF-EOS Workshop X, Landover, MD
h5diff
Using h5diff, you can
compares two objects in the same file
compares two objects between two files
compares all objects between two files
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

12. HDF and HDF-EOS Workshop X, Landover, MD
h5diff SDS.h5 SDS2.h5
Dataset: </IntArray> and </IntArray>
5 differences found
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

13. h5diff SDS.h5 SDS2.h5 -r /IntArray
Dataset: </IntArray> and </IntArray>
position IntArray IntArray difference
[ 0 0 ]
[ 1 0 ]
[ 2 0 ]
[ 3 0 ]
[ 4 0 ]
5 differences found
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

14. HDF and HDF-EOS Workshop X, Landover, MD
h5repack
Copies an HDF5 file to a new file with/without compression/chunking
Remove un-used space
Apply compression filter
Apply layout
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

15. HDF and HDF-EOS Workshop X, Landover, MD
h5repack: new filters
-f FILTER
GZIP, to apply GZIP compression
SZIP, to apply SZIP compression
SHUF, to apply the HDF5 shuffle filter
FLET, to apply the HDF5 checksum filter
NBIT, to apply NBIT compression
SOFF, to apply the HDF5 Scale/Offset filter
NONE, to remove all filters
For example
h5repack -i SDS2.h5 -o SDS2_compressed.h5 /IntArray:GZIP=9
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

16. HDF and HDF-EOS Workshop X, Landover, MD
h5repack: data layout
-l LAYOUT
CHUNK, to apply chunking layout
COMPA, to apply compact layout
CONTI, to apply continuous layout
For example
h5repack -i SDS.h5 -o SDS_chunk.h5
-l /Floats/FloatArray,/IntArray:CHUNK=2x3
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

17. new h5repack: using H5Ocopy()
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

18. HDF and HDF-EOS Workshop X, Landover, MD
h5repart
Repartitions a file or family of files
For example
h5repart -m 200m int16kx16k.h5 part200m%d.h5
200 MB
part200m0.h5
977 MB
200 MB
part200m1.h5
200 MB
part200m2.h5
200 MB
part200m3.h5
177 MB
part200m1.h5
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

19. HDF and HDF-EOS Workshop X, Landover, MD
h5import
Imports binary/ASCII data into an HDF5 file
h5import infile -c config_file [infile -c config_file2 ...] -outfile outfile
For eaxmple
h5import float5x4x2.txt -c First_set.conf -o First_set.h5
GROUP "/" {
GROUP "work" {
DATASET "First-set" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 5, 2, 4 ) / ( 8, 8, H5S_UNLIMITED ) }
DATA {
(0,0,0): 1.01, 1.02, 1.03, 1.04,
(0,1,0): 1.11, 1.12, 1.13, 1.14,
(1,0,0): 1.21, 1.22, 1.23, 1.24,
(1,1,0): 1.31, 1.32, 1.33, 1.34,
(2,0,0): 1.41, 1.42, 1.43, 1.44,
(2,1,0): 1.51, 1.52, 1.53, 1.54,
(3,0,0): 2.01, 2.02, 2.03, 2.04,
(3,1,0): 2.11, 2.12, 2.13, 2.14,
(4,0,0): 2.21, 2.22, 2.23, 2.24,
(4,1,0): 2.31, 2.32, 2.33, 2.34 } }}
PATH work/First-set
INPUT-CLASS TEXTFP
RANK 3
DIMENSION-SIZES 5 2 4
OUTPUT-CLASS FP
OUTPUT-SIZE 64
OUTPUT-ARCHITECTURE IEEE
OUTPUT-BYTE-ORDER LE
CHUNKED-DIMENSION-SIZES 2 2 2
MAXIMUM-DIMENSIONS
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

20. HDF and HDF-EOS Workshop X, Landover, MD
h5jam/h5unjam
h5jam -- add text to User Block
h5jam -u test_ub.txt -i test_ub.h5
h5unjam -- remove text from User Block
h5unjam -i test_ub.h5 -o out_ub.txt -o out_ub.h5
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

21. HDF and HDF-EOS Workshop X, Landover, MD
h5ls
Lists selected information about file objects in the specified format
For Example,
h5ls -r SDS2.h5
/Floats Group
/Floats/DoubleArray Dataset {10, 5}
/Floats/FloatArray Dataset {4, 3}
/Floats/subs Group
/IntArray Dataset {5, 6}
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

22. HDF and HDF-EOS Workshop X, Landover, MD
gif2h5 / h52gif
gif2h5 – converts a GIF file into HDF5
gif2h5 apollo17_earth.gif apollo17_earth.h5
h52gif – converts an HDF5 file into GIF
h52gif apollo17_earth.h5 apollo17_earth2.gif -i /apollo17_earth.gif/Image0 -p "/apollo17_earth.gif/Global Palette"
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

23. HDF and HDF-EOS Workshop X, Landover, MD
h5toh4 / h4toh5
h5toh4 -- Converts an HDF5 file to an HDF4 file
h4toh5 -- Converts an HDF4 file to an HDF5 file
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

24. HDF and HDF-EOS Workshop X, Landover, MD
New tools
h5copy
h5check
h5stat
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

25. HDF and HDF-EOS Workshop X, Landover, MD
h5copy
Copies an object from one location to another location within a file or across files / /
Floats
IntArray
FloatArray
FloatArray
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

26. HDF and HDF-EOS Workshop X, Landover, MD
h5copy
usage: h5copy [OPTIONS] [OBJECTS...]
-i, --input input file name
-o, --output output file name
-s, --source source object name
-d, --destination destination object name
-f, --flag
shallow Copy only immediate members for groups
soft Expand soft links into new objects
ext Expand external links into new objects
ref Copy objects that are pointed by references
noattr Copy object without copying attributes

12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

27. HDF and HDF-EOS Workshop X, Landover, MD
h5copy
For example
h5copy -i SDS.h5 -o SDS_cp.h5 -s /Floats/FloatArray -d /FloatArray / /
Floats
IntArray
FloatArray
FloatArray
SDS_cp.h5
SDS.h5
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

28. HDF and HDF-EOS Workshop X, Landover, MD
h5copy -f shallow /
floats
-f shallow
64-bit /
f32
floats
integers /
64-bit i1 i2
floats
f32 f1 f2
64-bit
f32 f1 f2
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

29. HDF and HDF-EOS Workshop X, Landover, MD
h5copy -f soft / / f1
dset_SL
/f1
-f soft f1
dset_SL
/f1 /
dset_SL
/f1
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

30. HDF and HDF-EOS Workshop X, Landover, MD
h5copy -f ref / d1 d2 /
dset_ref
679
1287
-f ref d2 d1 /
dset_ref
1895
763
dset_ref
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

31. HDF and HDF-EOS Workshop X, Landover, MD
h5copy todo
Fix references embedded in compound datatype
Follow external links
Test functionalities
Test performance
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

32. HDF and HDF-EOS Workshop X, Landover, MD
h5stat
Prints different statistics about HDF5 file
Helps
To troubleshoot size overhead in HDF5 files
To choose specific object’s properties and storage strategies
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

33. HDF and HDF-EOS Workshop X, Landover, MD
h5check
A validation tool that verifies if an HDF5 file is encoded according to the HDF5 File Format Specification
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

34. HDF and HDF-EOS Workshop X, Landover, MD
Why is it needed?
Verify if the file is compliant with the File Format to ensure the data model integrity and long term compatibility between evolving versions of the HDF5 library
As a verification tool required by the application of HDF5 File Format to be an ANSI standard
Serves as a watch dog that the HDF5 library implementation is compliant with the File Format
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

35. HDF and HDF-EOS Workshop X, Landover, MD
What does it do?
Given a file, it scans through the encoded content against the defined File Format
If it finds any non-compliance, it prints out the error and reason of non-compliance.
After finding any non-compliance, it tries to continue scanning the file if possible.
Eventually, it exits with non-zero.
If it does not find any non-compliance, it prints out an approval statement at the end and exits with zero.
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

36. How is it implemented? (1/2)
The tool is coded from scratch and does not use the formal HDF5 library API calls
It does not link with the HDF5 library at all
It may borrow coding, including algorithms or data structure from the HDF5 library source code but after close verification that they are in compliance with the File Format.
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

37. HDF and HDF-EOS Workshop X, Landover, MD
How is it implement? (2/2)
It links external libraries that HDF5 library uses. E.g.,
Zlib
szlib
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

38. HDF and HDF-EOS Workshop X, Landover, MD
How to use it?
H5check [-vn] <filename>
-vn verboseness mode
n=0 Terse—only prints if the file is compliant or not
n=1 Default—prints its progress and all errors found
n=2 Verbose—prints everything it knows, usually for debugging
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

39. Example: a compliant file
% h5check example1.h5
VALIDATING example1.h5
FOUND super block signature
VALIDATING the super block at 0...
VALIDATING the object header at
VALIDATING the btree at
FOUND btree signature.
VALIDATING the local heap at 96...
FOUND local heap signature. …
Result: File is in compliance.
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

40. Example: a non-compliant file
h5check invalid2.h5
FOUND super block signature
VALIDATING the super block at 0...
VALIDATING the object header at
VALIDATING the btree at
FOUND btree signature.
VALIDATING the SNOD at
FOUND SNOD signature.
VALIDATING the object header at
check_sym(at 1248): Errors from check_obj_header()
decode_validate_messages(): Failure in type->decode().
H5O_sdspace_decode(): Bad version number in simple dataspace message.
VALIDATING the local heap at 96...
FOUND local heap signature.
Main(): Errors from check_obj_header().
H5O_attr_decode(): Can't decode attribute dataspace. …
Result: File is not in compliance.
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

41. Implementation Status
All basic File Format components are implemented
Coding recognition of HDF5 files created by non-default Virtual File Driver such as the Multi-File format
Alpha release planned in December 2006
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD

42. HDF and HDF-EOS Workshop X, Landover, MD
h5ub
Combine nub, h5jam and h5unjam
nub -- NPOESS user block tool for HDF5 files (Richard)
Plan for h5ub development
3/22/07
3/25/07
2/23/07
3/5/07
4/2/07
Design
Implementation
Testing
Release
12/7/2018
HDF and HDF-EOS Workshop X, Landover, MD