May 30-31, 2012 HDF5 Workshop at PSI May Partial Edge Chunks Dana Robinson The HDF Group Efficient Use of HDF5 With High Data Rate X-Ray Detectors Paul Scherrer Institut

May 30-31, 2012 HDF5 Workshop at PSI Consider the following problem… Consider an extensible, filtered (compressed, etc.) dataset… DATASET Dataset Elements

May 30-31, 2012 HDF5 Workshop at PSI Consider the following problem… Which is chunked… DATASET Chunks

May 30-31, 2012 HDF5 Workshop at PSI Consider the following problem… Which will be 1) opened 2) extended 3) closed repeatedly. DATASET

May 30-31, 2012 HDF5 Workshop at PSI Problem: The chunk is filtered twice per extension DATASET 1)Read, Uncompress 2) Extend 3) Compress, Write

May 30-31, 2012 HDF5 Workshop at PSI Also, as the filtered/compressed size of the chunk changes, it will be relocated in the file. p0p0 p0p0 p1p1 p1p1 p0p0 p0p0 p1p1 p1p1 p0p0 p0p0 p2p2 p2p2 HDF5 File t0t0 t1t1 t2t2 holes

May 30-31, 2012 HDF5 Workshop at PSI Solution: Do not filter partial edge chunks DATASET Compressed Uncompressed U U C C When a chunk fills, it will automatically be compressed if filters are enabled. } }

May 30-31, 2012 HDF5 Workshop at PSI Solution: Do not filter partial edge chunks DATASET Compressed Uncompressed U U C C } } + Partial chunks are always the same size on the disk and do not move until full. + Less fragmentation. + No compression overhead on partial chunk I/O. - Possible size penalty for uncompressed edge data.

May 30-31, 2012 HDF5 Workshop at PSI Also consider the following problem… Consider a dataset… DATASET

May 30-31, 2012 HDF5 Workshop at PSI Also consider the following problem… Which is chunked… DATASET

May 30-31, 2012 HDF5 Workshop at PSI Also consider the following problem… Which is chunked… DATASET This space is empty and wasted

May 30-31, 2012 HDF5 Workshop at PSI Also consider the following problem… This space is allocated and exists on the disk. Compression can reduce but not eliminate the wasted space. Compression can have a performance penalty. Parallel HDF5 cannot use compression. DATASET

May 30-31, 2012 HDF5 Workshop at PSI Solution: Do not store the empty space DATASET This space is NOT stored

May 30-31, 2012 HDF5 Workshop at PSI Solution: Do not store the empty space DATASET + Saves space - Can result in file fragmentation if the dataset is later extended. This space is NOT stored

May 30-31, 2012 HDF5 Workshop at PSI Two New API Functions herr_t H5Pset_edge_chunk_opts(hid_t dcpl_id, unsigned opts); herr_t H5Pget_edge_chunk_opts(hid_t dcpl_id, unsigned *opts); OPTIONS H5D_STORE_PARTIAL_CHUNKS (default = disabled) H5D_DONT_FILTER_PARTIAL_CHUNKS (default = disabled)

May 30-31, 2012 HDF5 Workshop at PSI Final Notes This work requires a file format change, so it cannot appear in HDF5 1.8.x. Older versions of the library will not understand either of these options. Enabling/disabling filters on edge chunks should appear in HDF Partial storage of partial edge chunks has not been implemented and is currently unfunded.