Merger/Extract HDF5 Objects Peter Cao & Quincey Koziol June 16, 2005

2/12 Introduction copy a whole file into another file / /g1 /d1 /g1/d2 / /g2 /d3 /g2/d4 / /g2 /d3 /g2/d4 /g1 /d1 /g1/d2

3/12 Introduction copy a single object to another location / /g1 /d1 /g1/d2 / /g2 /d3 /g2/d4 /g1/d3

4/12 Introduction extract a single object into a new file / /g2 /d3 /g2/d4 / /g2 /g2/d4

5/12 Use Cases Time serial data Source variable data Large data in size Large number of objects HDFView and other applications Repack unused space Client and server application

6/12 Requirements Efficient low level implementation Simple and flexible high level API

7/12 Flags applying to copy H5FCOPY_EXPAND_FULL_TREE Recursively copy all objects below the group (full hierarchy) or only immediate members H5FCOPY_EXPAND_SOFT_LINK Keep link as it is or expend them into new objects H5FCOPY_EXPAND_EXTERNAL_LINK Keep external links as it is or expand them into new objects H5FCOPY_EXPAND_OBJECT_REFERENCE Leave object reference as it is or copy objects that pointed by the reference

8/12 Properties applying to copy Change dataset storage properties Apply data value conversion as data is copied Change datatype as data is copied

9/12 High Level APIs herr_t H5Fcopy ( hid_t id_src, hid_t id_dst, int h5fcopy_flag, hid_t plist_id); herr_t H5Gcopy ( hid_t id_src, hid_t loc_dst, const char *name_dst, int h5fcopy_flag, hid_t plist_id);

10/12 Low Level Implementation H5O_copy_header { loop_on_object_header_chunks { loop_on_messages { callback function for [ H5Oattr, H5Odtype, H5Osdspace, H5Ostab, H5Ocont, H5Oefl, H5Olayout, H5Oname, H5Opline, H5Oshared ] } } } Prefix Message #0 …. Continue … Prefix Message #0 …. storage layout continued chunk

11/12 Implementation Steps Named Datatypes without continued messages with continued messages Datasets with non-heap datatypes with compack/ contiguous storagee with chunks Groups empty Group with objects Datasets with heap datatypes

12/12 Question / Comments? More softlink options: (0- don’t copy, 1 – don’t extand, 2—extand) What to do with datasets with split files How to add properties to the new file What to do with objects have the same name Drop property list option in copy. It is too hard to implement and will destory the performance gain. May use command line tool such as repack to do the work Build user models (cases) based on the development steps Need design/requirement documentation