May 30-31, 2012 HDF5 Workshop at PSI May The HDF5 Virtual File Layer (VFL) and Virtual File Drivers (VFDs) Dana Robinson The HDF Group Efficient Use of HDF5 With High Data Rate X-Ray Detectors Paul Scherrer Institut

May 30-31, 2012 HDF5 Workshop at PSI HDF5 API Virtual File Layer VFD User-Supplied VFD disk HDF5 Library

May 30-31, 2012 HDF5 Workshop at PSI disk HDF5 API Virtual File Layer Virtual File Driver H5FDwrite() H5FD_sec2_write() function pointer internal generic I/O call VFD-specific I/O call

May 30-31, 2012 HDF5 Workshop at PSI HDF5 VFDs sec2 1 (default) Windows 2 STDIO 3 core (in-memory) 1) Uses POSIX I/O (sec2 = "POSIX section 2") 2) Currently a wrapper for SEC2. There is no driver which uses Win32 API calls. 3) "How to write a VFD" demo driver. Not intended for production use. log split multi family "logical" VFDs which perform no I/O themselves MPI MPI-POSIX

May 30-31, 2012 HDF5 Workshop at PSI memory The core VFD allows you to create/open HDF5 files in memory. disk Core VFD H5FD_core_write() HDF5 API H5FDwrite() Virtual File Layer File is (optionally) written to disk on close

May 30-31, 2012 HDF5 Workshop at PSI The family VFD allows you to split a logical HDF5 file among many smaller physical files. disk Family VFD H5FD_family_write() file B file A file C file Z … … HDF5 API H5FDwrite() Virtual File Layer

May 30-31, 2012 HDF5 Workshop at PSI disk 1 disk 2 Multi VFD H5FD_multi_write() file B file A file C raw data superblock global and local heap data, B-trees, object headers The multi (and split) VFD allows you to direct various categories of HDF5 data to different files and disks.

May 30-31, 2012 HDF5 Workshop at PSI Selecting a "Terminal" VFD 1) Create the property list. hid_t fapl_id = H5Pcreate(H5P_FILE_ACCESS); 2) Set the VFD using the appropriate API calls. herr_t err = H5Pset_fapl_sec2(fapl_id); 3) Create/open your file. hid_t fid = H5Fcreate("foo.h5", 0, H5P_DEFAULT, fapl_id); A "terminal" VFD does not require a second VFD

May 30-31, 2012 HDF5 Workshop at PSI Selecting a "Logical" VFD Step 1 Create a file access property list (FAPL) and set the VFD to use the terminal VFD. Step 2 Create a second FAPL and set the VFD to use the logical VFD (passing in the first VFD). Step 3 Open your file using the second (logical) FAPL. A "logical" VFD requires a second, underlying VFD Logical VFD Disk Terminal VFD

May 30-31, 2012 HDF5 Workshop at PSI 1) Create the first property list. hid_t fapl_id = H5Pcreate(H5P_FILE_ACCESS); 2) Set the terminal VFD using the appropriate API calls. herr_t err = H5Pset_fapl_sec2(terminal); 3) Create the second file create/access property list. hid_t fapl_id = H5Pcreate(H5P_FILE_ACCESS); 4) Set the logical VFD using the appropriate API calls. herr_t err = H5Pset_fapl_family(logical, SIZE, terminal); 5) Create/open your file. hid_t fid = H5Fcreate("foo.h5", 0, H5P_DEFAULT, logical);

May 30-31, 2012 HDF5 Workshop at PSI Stackable VFDs Virtual File Layer Non-Terminal VFD Disk Non-Terminal VFD Terminal VFD