HDF5 Q4 Demo

Architecture Friday, May 10, 2013 Friday Seminar2

Full VOL Plugin Support Skeletal IOD – Artificial Data Read at server New Features/Functionality: – Attributes (H5A*) – Links (H5L*) – Objects (H5O*) Datatype Conversion: – Done at server, avoids extra buffer allocation at client – Examples (BE to LE, 16 to 32 bit, etc…) Non-contiguous memory selections: – Hyperslabs, point selections – Now supported with Mercury non-contiguous transfers

Asynchronous Execution Nearly all HDF5 API routines are completely asynchronous now – Dependencies built at the client, executed at the server – No wait at client for previous operations to complete to be able to forward a child operation Few API routines still cause partly asynchronous behavior – Child operations need to wait for them to complete at the client. A small number of routines have to be synchronous: – Iterate/visit – H5Oopen Using event queue (EQ) objects now to manage requests instead of individual requests: – Very cumbersome to manage a request for every operation issued. – Event Queue can be though of as “bags” to wait/test a bunch of requests. – There is no effect on dependencies or influence on execution order for requests in an EQ. – User can still pop off the last inserted request in EQ to manage it separately.

Unimplemented Functionality Deferred – Iterate/visit (Does not make sense without a storage backend – Too much work to generate artificial metadata for them) – A few metadata query (H5*get_*) routines (same reason) – Variable Length Datatypes Probably or Definitely will not support: – External Links - probably – References (H5R*) – probably – H5*_by_index operations – probably – File Mounting – definitely – H5Oopen_by_addr – definitely – Some Query operations (H5Dget_offset) – definitely

Testing Status Testing is improved: – Verifying new VOL routines are shipped and call the correct IOD routines at server. – Regression testing framework is setup (Cmake only now, autotools later). – Need to beef up regression testing, but won’t be able to do much until we get something for a storage backend. Temporary Native HDF5 backend: – Call in to the native HDF5 storage format to verify routines and their parameters are shipped from the client and executed correctly at the server. – Very limited: 1 client & 1 server possible for this now Not asynchronous

Data Integrity Support in our level of the stack. Added a public routine for applications to generate checksums: H5checksum(). Added properties and routines to set properties for data integrity: herr_t H5Pset_dxpl_checksum(hid_t dxpl_id, uint32_t value); herr_t H5Pset_dxpl_checksum_ptr(hid_t dxpl_id, uint32_t *value); herr_t H5Pset_dxpl_inject_corruption(hid_t dxpl_id, hbool_t flag);

ACG Dynamic Data Structures Added 4 new routines and their FF version: herr_t H5DOappend(hid_t dataset_id, hid_t dxpl_id, unsigned axis, size_t extension, hid_t memtype, const void *buffer); herr_t H5DOsequence(hid_t dataset_id, hid_t dxpl_id, unsigned axis, hsize_t start, size_t sequence, hid_t memtype, void *buffer); herr_t H5DOset(hid_t dataset_id, hid_t dxpl_id, const hsize_t coord[], hid_t memtype, const void *buffer); herr_t H5DOget(hid_t dataset_id, hid_t dxpl_id, const hsize_t coord[],hid_t memtype, void *buffer);

H5DOappend() 3x5 Dataset 1.H5DOappend(did, dxpl, 0, 3, memtype, buf); 2.H5DOappend(did, dxpl, 1, 2, memtype, buf2); After 1: 6x5 DatasetAfter 2: 6x7 Dataset H5DOsequence is very similar: Read instead of write (no extension). Accepts a starting position to read from.

ACG support for Next Quarter Support for VL Datatypes: – Each array element in the dataset corresponds to an IOD BLOB object Add an Append only hint to the Dataset Creation property list: – Optimizes access and storage – Only 1 BLOB object needed – Each array element stores an offset into the BLOB object Map Objects: – KV stores at the HDF5 API level.