Moving applications to HDF5 1.8.0 12/25/2018 HDF and HDF-EOS Workshop XI

HDF and HDF-EOS Workshop XI Outline Quick overview of new features Move or not to move? HDF5 APIs versioning HDF5 library configuration HDF5 application configuration Examples 12/25/2018 HDF and HDF-EOS Workshop XI

HDF and HDF-EOS Workshop XI Goal of the tutorial Raise awareness about forward/backward compatibility issues with the new release Help with transition to the new release Get feedback from you before official release is done 12/25/2018 HDF and HDF-EOS Workshop XI

Current status of the HDF5 1.8.0 release HDF5 1.8.0-beta4 was released on October 22, 2007 Stable file format and APIs Has been successfully tested with NetCDF-4 and HDF-EOS5 on many supported platforms Tasks left before the full release Bugs fixing Incorporating contributed features Performance tuning (more will follow) Documentation Thorough backward/forward compatibility testing for APIs and files format 12/25/2018 HDF and HDF-EOS Workshop XI

HDF and HDF-EOS Workshop XI Quick overview of new features in HDF5 1.8.0 and their affect on current applications 12/25/2018 HDF and HDF-EOS Workshop XI

HDF and HDF-EOS Workshop XI HDF5 1.8.0 themes Better handling of HDF5 metadata Reduced HDF5 file overhead Efficient metadata caching Improved internal structures for better handling of variable-length data Features to support HDF and NetCDF harmonization New and improved tools Numerous bug fixes and improvements Result: More than 200 new APIs and revised file format 12/25/2018 HDF and HDF-EOS Workshop XI

Quick overview of the new features Complete list can be found at http://www.hdfgroup.uiuc.edu/HDF5/doc_1.8pre/WhatsNew180.html Two types of features Features that did not require file format and APIs changes Features that did require file format changes 12/25/2018 HDF and HDF-EOS Workshop XI

Quick overview of the new features Features that did not require file format changes and changes to existing APIs Improved metadata caching New filters (scale+offset, n-bit, arithmetic transformation) Improved error handling mechanism Support for dimension scales, packet tables and other high-level APIs Affect on applications: Applications can be extended (calls added) to use new features There is no need to modify existing calls Created files are compatible with the earlier versions of software 12/25/2018 HDF and HDF-EOS Workshop XI

Quick overview of the new features Features that did require file format changes and changes to existing APIs Creation order of the objects (groups, datasets, attributes) Support for UTF-8 encoding for strings that describe links to the objects and names of the attributes Configurable compact or indexed storage for groups Shared object headers, external links New file format features to reduce files overhead Affect on applications: Applications have to be modified to use new features Created files are NOT compatible with the earlier versions of software 12/25/2018 HDF and HDF-EOS Workshop XI

HDF and HDF-EOS Workshop XI To move or not to move? 12/25/2018 HDF and HDF-EOS Workshop XI

Why applications should move to HDF5 1.8.0? My application doesn’t need any new features, why should I consider moving to HDF5 1.8.0? Take advantage of improved library Bug fixes (e.g. corrupted files) Better metadata cache performance Smaller memory footprint Support for big attributes, new compression filters, datatype conversion, etc. Availability of new platforms like Windows 64-bit Better support for different Fortran compilers Improved tools First step in taking advantage of the new features in the future 12/25/2018 HDF and HDF-EOS Workshop XI

Why applications may not move to HDF5 1.8.0? May be expensive Moving to the new release may require Source code modification Building scripts modification No new features are desired 12/25/2018 HDF and HDF-EOS Workshop XI

HDF and HDF-EOS Workshop XI Making decision…. I would like to use new features, how much work it would be? I would like to use new library but I cannot modify my application, can I do it? The rest of the talk is about the ways to move applications to the new release without doing much work 12/25/2018 HDF and HDF-EOS Workshop XI

Introduction to HDF5 APIs versioning 12/25/2018 HDF and HDF-EOS Workshop XI

Example: Four ways to create a group Old way H5Gcreate ( loc_id, “New/My old group”, 0 ) H5Gcreate1( loc_id, “New/My old group”, 0 ) No new features can be invoked New way H5Gcreate ( loc_id, “New/My new group”, lcpl_id, gcpl_id, gapl_id) H5Gcreate2( loc_id, “New/My new group”, lcpl_id, gcpl_id, gapl_id) New features can be invoked Creation order Unicode names Compact storage Intermediate group creation 12/25/2018 HDF and HDF-EOS Workshop XI

HDF and HDF-EOS Workshop XI New APIs in HDF5 1.8.0 Examples of the new APIs H5Acreate1, H5Aopen1 H5Acreate2, H5Aopen2 H5Gcreate1, H5Gopen1 H5Gcreate2, H5Gopen2, H5G_link_hard H5Rget_obj_type1 H5Rget_obj_type2 New APIs have more parameters to set up creation and access properties for the objects Default values for new parameters H5P_DEFAULT will emulate old behavior H5Gcreate ( loc_id, “New/My old group”, 0 ) H5Gcreate ( loc_id, “New/My new group”, H5P_DEFAULT,H5P_DEFAULT, H5P_DEFAULT) 12/25/2018 HDF and HDF-EOS Workshop XI

HDF and HDF-EOS Workshop XI Removed APIs in HDF5 1.8.0 Examples of the removed APIs H5Acreate, H5Aopen H5Gcreate, H5Gopen H5Rget_obj_type Still can be called by using API mapping and macros H5Gcreate ( loc_id, “New/My old group”, 0 ) 12/25/2018 HDF and HDF-EOS Workshop XI

HDF and HDF-EOS Workshop XI Deprecated APIs in HDF5 1.8.0 Include both public and private APIs Functionality is replaced by new functions See files in the src directory H5Adeprec.c H5Ddeprec.c H5Edeprec.c H5Gdeprec.c H5Rdeprec.c H5Tdeprec.c Examples of deprecated APIs H5Acreate1, H5Aopen_name H5Gcreate1, H5Gopen1, H5G_link_hard H5Rget_obj_type1 For complete list see Changes.html file, “API Compatibility Macros in HDF5” Deprecated APIs will be removed with caution (in some cases will not be removed at all) 12/25/2018 HDF and HDF-EOS Workshop XI

HDF5 Macros for API compatibility Name: H5Gcreate Signature: hid_t H5Gcreate( hid_t loc_id, const char *name, size_t size_hint ) hid_t H5Gcreate( hid_t loc_id, const char *name, hid_t lcpl_id, hid_t gcpl_id, hid_t gapl_id Purpose: Creates a new empty group and links it to a location in the file. Description: H5Gcreate is a macro that is mapped to either H5Gcreate1 or H5Gcreate2, depending on the needs of the application. 12/25/2018 HDF and HDF-EOS Workshop XI

HDF5 Macros for API compatibility Defined in the H5version.h file Example of the H5Gcreate macro #ifdef H5_USE_16_API … #if !defined(H5Gcreate_vers) #define H5Gcreate_vers 1 #endif /* !defined(H5Gcreate_vers) */ #if !defined(H5Gcreate_vers) || H5Gcreate_vers == 2 #ifndef H5Gcreate_vers #define H5Gcreate_vers 2 #endif /* H5Gcreate_vers */ #define H5Gcreate H5Gcreate2 #elif H5Gcreate_vers == 1 #define H5Gcreate H5Gcreate1 #else /* H5Gcreate_vers */ #error "H5Gcreate_vers set to invalid value" 12/25/2018 HDF and HDF-EOS Workshop XI

HDF5 Library configuration Mapping of APIs is set at configuration time Several configuration flags are provided to enable different mappings Macro name is mapped to old API, e.g. H5Gcreate is mapped to H5Gcreate1 Macro name is mapped to new API, e.g. H5Gcreate is mapped to H5Gcreate2 (will have different parameters from 1.6 H5Gcreate) Disable old APIs completely, e.g. H5Gcreate1 is not available in the library 12/25/2018 HDF and HDF-EOS Workshop XI

HDF5 Library configuration Configure flag (global settings) Public symbols are mapped to Are deprecated symbols available? (default) --with-default-api-version=v18 1.8 (e.g. H5Gcreate is mapped to H5Gcreate2, old H5Gcreate is H5Greate1) Yes --disable-deprecated-symbols (e.g. H5Gcreate is mapped to H5Gcreate2, H5Gcreate1 is not available) No --with-default-api-version=v16 1.6 (e.g. H5Gcreate is mapped to H5Gcreate1, H5Gcreate2 is available) 12/25/2018 HDF and HDF-EOS Workshop XI

HDF5 Library configuration Library configuration information is available in the libhdf5.settings file Examples ……. Default Version of Public Symbols: v18 With Deprecated Public Symbols: Yes Default Version of Public Symbols: v16 With Deprecated Public Symbols: No 12/25/2018 HDF and HDF-EOS Workshop XI

HDF5 Application configuration options Global mapping HDF5 APIs mapping can be done by application Assuming both deprecated and new symbols are available in the library: h5cc my_program.c Both H5Gcreate1, H5Gcreate2 and H5Gcreate may be used h5cc -DH5_NO_DEPRECATED_SYMBOLS my_program.c Only new symbols are available for application; H5Gcreate is mapped to H5Gcreate2; application may use both H5Gcreate2 and H5Gcreate; cannot use H5Gcreate1 h5cc -DH5_USE_16_API my_program.c H5Gcreate is mapped to H5Gcreate1; all three H5Gcreate1, H5Gcreate2 and H5Gcreate can be used 12/25/2018 HDF and HDF-EOS Workshop XI

HDF5 Application configuration options Per-function scale Version and mapping can be set per function Assuming both deprecated and new symbols are available in the library: h5cc -D H5Gcreate_vers=1 -D H5Acreate_vers=2 my_program.c Maps H5Gcreate to H5Gcreate1 Maps H5Acreate to H5Acreate2 both H5Gcreate1 and H5Gcreate2 may be used; the same for H5Acreate1 and H5Acreate2 h5cc -D H5Gcreate_vers=2 my_program.c Maps H5Gcreate to H5Gcreate2 Both H5Gcreate1 and H5Gcreate2 may be used 12/25/2018 HDF and HDF-EOS Workshop XI

HDF5 Fortran and C++ applications HDF5 Fortran and C++ libraries use new APIs H5<name>2 using default parameters for new arguments Backward/forward compatible New features are not available yet 12/25/2018 HDF and HDF-EOS Workshop XI

Example: --with-default-api-version=v18 hid_t file_id, group_id; /* identifiers */ ... /* Open “file.h5” */ file_id = H5Fopen(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, H5P_DEFAULT); /* Create several groups in a file */ grp1_id = H5Gcreate (file_id, ”New/larry", H5P_DEAFULT, gcpt, gapt); grp2_id = H5Gcreate1(file_id,"/larry",0); … grp3_id = H5Gcreate2(file_id,”New/larry", 12/25/2018 HDF and HDF-EOS Workshop XI

Example: --with-default-api-version=v16 hid_t file_id, group_id; /* identifiers */ ... /* Open “file.h5” */ file_id = H5Fopen(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, H5P_DEFAULT); /* Create several groups in a file */ grp1_id = H5Gcreate (file_id, "/larry",0); grp2_id = H5Gcreate1(file_id,"/larry1",0); grp3_id = H5Gcreate2(file_id,”New/larry", H5P_DEAFULT, gcpt, gapt); 12/25/2018 HDF and HDF-EOS Workshop XI

Example: --disable-deprecated-symbols hid_t file_id, group_id; /* identifiers */ ... /* Open “file.h5” */ file_id = H5Fopen(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, H5P_DEFAULT); /* Create several groups in a file */ grp1_id = H5Gcreate (file_id, ”New/larry", H5P_DEAFULT, gcpt, gapt); /* Compilation will fail */ grp2_id = H5Gcreate1(file_id,"/larry",0); grp3_id = H5Gcreate2(file_id,”New/larry", 12/25/2018 HDF and HDF-EOS Workshop XI

Example: new version of the file format hid_t file_id, group_id; /* identifiers */ ... /* Open “file.h5” and create objects using new format*/ fapl = H5Pcreate(H5P_FILE_ACCESS); ret = H5Pset_latest_format(fapl, 1); fid = H5Fcreate(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, fapl); /* Compose group creation property list */ gcpl = H5Pcreate(H5P_GROUP_CREATE); ret = H5Pset_link_creation_order(gcpl, (H5P_CRT_ORDER_TRACKED | H5P_CRT_ORDER_INDEXED)); /* Create a group */ gid = H5Gcreate(fid, GNAME, H5P_DEFAULT, gcpl, H5P_DEFAULT); 12/25/2018 HDF and HDF-EOS Workshop XI

Example: h5dump --group=1 tordergr.h5 HDF5 "tordergr.h5" { GROUP "1" { GROUP "a" { GROUP "a1" { } GROUP "a2" { GROUP "a21" { GROUP "a22" { GROUP "b" { GROUP "c" { 12/25/2018 HDF and HDF-EOS Workshop XI

Example: h5dump … --sort_by=creation_order … tordergr.h5 HDF5 "tordergr.h5" { GROUP "1" { GROUP "c" { } GROUP "b" { GROUP "a" { GROUP "a1" { GROUP "a2" { GROUP "a22" { GROUP "a21" { 12/25/2018 HDF and HDF-EOS Workshop XI

Example: creation of intermediate groups /* Create link creation property list */ lcr_id = H5Pcreate(H5P_LINK_CREATE); /* Set flag for intermediate group creation */ status = H5Pset_create_intermediate_group(lcr_id, TRUE); grp_id = H5Gcreate (file_id, "/A/B/C", lcr_id, H5P_DEFAULT, access_id); Groups A and B will be created automatically 12/25/2018 HDF and HDF-EOS Workshop XI

HDF and HDF-EOS Workshop XI Thank you! Questions ? 12/25/2018 HDF and HDF-EOS Workshop XI