1. www.hdfgroup.org The HDF Group November 3-5, 2009HDF/HDF-EOS Workshop XIII1 HDF5 Advanced Topics Elena Pourmal The HDF Group The 13 th HDF and HDF-EOS Workshop November 3-5, 2009

2. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII2 Outline HDF5 Datatypes Partial I/O

3. www.hdfgroup.org The HDF Group November 3-5, 2009HDF/HDF-EOS Workshop XIII3 HDF5 Datatypes Overview

4. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII4 HDF5 Datatypes An HDF5 datatype Required description of a data element the set of possible values it can have Example: enumeration datatype the operations that can be performed Example: type conversion cannot be performed on opaque datatype how the values of that type are stored Example: values of variable-length type are stored in a heap in a file Stored in the file along with the data it describes Not to be confused with the C, C++, Java and Fortran types

5. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII5 HDF5 Datatypes Examples We provide examples how to create, write and read data of different types http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/api18-c.html

6. www.hdfgroup.org November 3-5, 2009HDF/HDF-EOS Workshop XIII6 HDF5 Datatypes Examples

7. www.hdfgroup.org November 3-5, 2009HDF/HDF-EOS Workshop XIII7 HDF5 Datatypes When HDF5 Datatypes are used? To describe application data in a file H5Dcreate, H5Acreate calls Example: A C application stores integer data as 32-bit little- endian signed two’s complement integer; it uses H5T_SDT_I32LE with the H5Dcreate call A C applications stores double precision data “as is” in the application memory; it uses H5T_NATIVE_DOUBLE with the H5Acreate call A Fortran application stores array of real numbers as 64-bit big-endian IEEE format; it uses H5T_IEEE_F64BE with h5dcreate_f call; HDF5 library will perform all necessary conversions

8. www.hdfgroup.org November 3-5, 2009HDF/HDF-EOS Workshop XIII8 HDF5 Datatypes When HDF5 Datatypes are used? To describe application data in memory Data buffer to be written or read into with H5Dwrite/ H5Dread and H5Awrite/H5Aread calls Example: C application reads data from the file and stores it in an integer buffer; it uses H5T_NATIVE_INT to describe the buffer. A Fortran application reads floating point data from the file and stores it an integer buffer; it uses H5T_NATIVE_INTEGER to describe the buffer; HDF5 library performs datatype conversion; overflow/underflow may occur.

9. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII9 Example C Array of integers on Linux platform Native integer (H5T_NATIVE_INT) is little-endian, 4 bytes H5T_SDT_I32LE H5Dwrite No conversion Fortran Array of integers on AIX platform Native integer (H5T_NATIVE_INTEGER) is big-endian, 8 bytes H5T_NATIVE_INT H5T_NATIVE_INTEGER H5Dread Conversion HDF5 File Data is stored as little-endian, converted to big-endian on read

10. www.hdfgroup.org November 3-5, 2009HDF/HDF-EOS Workshop XIII10 HDF5 Datatypes

11. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII11 Example: Writing/reading an Array to HDF5 file Calls you’ve already seen in Intro Tutorial H5LTmake_dataset H5Dwrite, H5Dread APIs to handle specific C data type H5LTmake_dataset_ is one of “char”, “short”, “int”, “long”, “float”, “double”, “string” All data array is written (no sub-setting) Data stored in a file as it is in memory H5LTread_dataset, H5LTread_dataset_

12. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII12 Example: Read data into array of longs #include "hdf5.h” #include "hdf5_hl.h” int main( void ){ long *data; … /* Open file from ex_lite1.c */ file_id = H5Fopen ("ex_lite1.h5", H5F_ACC_RDONLY, H5P_DEFAULT); /* Get information about dimensions to allocate memory buffer */ status = H5LTget_dataset_ndims(file_id,"/dset",rank); status = H5LTget_dataset_info(file_id,"/dset",dims,dt_class,dt_size); /* Allocate buffer to read data in */ data = (long*)malloc(… /* Read dataset */ status = H5LTread_dataset_long(file_id,"/dset",data); /

13. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII13 Example: Read data into array of longs #include hdf5.h … long *rdata; …. /* Open file and dataset. */ file_id = H5Fopen (“ex_lite1.h5”, H5F_ACC_RDONLY, H5P_DEFAULT); dset_id = H5Dopen (file, “/dset”, H5P_DEFAULT); /* Get information about dimensions to allocate memory buffer */ space = H5Dget_space (dset); rank = H5Sget_simple_extent_dims (space, dims, NULL); status = H5Dread (dset, H5T_NATIVE_LONG, H5S_ALL, H5S_ALL, H5P_DEFAULT, rdata);

14. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII14 Basic Atomic HDF5 Datatypes

15. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII15 Basic Atomic Datatypes Integers & floats Strings (fixed and variable size) Pointers - references to objects and dataset regions Bitfield Opaque

16. www.hdfgroup.orgNovember 3-5, 2009 HDF/HDF-EOS Workshop XIII 16 HDF5 Predefined Datatypes HDF5 Library provides predefined datatypes (symbols) for all basic atomic datatypes except opaque datatype H5T_ _ Examples: H5T_IEEE_F64LE H5T_STD_I32BE H5T_C_S1, H5T_FORTRAN_S1 H5T_STD_B32LE H5T_STD_REF_OBJ, H5T_STD_REF_DSETREG H5T_NATIVE_INT Predefined datatypes do not have constant values; initialized when library is initialized

17. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII17 HDF5 Pre-defined Datatypes

18. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII18 HDF5 Predefined Datatypes

19. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII19 HDF5 integer datatype HDF5 supports 1,2,4,and 8 byte signed and unsigned integers in memory and in the file Support differs by language C language All C integer types including C99 extended integer types (when available) Examples: H5T_NATIVE_INT16 for int16_t H5T_NATIVE_INT_LEAST64 for int_least64_t H5T_NATIVE_UINT_FAST16 for uint_fast16_t

20. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII20 HDF5 integer datatype Fortran language In memory supports only Fortran “integer” Examples: H5T_NATIVE_INTEGER for integer In the file supports all HDF5 integer types Example: one-byte integer has to be represented by integer in memory; can be stored as one-byte integer by creating an appropriate dataset (H5T_SDT_I8LE) Next major release of HDF5 will support ANY kinds of Fortran integers

21. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII 21 HDF5 floating-point datatype HDF5 supports 32 and 64-bit floating point IEEE big-endian, little-endian types in memory and in the file Support differs by language C languge H5T_IEEE_F64BE and H5T_IEEE_F32LE H5T_NATIVE_FLOAT H5T_NATIVE_DOUBLE H5T_NATIVE_LDOUBLE

22. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII22 HDF5 floating-point datatype Fortran language In memory supports only Fortran “real” and “double precision” (obsolete) Examples: H5T_NATIVE_REAL for real H5T_NATIVE_DOUBLE for double precision In the file supports all HDF5 floating-point types Next major release of HDF5 will support ANY kinds of Fortran reals

23. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII23 HDF5 string datatype HDF5 strings are characterized by The way each element of a string type is stored in a file NULL terminated (C type string) char *mystring[]=“Once upon a time”; HDF5 stores Space padded (Fortran string) character(len=16) :: mystring=‘Once upon a time’ HDF5 stores and adds spaces if required The sizes of elements in the same dataset or attribute Fixed-length string Variable-length string

24. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII24 Example: Creating fixed-length string C Example: “Once upon a time” has 16- characters string_id = H5Tcopy(H5T_C_S1); H5Tset_size(string_id, size); Size value have to include accommodate “/0”, i.e., size=17 for “Once upon a time” string Overhead for short strings, e.g., “Once” will have extra 13 bytes allocated for storage Compressed well

25. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII25 Example: Creating variable-length string C example: string_id = H5Tcopy(H5T_C_S1); H5Tset_size(string_id, H5T_VARIABLE); Overhead to store and access data Cannot be compressed (may be in the future)

26. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII26 Reference Datatype Reference to an HDF5 object Pointer to a group or a dataset in a file Predefined datatype H5T_STD_REG_OBJ describe object references

27. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII27 Reference to Object Group1 Integers MyType Group2 ref_obj.h5 / Object References

28. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII28 Reference to Object h5dump –d /”object_reference” ref_obj.h5 DATASET "OBJECT_REFERENCES" { DATATYPE H5T_REFERENCE DATASPACE SIMPLE { ( 4 ) / ( 4 ) } DATA { (0): GROUP 808 /GROUP1, GROUP 1848 /GROUP1/GROUP2, (2): DATASET 2808 /INTEGERS, DATATYPE 3352 /MYTYPE }

29. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII29 Reference to Object Create a reference to group object H5Rcreate(&ref[1], fileid, "/GROUP1/GROUP2", H5R_OBJECT, -1); Write references to a dataset H5Dwrite(dsetr_id, H5T_STD_REF_OBJ, H5S_ALL, H5S_ALL, H5P_DEFAULT, ref); Read reference back with H5Dread and find an object it points to type_id = H5Rdereference(dsetr_id, H5R_OBJECT, &ref[3]); name_size = H5Rget_name(dsetr_id, H5R_OBJECT, &ref_out[3], (char*)buf, 10);

30. www.hdfgroup.org Need to select and access the same elements of a dataset Saving Selected Region in a File November 3-5, 200930HDF/HDF-EOS Workshop XIII

31. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII31 Reference Datatype Reference to a dataset region (or to selection) Pointer to the dataspace selection Predefined datatype H5T_STD_REF_DSETREG to describe regions

32. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII32 Reference to Dataset Region REF_REG.h5 Root Region ReferencesMatrix 1 1 2 3 3 4 5 5 6 1 2 2 3 4 4 5 6 6

33. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII33 Reference to Dataset Region Example dsetr_id = H5Dcreate(file_id, “REGION REFERENCES”, H5T_STD_REF_DSETREG, …); H5Sselect_hyperslab(space_id, H5S_SELECT_SET, start, NULL, …); H5Rcreate(&ref[0], file_id, “MATRIX”, H5R_DATASET_REGION, space_id); H5Dwrite(dsetr_id, H5T_STD_REF_DSETREG, H5S_ALL, H5S_ALL, H5P_DEFAULT,ref);

34. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII34 Reference to Dataset Region HDF5 "REF_REG.h5" { GROUP "/" { DATASET "MATRIX" { …… } DATASET "REGION_REFERENCES" { DATATYPE H5T_REFERENCE DATASPACE SIMPLE { ( 2 ) / ( 2 ) } DATA { (0): DATASET /MATRIX {(0,3)-(1,5)}, (1): DATASET /MATRIX {(0,0), (1,6), (0,8)} }

35. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII35 Bitfield datatype C bitfield Bitfield – sequence of bytes packed in some integer type Examples of Predefined Datatypes H5T_NATIVE_B64 – native 8 byte bitfield H5T_STD_B32LE – standard 4 bytes bitfield Created by copying predefined bitfield type and setting precision, offset and padding Use n-bit filter to store significant bits only

36. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII36 Bitfield datatype 0 001011100111000 0 715 Example: LE 0-padding Offset 3 Precision 11

37. www.hdfgroup.org The HDF Group November 3-5, 2009HDF/HDF-EOS Workshop XIII37 Storing Variable Length Data in HDF5

38. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII38 Data Time Data Time HDF5 Fixed and Variable Length Array Storage Region references are represented as VL data when stored in HDF5

39. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII39 Storing Variable Length Data in HDF5 Each element is represented by C structure typedef struct { size_t length; void *p; } hvl_t; Base type can be any HDF5 type H5Tvlen_create(base_type)

40. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII40 Data Example hvl_t data[LENGTH]; for(i=0; i<LENGTH; i++) { data[i].p=malloc((i+1)*sizeof(unsigned int)); data[i].len=i+1; } tvl = H5Tvlen_create (H5T_NATIVE_UINT); data[0].p data[4].len

41. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII41 Reading HDF5 Variable Length Array hvl_t rdata[LENGTH]; /* Create the memory vlen type */ tvl = H5Tvlen_create (H5T_NATIVE_UINT); ret = H5Dread(dataset,tvl,H5S_ALL,H5S_ALL, H5P_DEFAULT, rdata); /* Reclaim the read VL data */ H5Dvlen_reclaim(tvl,H5S_ALL,H5P_DEFAULT,rdata ); On read HDF5 Library allocates memory to read data in, application only needs to allocate array of hvl_t elements (pointers and lengths).

42. www.hdfgroup.org The HDF Group November 3-5, 2009HDF/HDF-EOS Workshop XIII42 Storing Tables in HDF5 file

43. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII43 Example Time (integer) Pressure (float) Temp (double) 00.1.0000 11.0.5000 24.0.3333 39.0.2500 416.0.2000 525.0.1667 636.0.1429 749.0.1250 864.0.1111 981.0.1000 Multiple ways to store a table Dataset for each field Dataset with compound datatype If all fields have the same type: 2-dim array 1-dim array of array datatype continued….. Choose to achieve your goal! How much overhead each type of storage will create? Do I always read all fields? Do I need to read some fields more often? Do I want to use compression? Do I want to access some records?

44. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII44 HDF5 Compound Datatypes Compound types Comparable to C structs Members can be atomic or compound types Members can be multidimensional Can be written/read by a field or set of fields Not all data filters can be applied (shuffling, SZIP)

45. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII45 HDF5 Compound Datatypes Which APIs to use? H5TB APIs Create, read, get info and merge tables Add, delete, and append records Insert and delete fields Limited control over table’s properties (i.e. only GZIP compression, level 6, default allocation time for table, extendible, etc.) PyTables http://www.pytables.orghttp://www.pytables.org Based on H5TB Python interface Indexing capabilities HDF5 APIs H5Tcreate(H5T_COMPOUND), H5Tinsert calls to create a compound datatype H5Dcreate, etc. See H5Tget_member* functions for discovering properties of the HDF5 compound datatype

46. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII46 Creating and Writing Compound Dataset h5_compound.c example typedef struct s1_t { int a; float b; double c; } s1_t; s1_t s1[LENGTH];

47. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII47 Creating and Writing Compound Dataset /* Create datatype in memory. */ s1_tid = H5Tcreate (H5T_COMPOUND, sizeof(s1_t)); H5Tinsert(s1_tid, ”Time", HOFFSET(s1_t, a), H5T_NATIVE_INT); H5Tinsert(s1_tid, ”Temp", HOFFSET(s1_t, c), H5T_NATIVE_DOUBLE); H5Tinsert(s1_tid, ”Pressure", HOFFSET(s1_t, b), H5T_NATIVE_FLOAT); Note: Use HOFFSET macro instead of calculating offset by hand. Order of H5Tinsert calls is not important if HOFFSET is used.

48. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII48 Creating and Writing Compound Dataset /* Create dataset and write data */ dataset = H5Dcreate(file, DATASETNAME, s1_tid, space, H5P_DEFAULT, H5P_DEFAULT); status = H5Dwrite(dataset, s1_tid, H5S_ALL, H5S_ALL, H5P_DEFAULT, s1); Note: In this example memory and file datatypes are the same. Type is not packed. Use H5Tpack to save space in the file. status = H5Tpack(s1_tid); status = H5Dcreate(file, DATASETNAME, s1_tid, space, H5P_DEFAULT, H5P_DEFAULT);

49. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII49 File Content with h5dump HDF5 "SDScompound.h5" { GROUP "/" { DATASET "ArrayOfStructures" { DATATYPE { H5T_STD_I32BE ”Time"; H5T_IEEE_F32BE ”Pressure"; H5T_IEEE_F64BE ”Temp"; } DATASPACE { SIMPLE ( 10 ) / ( 10 ) } DATA { { [ 0 ], [ 1 ] }, { [ 1 ], …

50. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII50 Reading Compound Dataset /* Create datatype in memory and read data. */ dataset = H5Dopen(file, DATASETNAME, H5P_DEFAULT); s2_tid = H5Dget_type(dataset); mem_tid = H5Tget_native_type (s2_tid); s1 = malloc(H5Tget_size(mem_tid)*number_of_elements); status = H5Dread(dataset, mem_tid, H5S_ALL, H5S_ALL, H5P_DEFAULT, s1); Note: We could construct memory type as we did in writing example. For general applications we need to discover the type in the file, find out corresponding memory type, allocate space and do read.

51. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII51 Reading Compound Dataset by Fields typedef struct s2_t { double c; int a; } s2_t; s2_t s2[LENGTH]; … s2_tid = H5Tcreate (H5T_COMPOUND, sizeof(s2_t)); H5Tinsert(s2_tid, ”Temp", HOFFSET(s2_t, c), H5T_NATIVE_DOUBLE); H5Tinsert(s2_tid, “Time", HOFFSET(s2_t, a), H5T_NATIVE_INT); … status = H5Dread(dataset, s2_tid, H5S_ALL, H5S_ALL, H5P_DEFAULT, s2);

52. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII52 New Way of Creating Datatypes Another way to create a compound datatype #include H5LTpublic.h ….. s2_tid = H5LTtext_to_dtype( "H5T_COMPOUND {H5T_NATIVE_DOUBLE \”Temp\"; H5T_NATIVE_INT \”Time\"; }", H5LT_DDL);

53. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII53 Need Help with Datatypes? Check our support web pages http://www.hdfgroup.uiuc.edu/UserSupport/exa mples-by-api/api18-c.html http://www.hdfgroup.uiuc.edu/UserSupport/exa mples-by-api/api16-c.html

54. www.hdfgroup.org The HDF Group November 3-5, 2009HDF/HDF-EOS Workshop XIII54 Part II Working with subsets

55. www.hdfgroup.org Collect data one way …. Array of images (3D) November 3-5, 200955HDF/HDF-EOS Workshop XIII

56. www.hdfgroup.org Stitched image (2D array) Display data another way … November 3-5, 200956HDF/HDF-EOS Workshop XIII

57. www.hdfgroup.org Data is too big to read…. November 3-5, 200957HDF/HDF-EOS Workshop XIII

58. www.hdfgroup.org Need to select and access the same elements of a dataset Refer to a region… November 3-5, 200958HDF/HDF-EOS Workshop XIII

59. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII59 HDF5 Library Features HDF5 Library provides capabilities to Describe subsets of data and perform write/read operations on subsets Hyperslab selections and partial I/O Store descriptions of the data subsets in a file Object references Region references Use efficient storage mechanism to achieve good performance while writing/reading subsets of data Chunking, compression

60. www.hdfgroup.org The HDF Group November 3-5, 2009HDF/HDF-EOS Workshop XIII60 Partial I/O in HDF5

61. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII61 How to Describe a Subset in HDF5? Before writing and reading a subset of data one has to describe it to the HDF5 Library. HDF5 APIs and documentation refer to a subset as a “selection” or “hyperslab selection”. If specified, HDF5 Library will perform I/O on a selection only and not on all elements of a dataset.

62. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII62 Types of Selections in HDF5 Two types of selections Hyperslab selection Regular hyperslab Simple hyperslab Result of set operations on hyperslabs (union, difference, …) Point selection Hyperslab selection is especially important for doing parallel I/O in HDF5 (See Parallel HDF5 Tutorial)

63. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII63 Regular Hyperslab Collection of regularly spaced blocks of equal size

64. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII64 Simple Hyperslab Contiguous subset or sub-array

65. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII65 Hyperslab Selection Result of union operation on three simple hyperslabs

66. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII66 Hyperslab Description Start - starting location of a hyperslab (1,1) Stride - number of elements that separate each block (3,2) Count - number of blocks (2,6) Block - block size (2,1) Everything is “measured” in number of elements

67. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII67 Simple Hyperslab Description Two ways to describe a simple hyperslab As several blocks Stride – (1,1) Count – (2,6) Block – (2,1) As one block Stride – (1,1) Count – (1,1) Block – (4,6) No performance penalty for one way or another

68. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII68 H5Sselect_hyperslab Function space_id Identifier of dataspace op Selection operator H5S_SELECT_SET or H5S_SELECT_OR start Array with starting coordinates of hyperslab stride Array specifying which positions along a dimension to select count Array specifying how many blocks to select from the dataspace, in each dimension block Array specifying size of element block (NULL indicates a block size of a single element in a dimension)

69. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII69 Reading/Writing Selections Programming model for reading from a dataset in a file 1.Open a dataset. 2.Get file dataspace handle of the dataset and specify subset to read from. a.H5Dget_space returns file dataspace handle a.File dataspace describes array stored in a file (number of dimensions and their sizes). b.H5Sselect_hyperslab selects elements of the array that participate in I/O operation. 3.Allocate data buffer of an appropriate shape and size

70. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII70 Reading/Writing Selections Programming model (continued) 4.Create a memory dataspace and specify subset to write to. 1.Memory dataspace describes data buffer (its rank and dimension sizes). 2.Use H5Screate_simple function to create memory dataspace. 3.Use H5Sselect_hyperslab to select elements of the data buffer that participate in I/O operation. 5.Issue H5Dread or H5Dwrite to move the data between file and memory buffer. 6.Close file dataspace and memory dataspace when done.

71. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII71 Example : Reading Two Rows 123456 789101112 131415161718 192021222324 Data in a file 4x6 matrix Buffer in memory 1-dim array of length 14

72. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII72 Example: Reading Two Rows 123456 789101112 131415161718 192021222324 start = {1,0} count = {2,6} block = {1,1} stride = {1,1} filespace = H5Dget_space (dataset); H5Sselect_hyperslab (filespace, H5S_SELECT_SET, start, NULL, count, NULL)

73. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII73 Example: Reading Two Rows start[1] = {1} count[1] = {12} dim[1] = {14} memspace = H5Screate_simple(1, dim, NULL); H5Sselect_hyperslab (memspace, H5S_SELECT_SET, start, NULL, count, NULL)

74. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII74 Example: Reading Two Rows 123456 789101112 131415161718 192021222324 789101112131415161718 H5Dread (…, …, memspace, filespace, …, …);

75. www.hdfgroup.orgNovember 3-5, 2009HDF/HDF-EOS Workshop XIII75 Things to Remember Number of elements selected in a file and in a memory buffer must be the same H5Sget_select_npoints returns number of selected elements in a hyperslab selection HDF5 partial I/O is tuned to move data between selections that have the same dimensionality; avoid choosing subsets that have different ranks (as in example above) Allocate a buffer of an appropriate size when reading data; use H5Tget_native_type and H5Tget_size to get the correct size of the data element in memory.

76. www.hdfgroup.org The HDF Group Thank You! November 3-5, 2009HDF/HDF-EOS Workshop XIII76

77. www.hdfgroup.org Acknowledgements This work was supported by cooperative agreement number NNX08AO77A from the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space Administration. November 3-5, 2009HDF/HDF-EOS Workshop XIII77

78. www.hdfgroup.org The HDF Group Questions/comments? November 3-5, 2009HDF/HDF-EOS Workshop XIII78