Ruth Aydt Quincey Koziol The HDF Group {aydt,koziol}@hdfgroup.org Introduction to HDF5 Ruth Aydt Quincey Koziol The HDF Group {aydt,koziol}@hdfgroup.org Copyright © 2010 The HDF Group. All Rights Reserved

THE BLIND MEN AND THE ELEPHANT A HIDOO FABLE by John Godfrey Saxe It was six men of Indostan  To learning much inclined, Who went to see the Elephant  (Though all of them were blind), That each by observation  Might satisfy his mind. THE BLIND MEN AND THE ELEPHANT. A HIDOO FABLE. by John Godfrey Saxe Fables                  I. It was six men of Indostan  To learning much inclined, Who went to see the Elephant  (Though all of them were blind), That each by observation  Might satisfy his mind.                  II. The First approached the Elephant,  And happening to fall Against his broad and sturdy side,  At once began to bawl: "God bless me!-but the Elephant  Is very like a wall!"                  III. The Second, feeling of the tusk,  Cried: "Ho!-what have we here So very round and smooth and sharp?  To me't is mighty clear This wonder of an Elephant  Is very like a spear!"                  IV. The Third approached the animal,  And happening to take The squirming trunk within his hands,  Thus boldly up and spake: "I see," quoth he, "the Elephant  Is very like a snake!"                  V. The Fourth reached out his eager hand,  And felt about the knee. "What most this wondrous beast is like  Is mighty plain," quoth he; "'Tis clear enough the Elephant  Is very like a tree!"                  VI. The Fifth, who chanced to touch the ear,  Said: "E'en the blindest man Can tell what this resembles most;  Deny the fact who can, This marvel of an Elephant  Is very like a fan!"                  VII. The Sixth no sooner had begun  About the beast to grope, Than, seizing on the swinging tail  That fell within his scope, "I see," quoth he, "the Elephant  Is very like a rope!"                  VIII. And so these men of Indostan  Disputed loud and long, Each in his own opinion  Exceeding stiff and strong, Though each was partly in the right,  And all were in the wrong!                  MORAL. So, oft in theologic wars  The disputants, I ween, Rail on in utter ignorance  Of what each other mean, And prate about an Elephant Not one of them has seen! … (see wikipedia.org ) Copyright © 2010 The HDF Group. All Rights Reserved

Our Purpose Today Familiarize you with HDF5 and its capabilities. 2) Help you understand how HDF5 might be applied to your data management challenges. Copyright © 2010 The HDF Group. All Rights Reserved

Outline of Sessions Overview, HDF5 Data Model Data Model Comparisons, HDF5 File Format HDF5 Software Copyright © 2010 The HDF Group. All Rights Reserved

What is HDF5? Open file format Open source software A data model Designed for high volume or complex data Open source software Works with data in the format A data model Structures for data organization and specification Copyright © 2010 The HDF Group. All Rights Reserved

HDF = Hierarchical Data Format HDF4 is the first HDF Originally called HDF; last major release was version 4 Still supported by The HDF Group HDF5 benefits from lessons learned with HDF4 Changes to file format, software, and data model Not compatible with HDF4 No plans for HDF6! Copyright © 2010 The HDF Group. All Rights Reserved

Condensed History 1987: Graphics task force at NCSA began work on architecture-independent format and library, HDF. 1990: NSF provided funding to improve documentation, testing, and user support. 1994: NASA selected HDF as standard format for Earth Observing System. 1996–1998: DOE tri-labs and NCSA, with additional support from NASA, developed HDF5, initially called “BigHDF”. 2005: NASA funded development of netCDF-4, a new version of netCDF that uses the HDF5 file format. 2006: The HDF Group, a non-profit corporation, spun off from NCSA and the University of Illinois. Copyright © 2010 The HDF Group. All Rights Reserved

HDF5 in the Formative Years Focus: Serve science and engineering communities Variety of data sources, often in single workflow Simulation, observation, visualization, annotation High volume Data rates and data sizes Complex Data types and data relationships Variety of system architectures Data often shared widely Different users care about different subsets Data must be accessible far into the future Copyright © 2010 The HDF Group. All Rights Reserved

HDF5 Philosophy A single platform with multiple uses One general file format Self-describing, allows for discovery of objects in the file Designed for speed and storage efficiency One software library Options to adapt I/O and storage to data needs Layers above and below One data model Structures based on mathematical foundations Supports expression of complex data types and relationships Work well with other technologies Attention to compatibility Copyright © 2010 The HDF Group. All Rights Reserved

HDF5 is like… Copyright © 2010 The HDF Group. All Rights Reserved

HDF5 Technology Platform HDF5 data model The “building blocks” for data organization and specification HDF5 software Library, language interfaces, tools HDF5 file format Bit-level organization of HDF5 file Let’s look at… Recall… Copyright © 2010 The HDF Group. All Rights Reserved

a.k.a. HDF5 Abstract Data Model a.k.a. HDF5 Logical Data Model HDF5 Data Model Dataset Link Group HDF5 Objects Datatype Attribute Dataspace File a.k.a. HDF5 Abstract Data Model a.k.a. HDF5 Logical Data Model Copyright © 2010 The HDF Group. All Rights Reserved

There will be a Quiz! Use objects from the HDF5 data model to design an HDF5 file to store daily temperature measurements made at various locations throughout a building. Copyright © 2010 The HDF Group. All Rights Reserved

HDF5 File An HDF5 file is a container that holds data objects. lat | lon | temp ----|-----|----- 12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 An HDF5 file is a container that holds data objects. Experiment Notes: Serial Number: 99378920 Date: 3/13/09 Configuration: Standard 3 An HDF5 file is not necessarily a file on a filesystem. Copyright © 2010 The HDF Group. All Rights Reserved

HDF5 Dataset HDF5 Datatype HDF5 Dataspace Integer 32bit LE HDF5 Datatype 3 Rank Dim_2 = 7 Dimensions Dim_0 = 4 Dim_1 = 5 HDF5 Dataspace Specifications for single data element and array dimensions Multi-dimensional array of identically typed data elements HDF5 datasets organize and contain “raw data values”. HDF5 datatypes describe individual data elements. HDF5 dataspaces describe the logical layout of the data elements. Copyright © 2010 The HDF Group. All Rights Reserved

HDF5 Datatypes Describe individual data elements in an HDF5 dataset Wide range of datatypes supported integer, float, unsigned, bitfield, … user-defined (e.g., 13-bit integer) variable length types (e.g., strings) reference to object; reference to dataset region enumerations - names mapped to integers opaque array compound (similar to C structs) Can be named and shared Committed datatypes (a.k.a. named datatypes) Copyright © 2010 The HDF Group. All Rights Reserved

HDF5 Dataset & Datatype HDF5 Datatype Integer 32bit LE HDF5 Datatype Specifications for single data element Multi-dimensional array of identically typed data elements HDF5 datasets organize and contain “raw data values”. HDF5 datatypes describe individual data elements. Copyright © 2010 The HDF Group. All Rights Reserved

HDF5 Dataspaces Describe the logical layout of the elements in an HDF5 dataset NULL no elements Scalar single element Simple array (most common) multiple elements organized in a rectangular array rank = number of dimensions (1-32) dimension sizes = number of elements in each dimension maximum number of elements in each dimension may be fixed or unlimited multi-dimensional Copyright © 2010 The HDF Group. All Rights Reserved

HDF5 Dataset & Dataspace Dim_1 = 5 Dim_2 = 7 Dim_0 = 4 HDF5 Dataspace 3 Rank Dim_2 = 7 Dimensions Dim_0 = 4 Dim_1 = 5 Specifications for array dimensions Multi-dimensional array of identically typed data elements HDF5 datasets organize and contain “raw data values”. HDF5 dataspaces describe the logical layout of the data elements. Copyright © 2010 The HDF Group. All Rights Reserved

HDF5 Dataset HDF5 Datatype HDF5 Dataspace Integer 32bit LE HDF5 Datatype HDF5 Dataspace Rank Dimensions 3 Dim_0 = 4 Dim_1 = 5 Dim_2 = 7 Specifications for single data element and array dimensions Multi-dimensional array of identically typed data elements HDF5 datasets organize and contain “raw data values”. HDF5 datatypes describe individual data elements. HDF5 dataspaces describe the logical layout of the data elements. Copyright © 2010 The HDF Group. All Rights Reserved

Compound Datatype in HDF5 Dataset int8 int4 int16 2x3x2 array of float32 Compound Datatype PurpleGreenRedBlue Committed Datatype HDF5 Dataset PurpleGreenRedBlue HDF5 Datatype 2 Rank DimSizes 5 3 HDF5 Dataspace unlimited Max DimSizes ... … Copyright © 2010 The HDF Group. All Rights Reserved

HDF5 Data Model: Are we there yet? HDF5 Objects Group and Link Attribute Dataspace   Datatype  Dataset  File Copyright © 2010 The HDF Group. All Rights Reserved

HDF5 Attributes Typically contain user metadata Have a name and a value May be associated with HDF5 datasets HDF5 committed datatypes HDF5 groups Value is described by a datatype and a dataspace analogous to a dataset Copyright © 2010 The HDF Group. All Rights Reserved

HDF5 Groups and Links / HDF5 groups and links organize data objects. Every HDF5 file has a root group / Experiment Notes: Serial Number: 99378920 Date: 3/13/09 Configuration: Standard 3 Viz SimOut Parameters 10;100;1000 lat | lon | temp ----|-----|----- 12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 Timestep 36,000 Copyright © 2010 The HDF Group. All Rights Reserved

Quiz Use objects from the HDF5 data model to design an HDF5 file to store daily temperature measurements made at various locations throughout a building. meters south 1 / February January Building SRP-Z Sensor Type Temperature NW 2 east above ground 31 … 3 day of month Copyright © 2010 The HDF Group. All Rights Reserved

Review HDF5 consists of HDF5 designed to support file format software data model file, dataset, datatype, dataspace, attribute, group, link HDF5 designed to support management of high-volume, complex data data sharing and preservation Copyright © 2010 The HDF Group. All Rights Reserved

Test Data and HDF5 Data Model Objects Copyright © 2010 The HDF Group. All Rights Reserved

Stretch Break Copyright © 2010 The HDF Group. All Rights Reserved