Presentation is loading. Please wait.

Presentation is loading. Please wait.

Ruth Aydt Quincey Koziol The HDF Group

Published byRafe Tucker Modified over 6 years ago

Similar presentations

Presentation on theme: "Ruth Aydt Quincey Koziol The HDF Group"— Presentation transcript:

1 Ruth Aydt Quincey Koziol The HDF Group {aydt,koziol}@hdfgroup.org
Introduction to HDF5 Ruth Aydt Quincey Koziol The HDF Group Copyright © 2010 The HDF Group. All Rights Reserved

2 THE BLIND MEN AND THE ELEPHANT
A HIDOO FABLE by John Godfrey Saxe It was six men of Indostan  To learning much inclined, Who went to see the Elephant  (Though all of them were blind), That each by observation  Might satisfy his mind. THE BLIND MEN AND THE ELEPHANT. A HIDOO FABLE. by John Godfrey Saxe Fables                  I. It was six men of Indostan  To learning much inclined, Who went to see the Elephant  (Though all of them were blind), That each by observation  Might satisfy his mind.                  II. The First approached the Elephant,  And happening to fall Against his broad and sturdy side,  At once began to bawl: "God bless me!-but the Elephant  Is very like a wall!"                  III. The Second, feeling of the tusk,  Cried: "Ho!-what have we here So very round and smooth and sharp?  To me't is mighty clear This wonder of an Elephant  Is very like a spear!"                  IV. The Third approached the animal,  And happening to take The squirming trunk within his hands,  Thus boldly up and spake: "I see," quoth he, "the Elephant  Is very like a snake!"                  V. The Fourth reached out his eager hand,  And felt about the knee. "What most this wondrous beast is like  Is mighty plain," quoth he; "'Tis clear enough the Elephant  Is very like a tree!"                  VI. The Fifth, who chanced to touch the ear,  Said: "E'en the blindest man Can tell what this resembles most;  Deny the fact who can, This marvel of an Elephant  Is very like a fan!"                  VII. The Sixth no sooner had begun  About the beast to grope, Than, seizing on the swinging tail  That fell within his scope, "I see," quoth he, "the Elephant  Is very like a rope!"                  VIII. And so these men of Indostan  Disputed loud and long, Each in his own opinion  Exceeding stiff and strong, Though each was partly in the right,  And all were in the wrong!                  MORAL. So, oft in theologic wars  The disputants, I ween, Rail on in utter ignorance  Of what each other mean, And prate about an Elephant Not one of them has seen! … (see wikipedia.org ) Copyright © 2010 The HDF Group. All Rights Reserved

3 Our Purpose Today Familiarize you with HDF5 and its capabilities.
2) Help you understand how HDF5 might be applied to your data management challenges. Copyright © 2010 The HDF Group. All Rights Reserved

4 Outline of Sessions Overview, HDF5 Data Model
Data Model Comparisons, HDF5 File Format HDF5 Software Copyright © 2010 The HDF Group. All Rights Reserved

5 What is HDF5? Open file format Open source software A data model
Designed for high volume or complex data Open source software Works with data in the format A data model Structures for data organization and specification Copyright © 2010 The HDF Group. All Rights Reserved

6 HDF = Hierarchical Data Format
HDF4 is the first HDF Originally called HDF; last major release was version 4 Still supported by The HDF Group HDF5 benefits from lessons learned with HDF4 Changes to file format, software, and data model Not compatible with HDF4 No plans for HDF6! Copyright © 2010 The HDF Group. All Rights Reserved

7 Condensed History 1987: Graphics task force at NCSA began work on architecture-independent format and library, HDF. 1990: NSF provided funding to improve documentation, testing, and user support. 1994: NASA selected HDF as standard format for Earth Observing System. 1996–1998: DOE tri-labs and NCSA, with additional support from NASA, developed HDF5, initially called “BigHDF”. 2005: NASA funded development of netCDF-4, a new version of netCDF that uses the HDF5 file format. 2006: The HDF Group, a non-profit corporation, spun off from NCSA and the University of Illinois. Copyright © 2010 The HDF Group. All Rights Reserved

8 HDF5 in the Formative Years
Focus: Serve science and engineering communities Variety of data sources, often in single workflow Simulation, observation, visualization, annotation High volume Data rates and data sizes Complex Data types and data relationships Variety of system architectures Data often shared widely Different users care about different subsets Data must be accessible far into the future Copyright © 2010 The HDF Group. All Rights Reserved

9 HDF5 Philosophy A single platform with multiple uses
One general file format Self-describing, allows for discovery of objects in the file Designed for speed and storage efficiency One software library Options to adapt I/O and storage to data needs Layers above and below One data model Structures based on mathematical foundations Supports expression of complex data types and relationships Work well with other technologies Attention to compatibility Copyright © 2010 The HDF Group. All Rights Reserved

10 HDF5 is like… Copyright © 2010 The HDF Group. All Rights Reserved

11 HDF5 Technology Platform
HDF5 data model The “building blocks” for data organization and specification HDF5 software Library, language interfaces, tools HDF5 file format Bit-level organization of HDF5 file Let’s look at… Recall… Copyright © 2010 The HDF Group. All Rights Reserved

12 a.k.a. HDF5 Abstract Data Model a.k.a. HDF5 Logical Data Model
HDF5 Data Model Dataset Link Group HDF5 Objects Datatype Attribute Dataspace File a.k.a. HDF5 Abstract Data Model a.k.a. HDF5 Logical Data Model Copyright © 2010 The HDF Group. All Rights Reserved

13 There will be a Quiz! Use objects from the HDF5 data model to design an HDF5 file to store daily temperature measurements made at various locations throughout a building. Copyright © 2010 The HDF Group. All Rights Reserved

14 HDF5 File An HDF5 file is a container that holds data objects.
lat | lon | temp ----|-----|----- 12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 An HDF5 file is a container that holds data objects. Experiment Notes: Serial Number: Date: 3/13/09 Configuration: Standard 3 An HDF5 file is not necessarily a file on a filesystem. Copyright © 2010 The HDF Group. All Rights Reserved

15 HDF5 Dataset HDF5 Datatype HDF5 Dataspace
Integer 32bit LE HDF5 Datatype 3 Rank Dim_2 = 7 Dimensions Dim_0 = 4 Dim_1 = 5 HDF5 Dataspace Specifications for single data element and array dimensions Multi-dimensional array of identically typed data elements HDF5 datasets organize and contain “raw data values”. HDF5 datatypes describe individual data elements. HDF5 dataspaces describe the logical layout of the data elements. Copyright © 2010 The HDF Group. All Rights Reserved

16 HDF5 Datatypes Describe individual data elements in an HDF5 dataset
Wide range of datatypes supported integer, float, unsigned, bitfield, … user-defined (e.g., 13-bit integer) variable length types (e.g., strings) reference to object; reference to dataset region enumerations - names mapped to integers opaque array compound (similar to C structs) Can be named and shared Committed datatypes (a.k.a. named datatypes) Copyright © 2010 The HDF Group. All Rights Reserved

17 HDF5 Dataset & Datatype HDF5 Datatype
Integer 32bit LE HDF5 Datatype Specifications for single data element Multi-dimensional array of identically typed data elements HDF5 datasets organize and contain “raw data values”. HDF5 datatypes describe individual data elements. Copyright © 2010 The HDF Group. All Rights Reserved

18 HDF5 Dataspaces Describe the logical layout of the elements in an HDF5 dataset NULL no elements Scalar single element Simple array (most common) multiple elements organized in a rectangular array rank = number of dimensions (1-32) dimension sizes = number of elements in each dimension maximum number of elements in each dimension may be fixed or unlimited multi-dimensional Copyright © 2010 The HDF Group. All Rights Reserved

19 HDF5 Dataset & Dataspace
Dim_1 = 5 Dim_2 = 7 Dim_0 = 4 HDF5 Dataspace 3 Rank Dim_2 = 7 Dimensions Dim_0 = 4 Dim_1 = 5 Specifications for array dimensions Multi-dimensional array of identically typed data elements HDF5 datasets organize and contain “raw data values”. HDF5 dataspaces describe the logical layout of the data elements. Copyright © 2010 The HDF Group. All Rights Reserved

20 HDF5 Dataset HDF5 Datatype HDF5 Dataspace
Integer 32bit LE HDF5 Datatype HDF5 Dataspace Rank Dimensions 3 Dim_0 = 4 Dim_1 = 5 Dim_2 = 7 Specifications for single data element and array dimensions Multi-dimensional array of identically typed data elements HDF5 datasets organize and contain “raw data values”. HDF5 datatypes describe individual data elements. HDF5 dataspaces describe the logical layout of the data elements. Copyright © 2010 The HDF Group. All Rights Reserved

21 Compound Datatype in HDF5 Dataset
int8 int4 int16 2x3x2 array of float32 Compound Datatype PurpleGreenRedBlue Committed Datatype HDF5 Dataset PurpleGreenRedBlue HDF5 Datatype 2 Rank DimSizes 5 3 HDF5 Dataspace unlimited Max DimSizes ... … Copyright © 2010 The HDF Group. All Rights Reserved

22 HDF5 Data Model: Are we there yet?
HDF5 Objects Group and Link Attribute Dataspace Datatype Dataset File Copyright © 2010 The HDF Group. All Rights Reserved

23 HDF5 Attributes Typically contain user metadata
Have a name and a value May be associated with HDF5 datasets HDF5 committed datatypes HDF5 groups Value is described by a datatype and a dataspace analogous to a dataset Copyright © 2010 The HDF Group. All Rights Reserved

24 HDF5 Groups and Links / HDF5 groups and links organize data objects.
Every HDF5 file has a root group / Experiment Notes: Serial Number: Date: 3/13/09 Configuration: Standard 3 Viz SimOut Parameters 10;100;1000 lat | lon | temp ----|-----|----- 12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 Timestep 36,000 Copyright © 2010 The HDF Group. All Rights Reserved

25 Quiz Use objects from the HDF5 data model to design an HDF5 file to store daily temperature measurements made at various locations throughout a building. meters south 1 / February January Building SRP-Z Sensor Type Temperature NW 2 east above ground 31 … 3 day of month Copyright © 2010 The HDF Group. All Rights Reserved

26 Review HDF5 consists of HDF5 designed to support file format software
data model file, dataset, datatype, dataspace, attribute, group, link HDF5 designed to support management of high-volume, complex data data sharing and preservation Copyright © 2010 The HDF Group. All Rights Reserved

27 Test Data and HDF5 Data Model Objects
Copyright © 2010 The HDF Group. All Rights Reserved

28 Stretch Break Copyright © 2010 The HDF Group. All Rights Reserved

Download ppt "Ruth Aydt Quincey Koziol The HDF Group"

Similar presentations

Property of JB Innovation Loving God, we move too quickly past the joy and celebration of Easter Sunday, forgetting your amazing power, forgetting the.

Property of JB Innovation Loving God, we move too quickly past the joy and celebration of Easter Sunday, forgetting your amazing power, forgetting the.
THE ELEPHANT And the 6 (billion) blind men A treatise based on an old poem by John Godfrey Saxe (1816 - 1887)

THE ELEPHANT And the 6 (billion) blind men A treatise based on an old poem by John Godfrey Saxe ( )
Project Senses Six blind men and the Elephant. Once upon a time there were six blind men. They lived in a town in India. They thought they were very clever.

Project Senses Six blind men and the Elephant. Once upon a time there were six blind men. They lived in a town in India. They thought they were very clever.
A PLFS Plugin for HDF5 for Improved I/O Performance and Analysis Kshitij Mehta 1, John Bent 2, Aaron Torres 3, Gary Grider 3, Edgar Gabriel 1 1 University.

A PLFS Plugin for HDF5 for Improved I/O Performance and Analysis Kshitij Mehta 1, John Bent 2, Aaron Torres 3, Gary Grider 3, Edgar Gabriel 1 1 University.
A Paradigm - a perspective of way of looking at the world.

A Paradigm - a perspective of way of looking at the world.
The Blind Men and the Elephant

The Blind Men and the Elephant
Writing and Reading: Lesson 4

Writing and Reading: Lesson 4
CS 490 Design Patterns. Introduce Yourself Your name Your name What attracted you to this course? What attracted you to this course? Experience with Design.

CS 490 Design Patterns. Introduce Yourself Your name Your name What attracted you to this course? What attracted you to this course? Experience with Design.
Blind Men and the Elephant (by John Godfrey Saxe) American poet John Godfrey Saxe (1816-1887) based this poem, The Blind Men and the Elephant, on a fable.

Blind Men and the Elephant (by John Godfrey Saxe) American poet John Godfrey Saxe ( ) based this poem, "The Blind Men and the Elephant", on a fable.
Database Principles Conceptual Modeling. Database Principles What is a Conceptual Model? A mental model of the world around us Models necessarily mean.

Database Principles Conceptual Modeling. Database Principles What is a Conceptual Model? A mental model of the world around us Models necessarily mean.
Science According to the National Academy of Sciences, Science is the use of evidence to construct testable explanations and predictions of natural phenomena,

Science According to the National Academy of Sciences, Science is the use of evidence to construct testable explanations and predictions of natural phenomena,
Modeling Lecture 1. Modeling and Sustainability CE4505 Surface Water Quality Engineering CE4505 – Surface Water Quality Engineering.

Modeling Lecture 1. Modeling and Sustainability CE4505 Surface Water Quality Engineering CE4505 – Surface Water Quality Engineering.
Making Movies, Building Community: Community Produced Video As A Tool for Community Development Amy Lake Grace Njeru, PhD Steve Jeanetta, PhD University.

Making Movies, Building Community: Community Produced Video As A Tool for Community Development Amy Lake Grace Njeru, PhD Steve Jeanetta, PhD University.
Observation, Perception & Theory Theoretical Framework.

Observation, Perception & Theory Theoretical Framework.
® Sponsored by Towards a Systems Model for Urban Planning: Moving SDI Towards Active Models of Reality 92nd OGC Technical Committee Calgary Canada John.

® Sponsored by Towards a Systems Model for Urban Planning: Moving SDI Towards Active Models of Reality 92nd OGC Technical Committee Calgary Canada John.
Supporting Inclusion Through Value Based Practice Best Start Annual Conference January 2006 Presented by Leslie McDiarmid Better Beginnings Better Futures.

Supporting Inclusion Through Value Based Practice Best Start Annual Conference January 2006 Presented by Leslie McDiarmid Better Beginnings Better Futures.
THE ROLE OF MODELS IN THE CONVERGENCE OF SCIENTIFIC KNOWLEDGE Linda Weiser Friedman Zicklin School of Business, Baruch College and the Graduate Center.

THE ROLE OF MODELS IN THE CONVERGENCE OF SCIENTIFIC KNOWLEDGE Linda Weiser Friedman Zicklin School of Business, Baruch College and the Graduate Center.
Banksy – graffiti artist & elephant wrangler “The show is about the elephant in the room, the problems that we don't talk about. The fact that 10 billion.

Banksy – graffiti artist & elephant wrangler “The show is about the elephant in the room, the problems that we don't talk about. The fact that 10 billion.
A poem written by John Godfrey Saxe ( )

A poem written by John Godfrey Saxe ( )
The Fifth, who chanced to touch the ear, It was six men of Indostan To learning much inclined, Said: E'en the blindest man Who went to see the Elephant.

The Fifth, who chanced to touch the ear, It was six men of Indostan To learning much inclined, Said: "E'en the blindest man Who went to see the Elephant.

Similar presentations

Ads by Google