Www.hdfgroup.org The HDF Group Introduction to HDF5 Session Three HDF5 Software Overview 1 Copyright © 2010 The HDF Group. All Rights Reserved.

Slides:



Advertisements
Similar presentations
The HDF Group November 3-5, 2009HDF/HDF-EOS Workshop XIII1 HDF-Java Products Peter Cao The HDF Group The 13 th HDF and HDF-EOS Workshop.
Advertisements

11/6/07HDF and HDF-EOS Workshop XI, Landover, MD1 Introduction to HDF5 HDF and HDF-EOS Workshop XI November 6-8, 2007.
The HDF Group Introduction to HDF5 Barbara Jones The HDF Group The 13 th HDF & HDF-EOS Workshop November 3-5, HDF/HDF-EOS Workshop.
Status of netCDF-3, netCDF-4, and CF Conventions Russ Rew Community Standards for Unstructured Grids Workshop, Boulder
© 2008The MathWorks, Inc. ® ® The MATLAB Low-Level HDF5 Interface John Evans.
C++ Programming. Table of Contents History What is C++? Development of C++ Standardized C++ What are the features of C++? What is Object Orientation?
Parallel HDF5 Introductory Tutorial May 19, 2008 Kent Yang The HDF Group 5/19/20081SCICOMP 14 Tutorial.
HDF 1 HDF5 Advanced Topics Object’s Properties Storage Methods and Filters Datatypes HDF and HDF-EOS Workshop VIII October 26, 2004.
The HDF Group April 17-19, 2012HDF/HDF-EOS Workshop XV1 Introduction to HDF5 Barbara Jones The HDF Group The 15 th HDF and HDF-EOS Workshop.
1 High level view of HDF5 Data structures and library HDF Summit Boeing Seattle September 19, 2006.
HDF5 A new file format & software for high performance scientific data management.
The HDF Group Parallel HDF5 Design and Programming Model May 30-31, 2012HDF5 Workshop at PSI 1.
A Metadata Based Approach For Supporting Subsetting Queries Over Parallel HDF5 Datasets Vignesh Santhanagopalan Graduate Student Department Of CSE.
February 2-3, 2006SRB Workshop, San Diego P eter Cao, NCSA Mike Wan, SDSC Sponsored by NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration Object-level.
1 Introduction to HDF5 Data Model, Programming Model and Library APIs HDF and HDF-EOS Workshop VIII October 26, 2004.
April 28, 2008LCI Tutorial1 HDF5 Tutorial LCI April 28, 2008.
The HDF Group Virtual Object Layer in HDF5 Exploring new HDF5 concepts May 30-31, 2012HDF5 Workshop at PSI 1.
Why do I want to know about HDF and HDF- EOS? Hierarchical Data Format for the Earth Observing System (HDF-EOS) is NASA's primary format for standard data.
May 30-31, 2012HDF5 Workshop at PSI1 HDF5 at Glance Quick overview of known topics.
1 Computer Programming (ECGD2102 ) Using MATLAB Instructor: Eng. Eman Al.Swaity Lecture (1): Introduction.
The HDF Group HDF5 Datasets and I/O Dataset storage and its effect on performance May 30-31, 2012HDF5 Workshop at PSI 1.
The netCDF-4 data model and format Russ Rew, UCAR Unidata NetCDF Workshop 25 October 2012.
The HDF Group HDF5 Tools Updates Peter Cao, The HDF Group September 28-30, 20101HDF and HDF-EOS Workshop XIV.
The HDF Group October 28, 2010NetcDF Workshop1 Introduction to HDF5 Quincey Koziol The HDF Group Unidata netCDF Workshop October 28-29,
October 15, 2008HDF and HDF-EOS Workshop XII1 What will be new in HDF5?
1 HDF5 Life cycle of data Boeing September 19, 2006.
Page 1 TOOLKIT / HDF-EOS Status and Development Abe Taaheri, Raytheon IIS Aura DSWG meeting October 2007.
CCGrid 2014 Improving I/O Throughput of Scientific Applications using Transparent Parallel Compression Tekin Bicer, Jian Yin and Gagan Agrawal Ohio State.
5 Copyright © 2005, Oracle. All rights reserved. Managing Database Storage Structures.
6 Copyright © 2007, Oracle. All rights reserved. Managing Database Storage Structures.
A. Frank - P. Weisberg Operating Systems Structure of Operating Systems.
The HDF Group Introduction to netCDF-4 Elena Pourmal The HDF Group 110/17/2015.
Connections to Other Packages The Cactus Team Albert Einstein Institute
1 Status of HDF-EOS, Related Software and Tools. 2 TOOLKIT / HDF-EOS Support.
UNIX & Windows NT Name: Jing Bai ID: Date:8/28/00.
The HDF Group Single Writer/Multiple Reader (SWMR) 110/17/15.
Intro to Parallel HDF5 10/17/151ICALEPCS /17/152 Outline Overview of Parallel HDF5 design Parallel Environment Requirements Performance Analysis.
September 9, 2008SPEEDUP Workshop - HDF5 Tutorial1 HDF5 Tutorial 37 th SPEEDUP Workshop on HPC Albert Cheng, Elena Pourmal The HDF Group.
The HDF Group 10/17/151 Introduction to HDF5 ICALEPCS 2015.
The HDF Group Introduction to HDF5 Session Two Data Model Comparison HDF5 File Format 1 Copyright © 2010 The HDF Group. All Rights Reserved.
The HDF Group Single Writer/Multiple Reader (SWMR) 110/17/15.
The HDF Group Introduction to HDF5 Session 7 Datatypes 1 Copyright © 2010 The HDF Group. All Rights Reserved.
1 Chapter 2: Operating-System Structures Services Interface provided to users & programmers –System calls (programmer access) –User level access to system.
Copyright © 2010 The HDF Group. All Rights Reserved1 Data Storage and I/O in HDF5.
- 1 - Overview of Parallel HDF Overview of Parallel HDF5 and Performance Tuning in HDF5 Library NCSA/University of Illinois at Urbana- Champaign.
1 January 11-13, 2010ESRF Workshop – Introduction to HDF5 Introduction to HDF5 Francesc Alted Consultant and PyTables creator.
Sung-Dong Kim, Dept. of Computer Engineering, Hansung University Java - Introduction.
HDF and HDF-EOS Workshop XII
Hierarchical Data Formats (HDF) Update
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING CLOUD COMPUTING
Parallel HDF5 Introductory Tutorial
Introduction to HDF5 Session Five Reading & Writing Raw Data Values
CASE STUDY 1: Linux and Android
Java Beans Sagun Dhakhwa.
Introduction to HDF5 Session Four Java Products
In-situ Visualization using VisIt
Ruth Aydt Quincey Koziol The HDF Group
Introduction to HDF5 Tutorial.
Chapter 3: Windows7 Part 1.
Getting Started ARCS Lab..
Chapter 6 System and Application Software
HDF5 Virtual Dataset Elena Pourmal Copyright 2017, The HDF Group.
Introduction to HDF5 Mike McGreevy The HDF Group
Moving applications to HDF
Virtual Machines (Introduction to Virtual Machines)
MPJ: A Java-based Parallel Computing System
Hierarchical Data Format (HDF) Status Update
Outline Operating System Organization Operating System Examples
System calls….. C-program->POSIX call
Presentation transcript:

The HDF Group Introduction to HDF5 Session Three HDF5 Software Overview 1 Copyright © 2010 The HDF Group. All Rights Reserved

Our Purpose Today 1)Familiarize you with HDF5 and its capabilities. 2) Help you understand how HDF5 might be applied to your data management challenges. Copyright © 2010 The HDF Group. All Rights Reserved2

Project Data Model Copyright © 2010 The HDF Group. All Rights Reserved3 Project Domain Concepts Schema Logical Model built from HDF5 Data Model Objects HDF5 File(s) in the HDF5 Format

HDF5 Technology Platform HDF5 data model The “building blocks” for data organization and specification HDF5 software Library, language interfaces, tools HDF5 file format Bit-level organization of HDF5 file Self-describing Designed for high-performance Copyright © 2010 The HDF Group. All Rights Reserved4 Software: the missing link

HDF5 Software Fundamentally, HDF5 software operates on: 1.Objects in the HDF5 Data Model Write a logical model to an HDF5 file Reconstruct a logical model from an HDF5 file 2.Raw data values in datasets and attributes Write values to an HDF5 file Read values from an HDF5 file Note: Updates, partial writes, and partial reads are supported. Copyright © 2010 The HDF Group. All Rights Reserved5

The Big Picture Copyright © 2010 The HDF Group. All Rights Reserved6 HDF5 Software User Application Schema Data Values mental model of data HDF5 File

HDF5 Philosophy Review One software library (from The HDF Group) Options to adapt I/O and storage to data needs Layers above and below Work well with other technologies Attention to compatibility Copyright © 2010 The HDF Group. All Rights Reserved7

Software Layers – Library View Copyright © 2010 The HDF Group. All Rights Reserved8 HDF5 Library Internals: memory management, conversions, other details… User Application File Virtual File I/O Drivers: Posix I/O, Split Files, MPI I/O, … HDF5 File Split Files File on Parallel Filesystem OS, MPI-IO, Filesystem, SAN,... HDF5 Object APIs (in C) Schema + Data Schema Data Values HDF5 Object APIs (in C): Schema + Data + Properties Properties

HDF5 Software (in C) Library and full set of HDF5 Object APIs written in C Portable across platforms (in 1996) High-performance C is not object-oriented, but we have HDF5 Objects No classes in C Simulated through naming conventions No object instances in C Simulated through identifiers Identifer (handle) returned when object created Identifer used to invoke methods on specific instance of object Copyright © 2010 The HDF Group. All Rights Reserved9

HDF5 Object APIs (Schema + Data Values) Prefix H5F H5D H5T H5S H5A H5G H5L Copyright © 2010 The HDF Group. All Rights Reserved10 File Dataset Link Group Attribute Dataspace Datatype HDF5 Objects For Example… H5Fcreate H5Dwrite H5Tget_order H5Sclose H5Aget_space H5Gopen H5Literate hid_t file_id, group_id; file_id = H5Fcreate(“file.h5”, … ); group_id = H5Gcreate(file_id, “January”, … );

HDF5 Properties Mechanism for passing information between applications and HDF5 software Property information is not directly related to the HDF5 data model objects or data values “Knobs” that control the advanced features of HDF5 Copyright © 2010 The HDF Group. All Rights Reserved11

HDF5 Properties Creation Properties Set when HDF5 Object is created; persist in HDF5 file Size of symbol table B-trees for File Storage layout for Dataset Access Properties Set when HDF5 Object is opened; persist until Object closed File driver Type conversion buffer size Property Lists exposed by H5P API Copyright © 2010 The HDF Group. All Rights Reserved12

General Programming Paradigm Properties of object are optionally defined Creation properties Access properties Default values used if none are defined Object is opened or created Object is accessed, possibly many times Object is closed Copyright © The HDF Group. All Rights Reserved13 hid_t plist_id, dset_id; plist_id = H5Pcreate(H5P_DATASET_CREATE); status = H5Pset_chunk(plist_id, …); dset_id = H5Dcreate(group_id,”1”,…,plist_id,H5P_DEFAULT);

14 Dataset: Library and Format View Integer 32bit LE HDF5 Datatype 3 Rank Dim_2 = 7 Dimensions Dim_0 = 4 Dim_1 = 5 HDF5 Dataspace Time = 32.4 Pressure = 987 Temp = 56 Attributes Chunked Storage Info Copyright © 2010 The HDF Group. All Rights Reserved

Software Layers – Languages View Copyright © 2010 The HDF Group. All Rights Reserved15 HDF5 Library Internals User Application File Virtual File I/O Drivers HDF Java Object Package Schema Fortran 90 HDF5 Object APIs (in C) C++ Java HDF5 Interface (JHI5) h5py MATLAB™ Properties Data Values

Software Layers – Tools View Copyright © 2010 The HDF Group. All Rights Reserved16 HDF5 Library Internals HDFView File Virtual File I/O Drivers HDF Java Object Package h5dump HDF5 Object APIs (in C) h5ls Java HDF5 Interface (JHI5) h5repack …

Portability & Robustness Runs on many platforms* Linux and UNIX workstations Windows, Mac OS X Crays, VMS systems Large distributed-memory clusters Quality Assurance Daily regression tests on key platforms Meets NASA’s highest technology readiness level *platform = architecture + OS + compiler Copyright © 2010 The HDF Group. All Rights Reserved17

Software Layers – Project Domain View Copyright © 2010 The HDF Group. All Rights Reserved18 HDF5 Software Building Temperature Monitoring Application sensor reading location date building HDF5 File saveReading(building, location, value, date) getAverageReading(building, start_date, end_date) … Building Sensor APIs Building Temperature Monitoring Application

Software Layers – CFD Domain View Copyright © 2010 The HDF Group. All Rights Reserved19 HDF5 Software boundary conditions flow equations geometry definition turbulence … HDF5 File CGNS: CFD General Notation System My Computational Fluid Dynamics Application Your CFD Application

Software Layers – Voxel Domain View Copyright © 2010 The HDF Group. All Rights Reserved20 HDF5 Software voxels, fluid simulation, volume rendering, movies HDF5 File Field3D: an open source library for storing voxel data developed by Sony Pictures Imageworks to replace three different in-house file formats. Alice in Wonderland imageworks.com

Field3D Programmers Guide Copyright © 2010 The HDF Group. All Rights Reserved21

Software Layers – EOS Domain View Copyright © 2010 The HDF Group. All Rights Reserved22 HDF5 Software Grids Swaths Points UVReflectivity Instrument Name HDF5 File NASA HDF-EOS5 APIs NASA Data Product Application Climage Modeling Application OMI-Aura_L3-OMTO3e_2005m1214_v m0929t he5 MATLAB™

h5ls Copyright © 2010 The HDF Group. All Rights Reserved23 > h5ls –r –f / Group /HDFEOS Group /HDFEOS/ADDITIONAL Group /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES Group /HDFEOS/GRIDS Group /HDFEOS/GRIDS/OMI\ Column\ Amount\ O3 Group /HDFEOS/GRIDS/OMI\ Column\ Amount\ O3/Data\ Fields Group /HDFEOS/GRIDS/OMI\ Column\ Amount\ O3/Data\ Fields/ColumnAmountO3 Dataset {720,1440} /HDFEOS/GRIDS/OMI\ Column\ Amount\ O3/Data\ Fields/Reflectivity331 Dataset {720,1440} /HDFEOS/GRIDS/OMI\ Column\ Amount\ O3/Data\ Fields/UVAerosolIndex Dataset {720,1440} /HDFEOS\ INFORMATION Group /HDFEOS\ INFORMATION/StructMetadata.0 Dataset {SCALAR} >

h5dump Copyright © 2010 The HDF Group. All Rights Reserved24 > h5dump -H HDF5 "OMI-Aura_L3-OMTO3e_2005m1214_v m0929t he5" { GROUP "/" { GROUP "HDFEOS" { GROUP "ADDITIONAL" { GROUP "FILE_ATTRIBUTES" { ATTRIBUTE "EndUTC" { DATATYPE H5T_STRING {... GROUP "Data Fields" { DATASET "ColumnAmountO3" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 720, 1440 ) / ( 720, 1440 ) } ATTRIBUTE "MissingValue" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 1 ) / ( 1 ) } }...

Review HDF5 consists of file format self-describing, structures to support high-performance software layers for compatibility and extensibility performance features data model file, dataset, datatype, dataspace, attribute, group, link HDF5 designed to support management of high-volume, complex data data sharing and preservation Copyright © 2010 The HDF Group. All Rights Reserved25

Stretch Break Copyright © 2010 The HDF Group. All Rights Reserved26 … while I start HDFView demo with AURA file