1. Mohamad Chaarawi chaarawi@hdfgroup.org The HDF Group
HDF5: State of the Union
Mohamad Chaarawi
The HDF Group
November 18, 2014

2. What is HDF5?
A versatile data model that can represent very complex data objects and a wide variety of metadata.
A completely portable file format with no limit on the number or size of data objects stored.
An open source software library that runs on a wide range of computational platforms, from cell phones to massively parallel systems, and implements a high-level API with C, C++, Fortran 90, and Java interfaces.
A rich set of integrated performance features that allow for access time and storage space optimizations.
Tools and applications for managing, manipulating, viewing, and analyzing the data in the collection.
November 18, 2014

3. HDF5 Technology Platform
HDF5 Abstract Data Model
Defines the “building blocks” for data organization and specification
Files, Groups, Links, Datasets, Attributes, Datatypes, Dataspaces
HDF5 Software
Tools
Language Interfaces
HDF5 Library
HDF5 Binary File Format
Bit-level organization of HDF5 file
Defined by HDF5 File Format Specification
November 18, 2014

4. Everything else is built essentially from these parts.
HDF5 Data Model
Groups – provide structure among objects
Datasets – where the primary data goes
Data arrays
Rich set of datatype options
Flexible, efficient storage and I/O
Attributes, for metadata
Everything else is built essentially from these parts.
November 18, 2014

5. Structures to organize objects
“Groups”
“/” (root)
3-D array
“/TestData”
lat | lon | temp
----|-----|-----
12 | 23 | 3.1
15 | 24 | 4.2
17 | 21 | 3.6
Table
“Datasets”
Raster image
This shows that you can mix objects of different types according to your needs. Typically, there will be metadata stored with objects to indicate what type of object they are.
Like HDF4, HDF5 has a grouping structure. The main difference is that every HDF5 file starts with a root group, whereas HDF4 doesn’t need any groups at all.
palette
Raster image
2-D array
November 18, 2014

6. Why use HDF5? Challenging data: Software solutions:
Application data that pushes the limits of what can be addressed by traditional database systems, XML documents, or in-house data formats.
Software solutions:
For very large datasets, very fast access requirements, or very complex datasets.
To easily share data across a wide variety of computational platforms using applications written in different programming languages.
That take advantage of the many open-source and commercial tools that understand HDF5.
Enabling long-term preservation of data.
November 18, 2014

7. Who uses HDF5? Examples of HDF5 user communities
Astrophysics
Astronomers
NASA Earth Science Enterprise
Dept. of Energy Labs
Supercomputing centers in US, Europe and Asia
Financial Institutions
NOAA
Manufacturing industries
Many others
For a more detailed list, visit
November 18, 2014

8. Topics What's up with The HDF Group? Library Update Tools update
Library development in the works
Other activities
November 18, 2014

9. Brief History of HDF
1987 At NCSA (University of Illinois), a task force formed to create an architecture-independent format and library:
AEHOO (All Encompassing Hierarchical Object Oriented format)
Became HDF
Early NASA adopted HDF for Earth Observing System project
1990’s
1996 DOE’s ASC (Advanced Simulation and Computing) Project began collaborating with the HDF group (NCSA) to create “Big HDF” (Increase in computing power of DOE systems at LLNL, LANL and Sandia National labs, required bigger, more complex data files).
“Big HDF” became HDF5.
1998 HDF5 was released with support from DOE Labs, NASA, NCSA
2006 The HDF Group spun off from University of Illinois as non-profit corporation
November 18, 2014

10. The HDF Group Established in 1988 The HDF Group owns HDF4 and HDF5
18 years at University of Illinois’ National Center for Supercomputing Applications
8 years as independent non-profit company, “The HDF Group”
The HDF Group owns HDF4 and HDF5
HDF4 & HDF5 formats, libraries, and tools are open source and freely available with BSD-style license
Currently employ 37 FTEs
Looking for more developers now!
November 18, 2014

11. The HDF Group Mission To ensure long-term accessibility of HDF data through sustainable development and support of HDF technologies.
November 18, 2014

12. Goals of The HDF Group
Maintain and evolve HDF for sponsors and communities that depend on it
Provide support to the HDF communities through consulting, training, tuning, development, research
Sustain the company for the long term to assure data access over time
November 18, 2014

13. The HDF Group Services Helpdesk and Mailing Lists Priority Support
Available to all users as a first level of support:
Priority Support
Rapid issue resolution and advice
Consulting
Needs assessment, troubleshooting, design reviews, etc.
Training
Tutorials and hands-on practical experience
Enterprise Support
Coordinating HDF activities across departments
Special Projects
Adapting customer applications to HDF
New features and tools
Research and Development
November 18, 2014

14. Members of the HDF support community
NASA – Earth Observing System
NASA – CGNS Project
DOE – Exascale FastForward w/Intel
DOE – projects w/LBNL, ANL, LLNL, SNL
NSF – projects w/CH2M Hill, Consortium for Ocean Leadership, Arizona Geological Survey
ITER – project with General Atomics
A leading U.S. aerospace company
European Light Sources - Dectris, Diamond Light Source and ESRF
Priority Support Customers
“In kind” Support
November 18, 2014

15. Income Profile – 2013/2014 Total Income: $4.1 million
November 18, 2014

16. Heavily used at Supercomputer Centers
Data collected at NERSC using ALTD (Automatic Library Tracking Database Infrastructure) that tracks all libraries linked into an application at compile time, as well as the libraries used at run time, by intercepting the linker (ld) and the job launcher (aprun, or mpirun).
This graph shows the library usage on Edison at NERSC from March to August
It can be seen HDF5, parallel HDF5, and netcdf-4 with HDF5 are some of the heavily used libraries.

17. Recent Success Story Performance of VPIC-IO on Bluewaters
I/O Kernel of a Plasma Physics application
56 GB/s I/O rate in writing 5TB data using 5K cores with multi-dataset write optimization
VPIC-IO kernel running on 298,048 cores
~10 Trillion particles
291 TB, single file
1 GB stripe size and 160 Lustre OSTs
52 GB/s
53% of the peak performance

18. Topics What's up with The HDF Group? Library Update Tools Update
Library development in the works
Other activities
November 18, 2014

19. Where We’ve Been Release 1.0 Release 1.2.0 - Oct, 1999
First “prototype” release in Oct, 1997
Incorporated core data model: datatypes, dataspaces & datasets and groups
Parallel support added in r1.0.1, in Jan, 1999
Release Oct, 1999
Added support for bitfield, opaque, enumeration, variable-length and reference datatypes.
Added new ‘h5toh4’ tool
Lots of polishing
Performance optimizations
November 18, 2014

20. Where We’ve Been Release 1.4.0 - Feb, 2001 Release 1.6.0 - July, 2003
Added Virtual File Driver (VFD) API layer, with many drivers
Added ‘h4toh5’, h5cc tools, XML output to h5dump
Added array datatype
F90 & C++ API wrappers
Performance optimizations
Release July, 2003
Generic Property API
Compact dataset storage
Added ‘h5diff’, ‘h5repack’, ‘h5jam’, ‘h5import’ tools
November 18, 2014

21. Where We’re At Now Release 1.8.0 - Feb, 2008
Features to support netCDF-4
Creation order indexing on links and attributes
Integer-to-floating point conversion support
NULL dataspace
More efficient group storage
External Links
New H5L (links) and H5O (objects) APIs
Shared Object Header Messages
Unicode-8 Support
Anonymous object creation
New tools: ‘h5mkgrp’, ‘h5stat’, ‘h5copy’
CMake build support
Performance optimizations
November 18, 2014

22. Repository Statistics
HDF5 Library Source Code, Lines of Code by Date
November 18, 2014

23. Repository Statistics
HDF5 Library Source Code, Lines of Code by Release
November 18, 2014

24. Software Engineering in HDF5
We spend considerable effort in ensuring we produce very high quality code for HDF5.
Current efforts:
Correctness regression testing
Nightly testing of >60 configurations on >20 machines
Performance regression testing
Applying static code analysis – Coverity, Klocwork
Memory leak detection – valgrind
Code coverage – coming soon
November 18, 2014

25. Performance Regression Test Suite
November 18, 2014

26. Where We’ll Be Soon Release 1.10 - Overview Beta release ~ mid 2015
Stopped adding major features, fleshing out our current efforts now
Major Efforts:
Improved scalability of chunked dataset access
Single-Writer/Multiple Reader (SWMR) Access
Improved fault tolerance
Initial support for asynchronous I/O
Virtual Object Layer
November 18, 2014

27. Where We’ll Be Soon Release 1.10 - Details
New chunked dataset indexing methods
Single-Writer/Multiple-Reader (SWMR) Access
Improved Fault Tolerance
Journaled Metadata Writing
Ordered Updates
Page-aligned and buffered metadata access
Persistent file free space tracking
F2003 Support
Compressed group information
High-level “HPC” API
Collective I/O on multiple datasets
Metadata broadcast between processes
Performance optimizations
Avoid file truncation
November 18, 2014

28. Where We Might Get To Release 1.10 - Maybe?
Full C99 type support (long double, complex, boolean types, etc)
Support for runtime file format limits
Improved variable-length datatype storage
Basic support for asynchronous I/O
Expanded Virtual File Driver (VFD) interface
Lazy metadata writes (in serial)
Virtual Object Layer
Abstraction layer to allow HDF5 objects to be stored in any container
Allows application to break “all collective” metadata modification limit in current HDF5
November 18, 2014

29. Where We’re Not Going
We’re not changing multi-threaded concurrency support
Keep “global lock” on library
Will use asynchronous I/O heavily
Will be using threads internally though
November 18, 2014

30. Topics What's up with The HDF Group? Library Update Tools update
Library development in the works
Other activities
November 18, 2014

31. Tool activities in the works
New tool – h5watch
Display changes to a dataset, metadata and raw data
Improved code quality and testing
Tools library: general purpose APIs for tools
Tools library currently only for our developers
Want to make it public so that people can use it in their products
November 18, 2014

32. Topics What's up with The HDF Group? Library Update Tools update
Library development in the works
Other activities
November 18, 2014

33. “The best way to predict the future is to invent it.” – Alan Kay
HDF5 in the Future
“The best way to predict the future is to invent it.” – Alan Kay
November 18, 2014

34. Plans, Guesses & Speculations: HPC
Improve Parallel I/O Performance:
Continue to improve our use of MPI and parallel file system features
Reduce # of I/O accesses for metadata access
Collective Read/Write of metadata
Support for compression in parallel
Collective access mode only
Support asynchronous parallel I/O
Support Single-Write/Multiple-Reader (SWMR) access in parallel
November 18, 2014

35. Plans, Guesses & Speculations: Safety
Improve Journaled HDF5 File Access:
Journal raw data operations
Allow multi-operation journal transactions to be created by applications
Support fully asynchronous journal operations
Enable journaling for Parallel HDF5
November 18, 2014

36. Plans, Guesses & Speculations: Threadsafety
Focus on asynchronous I/O, instead of improving multi-threaded concurrency of HDF5 library:
Library currently thread-safe, but not concurrent
Instead of improving concurrency, focus on heavily leveraging asynchronous I/O
Use internal multi-threading where possible
November 18, 2014

37. Plans, Guesses & Speculations: Grab-bag
Improve data model:
Shared dataspaces
Attributes on dataspaces and datatypes
Improve raw data chunk cache implementation
More efficient storage and I/O of variable-length data, including compression
November 18, 2014

38. Topics What's up with The HDF Group? Library Update Tools update
Library development in the works
Other activities
November 18, 2014

39. Virtual Object Layer
The VFL is implemented below the HDF5 abstract model:
deals with blocks of bytes in the storage container
does not recognize HDF5 objects nor abstract operations on those objects.
The VOL would be layered right below the API to capture the HDF5 model:
November 18, 2014

40. Exascale FastForward Research
Intel, EMC & The HDF Group recently completed a research project for exascale storage research and prototyping:
Using HDF5 data model and interface as top layer of next generation storage system for future exascale systems
Laundry list of new features to prototype in HDF5
Asynchronous I/O, transactions, end-to-end consistency checking, query/index features, Analysis Shipping, python wrappers, etc.
November 18, 2014

41. Autotuning and Performance Tracing
Why?
Because the dominant I/O support request at NERSC is poor I/O performance, many/most of which can be solved by enabling Lustre striping, or tuning another I/O parameter
Scientists shouldn’t have to figure this stuff out!
Three Areas of Focus:
Evaluate techniques for autotuning HPC application I/O
File system, MPI, HDF5
I/O Modeling to narrow down the autotuning space.
Record and Replay HDF5 I/O operations
November 18, 2014

42. Autotuning HPC I/O
Goal: Avoid tuning each application to each machine and file system
Create I/O autotuner library that can inject “optimal” parameters for I/O operations on a given system
Apply to precompiled application binaries
Application can be dynamically linked with I/O autotuning library
No changes to application or HDF5 library
Tested with several HPC applications already:
VPIC, GCRM, Vorpal
Up to 100x performance improvement, compared to system default settings
November 18, 2014

43. References
[1] "Taming Parallel I/O Complexity with Auto-Tuning", Babak Behzad, Huong Vu Thanh Luu, Joseph Huchette, Surendra Byna, Prabhat, Ruth Aydt, Quincey Koziol, Marc Snir. Supercomputing (SC) 2013
[2] "Improving parallel I/O autotuning with performance modeling", Babak Behzad, Suren Byna, Stefan Wild, Prabhat, and Marc Snir. International Symposium on High-Performance Parallel and Distributed Computing (HPDC) 2014
November 18, 2014

44. Recording and Replaying HDF5
Goal:
Extract an “I/O Kernel” from application, without examining application code
Method:
Dynamically link library at run-time to record all HDF5 calls and parameters in “HDF5 I/O trace file”
Create parallel replay tool that uses recording to replay HDF5 operations
November 18, 2014

45. Recording and Replaying HDF5
Benefits:
Easy to create I/O kernel from any application, even one with no source code
Can use replay file as accurate application I/O benchmark
Can move replay file to another system for comparison
Can autotune from replay, instead of application
Challenges:
Accurate parallel replay is difficult
Reference:
"Automatic Generation of I/O Kernels for HPC Applications", Babak Behzad, Hoang-Vu Dang, Farah Hariri, Weizhe Zhang, and Marc Snir. Parallel Data Storage Workshop (PDSW) 2014
November 18, 2014

46. Codename “HEXAD”
Excel is a great frontend with a not so great rear ;-)
We’ve fixed that with an HDF5 Excel Add-in
Let’s you do the usual things including:
Display content (file structure, detailed object info)
Create/read/write datasets
Create/read/update attributes
Plenty of ideas for bells an whistles, e.g., HDF5 image & PyTables support
Send in* your Must Have/Nice To Have feature list!
Stay tuned for the beta program *

47. Restless About HDF5/REST

48. HDF Server REST-based service for HDF5 data
Reference Implementation for REST API
Developed in Python using Tornado Framework
Supports Read/Write operations
Clients can be Python/C/Fortran or Web Page
Let us know what specific features you’d like to see. E.g. VOL REST Client Plugin

49. HDF Server Architecture

50. HDF Compass “Simple” HDF5 Viewer application
Cross platform (Windows/Mac/Linux)
Native look and feel
Can display extremely large HDF5 files
View HDF5 files and OpenDAP resources
Plugin model enables different file formats/remote resources to be supported
Community-based development model

51. Compass Architecture

52. You tell us! Community Outreach
Work with HPC community to serve their needs:
Focus on high-profile applications or “I/O kernels” and remove HDF5 bottlenecks discovered
You tell us!
November 18, 2014

53. Thank You! Questions & Comments?
November 18, 2014