HDF Update Mike Folk The HDF Group HDF and HDF-EOS Workshop X November 29, 2006HDF

Nov. 29, 2006HDF Workshop X, Landover MD2 Outline Organizational info HDF Software Update Other Activities of Interest

Organizational info

Nov. 29, 2006HDF Workshop X, Landover MD4 “The HDF Group” = “THG” Founded Dec. 2006Went solo July 15, 2006 Non-profit

THG mission To support the vast community of HDF users and to ensure the sustainable development of HDF technologies and the ongoing accessibility of HDF-stored data.

Nov. 29, 2006HDF Workshop X, Landover MD6 The HDF Team Frank Baker Christian Chilan Peter Cao Vailin Choi Mike Folk Anne Jennings Barbara Jones Quincey Koziol James Laird Raymond Lu John Mainzer Matthew Needham Pedro Nunes Tammi O’Neill Elena Pourmal Binh-minh Ribler Randy Ribler Rishi Sinha Kent Yang And all those wonderful folks out there who contribute ideas, requests, bug reports, code, and support.

Nov. 29, 2006HDF Workshop X, Landover MD7 Who is supporting HDF? Organizations providing broad support NASA, DOE, Boeing Agencies supporting R&D (2006) NASA, NARA, DOE, NCSA, Agilent, Aberdeen Test Center, DD(X) Collaborators who make in-kind contributions Cactus, PyTables, NeXUS, CGNS, many others

HDF Software Update

HDF4 update

Nov. 29, 2006HDF Workshop X, Landover MD10 Platforms to be dropped Operating systems HPUX Crays SV1 and TS IEEE AIX 5.1 and 5.2 SGI IRIX Linux 2.4 Solaris 2.7, 2.8, 2.9 Windows 2000 MAC OSX 10.3 Compilers GNU C compilers older than 3.4 (Linux) Intel 8.* PGI V. 5.*, 6.0

Nov. 29, 2006HDF Workshop X, Landover MD11 Platforms to be added Systems MAC OSX 10.4 (Intel) Solaris 2.* on Intel Cray XT3 Windows 64-bit (?) Linux 2.6 HPUX IBM Power 5 Compilers g95 PGI V. 6.1 Intel 9.*

Nov. 29, 2006HDF Workshop X, Landover MD12 New features Configuration Switched to use F77_FUNC macro for better Fortran support (no hard-coded compilers anymore!) Support for shared libraries Library No hard-coded limit on number of opened files New APIs to control number of files opened by application Fortran support for SZIP compression

Nov. 29, 2006HDF Workshop X, Landover MD13 Bugs fixes Tools A lot of improvements to the hdp, hrepack, hdiff and hdfimport utilites based on users’ feedback Library Data corruption bug for several opened unlimited dimension SDSs Better handling of SDSs with duplicated names in SDgetdimscale and more

HDF5 update

Nov. 29, 2006HDF Workshop X, Landover MD15 No new releases! Focus on HDF5 release 1.8 HDF Alpha 5 release is available from: hdf.ncsa.uiuc.edu/HDF5/release/alpha/obtain518.html hdf.ncsa.uiuc.edu/HDF5/release/alpha/obtain518.html

Nov. 29, 2006HDF Workshop X, Landover MD16 Platforms to be dropped Operating systems HPUX MAC OS 10.3 AIX 5.1 and 5.2 SGI IRIX Linux 2.4 Solaris 2.8 and 2.9 Compilers GNU C compilers older than 3.4 (Linux) Intel 8.* PGI V. 5.*, 6.0 MPICH

Nov. 29, 2006HDF Workshop X, Landover MD17 Platforms to be added Systems Alpha Open VMS MAC OSX 10.4 (Intel) Solaris 2.* on Intel (?) Cray XT3 Windows 64-bit (32-bit binaries) Linux 2.6 BG/L Compilers g95 PGI V. 6.1 Intel 9.* MPICH MPICH2

New Features in HDF5 1.8

Nov. 29, 2006HDF Workshop X, Landover MD19 HDF5 1.8 new library features Datatype and dataspace features Serialized dataspaces and datatypes Ability to create data type from text description Integer to float conversions during I/O Revised exception handling during type conversion Compact storage for N-bit data types Offset+size storage filter, saving space “Null” dataspace – datasets with no elements Data transformation filter

Nov. 29, 2006HDF Workshop X, Landover MD20 HDF5 1.8 – new library features Group revisions Creation order access Compact groups – small groups take less space Large group storage improvements Intermediate group creation

Nov. 29, 2006HDF Workshop X, Landover MD21 HDF5 1.8 – new library features Link improvements External links -- can refer to objects in another file User defined links – apps create own kinds of links Attribute improvments Storage improvements for large numbers of attr Iterate or look up by creation order

Nov. 29, 2006HDF Workshop X, Landover MD22 HDF5 1.8 – new library features Support for Unicode UTF-8 character set Shared header info – duplicate header info shared, possibly saving space Metadata cache improvements – faster I/O on files with many objects Data transformation filter Stackable Virtual File Drivers Better UNIX/Linux portability

Nov. 29, 2006HDF Workshop X, Landover MD23 HDF5 1.8– new APIs New extendible error-handling API New APIs to copy objects between files fast Dimension scale model and API “HDFpacket” – API to read/write packets efficiently

HDF5 1.8 – backward and forward compatibility

Nov. 29, 2006HDF Workshop X, Landover MD25 HDF5 1.8 vs Differences between 1.8 vs Some file format changes Several new routines added Old APIs deprecated -- removed in later release Consequences Application requiring 1.8 format changes will write objects that library cannot read To exploit 1.8 changes, apps need to be rewritten

Nov. 29, 2006HDF Workshop X, Landover MD26 Principle of “Maximum file format compatibility” Unless instructed otherwise, the HDF5 library will write objects using the earliest version of the format possible for describing the information. Assures forward compatibility with the older versions whenever possible – objects in new files can be read with old libraries if those objects are “known” to the old libraries.

Nov. 29, 2006HDF Workshop X, Landover MD27 Example: Datatype header message Compound datatype encoding Version 1 used by and earlier encodes compound datatypes with explicit array fields Version 2 used for has a new encoding, reducing storage overhead for compound data By default writes compound data in format compatible with – 1.6.X libraries But if feature is requested, compound data created by will not be readable by earlier versions

Nov. 29, 2006HDF Workshop X, Landover MD28 HDF5 Forward Compatibility Format Can old libraries access files made by new library? Old library versions will read all objects in a file created by a newer library if objects are known to the old library API Can old applications link with the new library? Applications written to work with an older version of library will compile, link and run as expected with a newer version

Nov. 29, 2006HDF Workshop X, Landover MD29 HDF5 Backward Compatibility File Format Can new library access files made by old library? Newer version of the library will always read files created with an older version Library APIs Can new applications link with the older libraries? Application written for the newer version will compile and link with the older library unless new features are used

Nov. 29, 2006HDF Workshop X, Landover MD30 HDF5 Compatibility information Backward and forward compatibility issues faq/bkfwd-compat.htmlhttp://hdfgroup.org/HDF5/ faq/bkfwd-compat.html API changes from release to release doc_1.8pre/doc/ADGuide/Changes.htmlhttp://hdfgroup.org/HDF5/ doc_1.8pre/doc/ADGuide/Changes.html File Format changes doc/H5.format.htmlhttp://hdfgroup.org/HDF5/ doc/H5.format.html

Command line tools

Nov. 29, 2006HDF Workshop X, Landover MD32 New features for old tools h5dump Dump data in binary format h5diff Compare dataset regions Parallel h5diff (ph5diff) Compare two files in MPI parallel environment h5repack Efficient data copy using H5Gcopy() Able to handle big datasets

Nov. 29, 2006HDF Workshop X, Landover MD33 New HDF5 Tools h5copy Copies an group, dataset or named datatype from one location to another location Copies within a file or across files h5check Verifies an HDF5 file against the defined HDF5 File Format Specification h5stat Reports statistics about a file and objects in a file

HDF Java Products

Nov. 29, 2006HDF Workshop X, Landover MD35 HDFView changes Quality improvements for HDF-java package Full documentation of hdf-java object package Test suite for hdf-java object package Support 64-bit Java on Linux and Solaris Many new features, including Change font size easily Grab and move image Create new table (compound dataset) from template Filter out fill value for image creation -geometry option for very high resolution displays

Nov. 29, 2006HDF Workshop X, Landover MD36 Future work for Java Update HDF5 JNI APIs for HDF5 1.8 release Release HDFView 2.4 with bug fixes/new features with HDF5 1.8 release New GUI features dealing with table, image and animation Writing capability for HDF5-SRB model

Website Development for HDF-EOS Tools & Information Center

Nov. 29, 2006HDF Workshop X, Landover MD38 Website for HDF-EOS Tools THG now manages HDF-EOS web site Registered domain names: hdfeos.net/.org/.com Re-implemented major topic areas Re-designed interface Registered google search Will continue maintenance Phase two Host mailing list Support simple forum features

Nov. 29, 2006HDF Workshop X, Landover MD39 Website for HDF-EOS Tools

Other Activities of Interest

Performance R&D

Nov. 29, 2006HDF Workshop X, Landover MD42 HDF5 - PnetCDF performance comparison uP: Power 5 I/O performance of PnetCDF is comparable with parallel HDF5 when the libraries are used in similar manners.

Nov. 29, 2006HDF Workshop X, Landover MD43 PnetCDF4 - PnetCDF comparison I/O performance of parallel NetCDF4 is comparable with PnetCDF with about 15% slowness on average for the output of ROMS history file.

Nov. 29, 2006HDF Workshop X, Landover MD44 Collective I/O improvements HDF5 supports collective IO for non-regular selections Collective IO for chunked storage is not trivial. Non-regular selection performance optimizations: Added IO options to achieve good collective IO performance Added APIs for applications to participate in the optimization process See the poster

DOE Labs Sandia National Laboratory Lawrence Livermore National Laboratory

Nov. 29, 2006HDF Workshop X, Landover MD46 DOE ASC* and Others Support HDF5 on major systems at Sandia & Lawrence Livermore National Laboratories R&D efforts underway File recovery after a crash Very fast write speed – goal is 300 MB/sec Read-while-writing capability Java library and HDFView improvements * Advanced Scientific Computing project

Flight test

Nov. 29, 2006HDF Workshop X, Landover MD48 Flight test – collect, then process

Nov. 29, 2006HDF Workshop X, Landover MD49 Boeing HDF5 for flight test data Boeing 787 active archive 10 TB per flight-test day Must handle raw, real-time data High speed ingest, by “packet” Post-processing, by “time-history” Boeing High Level API’s HDFpacket – released with HDF5 1.8 HDFtime_history – new, open version likely

Product data STEP

Bioinformatics caacaagccaaaactcgtacaaCgagatatctcttggaaaaactgctcacaatattgacgtacaaggttgttcatgaaactttcggtaAcaatcgttgacattgcgacctaatacagcccagcaagcagaat Managing genomic data

C# HDF5 API for Agilent

Nov. 29, 2006HDF Workshop X, Landover MD53 Agilent C# project Why? Heavy use of C# at Agilent Compatibility with Matlab Other interest in HDF5 at Agilent What? Prototype API in C# for Windows XP Basic functions to create, open, close, read, write Limited datatypes, no partial I/O When? March 2007

Nov. 29, 2006HDF Workshop X, Landover MD54 HDF5 Software HDF File Tools & Applications HDF I/O Library C API FortranC++JavaC#

NetCDF 4

Nov. 29, 2006HDF Workshop X, Landover MD56 NetCDF 4 project Enhanced NetCDF-4 Interface to HDF5 Combine features of netCDF and HDF5 Take advantage of their separate strengths Collaboration between NCSA, THG, Unidata Currently in Alpha Release Waiting for beta release

Nov. 29, 2006HDF Workshop X, Landover MD57 NetCDF-4 Architecture HDF5 Library netCDF-4Library netCDF-3 Interface netCDF-3 applications netCDF-3 applicationsnetCDF-4applicationsnetCDF-4applications HDF5 applications HDF5 applications netCDF files netCDF files netCDF-4 HDF5 files HDF5 files Supports access to netCDF files and HDF5 files created through netCDF-4 interface

Nov. 29, 2006HDF Workshop X, Landover MD58 Archival formats Proposal to NOAA Scientific Data Stewardship program Will investigate use of OAIS “Archive Information Package” standard with HDF5 PI: Ruth Duerr (NSIDC) and Kent Yang OAIS: Open Archival Information System

Asymmetries between collecting and accessing data

Nov. 29, 2006HDF Workshop X, Landover MD60 Huge streams of data collected … To be accessed in little bits…

Nov. 29, 2006HDF Workshop X, Landover MD61 Challenge – efficient remote access How do we efficiently find and access data from distributed repositories, when the data are big and complex? Storage Resource Broker (SRB) Efficient access to HDF5 objects in repository OPeNDAP Powerful protocol for remote querying and subsetting of scientific data

Nov. 29, 2006HDF Workshop X, Landover MD62 Example – Storage resource broker Storage Resource Broker – repository for heterogeneous data collections Simplifies storage, query and access to massive amounts of scientific data Has data in HDF5, netCDF, other formats

Nov. 29, 2006HDF Workshop X, Landover MD63 Normal SRB configuration SRB Server HDF5 MCAT HDF5 File (whole file or a sequence of bytes) client

Nov. 29, 2006HDF Workshop X, Landover MD64 OPeNDAP-HDF5 project OPeNDAP Powerful protocol for remote querying and subsetting of scientific data Replaces direct file access with remote query and access Widely used in Earth Sciences

Nov. 29, 2006HDF Workshop X, Landover MD65 OPeNDAP – HDF5 Project A NASA ROSES NRA project Tasks HDF5-DAP2 server (now a prototype) HDF5-DAP4 server DAP4 to HDF5 conversion utility Investigate integrated DAP-aware HDF5 library

SQL Server and HDF5 with Microsoft

Nov. 29, 2006HDF Workshop X, Landover MD67 SQL Server and HDF5 Microsoft “dream environment for scientists” Combine data management, computing SQL Server 2005 solution Combine RDBMS with scientific analysis tools, together in one integrated system. HDF5 & other formats manage scientific objects

Nov. 29, 2006HDF Workshop X, Landover MD68 HDF5 in SQL server Entity Framework (EDM, eSQL, O-R mapping) HDF5 EDM model Visualization Libraries (MATLAB,…) HDF5 files Web Services (XML, REST, RSS) OLAP and Data Mining Reporting HDF 5 type HDF5 Index HDF5 FS blob HDF5 TVFs.NET Languages with Language Integrated Query SQL Server

Thank you all and Thank you NASA!

Acknowledgement This report is based upon work supported in part by a Cooperative Agreement with NASA under NASA NNG05GC60A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration.

Questions/comments?

Nov. 29, 2006HDF Workshop X, Landover MD72 Information Sources HDF website HDF5 Information Center HDF Helpdesk HDF users mailing list coming soon: