University of Illinois at Urbana-ChampaignHDF Mike Folk HDF-EOS Workshop IV Sept , 2000 HDF Update HDF

University of Illinois at Urbana-ChampaignHDF - 2 -Topics What is HDF? HDF community HDF4 –past and future work HDF5 –HDF5-HDF5 interoperability –Activities and growth –Tools and utilities –Focus for 2001

University of Illinois at Urbana-ChampaignHDF NCSA HDF Mission To develop, promote, deploy, and support open and free technologies that facilitate scientific data storage, exchange, access, analysis and discovery.

University of Illinois at Urbana-ChampaignHDF What is HDF? Format and software for scientific data Stores images, arrays, tables, etc. Emphasis on storage and I/O efficiency Free and commercial software support Emphasis on standards Users from many engineering & scientific fields

University of Illinois at Urbana-ChampaignHDF HDF People NCSA –QA, maintanance & support –Library development & performance –Tools development EOS & ASCI –Users, applications and tools –Performance Volunteers from all over –Users, applications & tools

University of Illinois at Urbana-ChampaignHDF Who is supporting HDF? NASA/ESDIS –Earth science applications, instrument data –All aspects of data management DOE/ASCI (Accelerated Strategic Computing Init.) –Simulations on massively parallel machines –Emphasis on parallel I/O performance, functionality NCSA –Grid, Vis, other R&D, user support Others –Applications, support, some R&D

University of Illinois at Urbana-ChampaignHDF - 7 -HDF4 User support, tools, maintenance, QA JHV (Java viewer) - two update releases HDF4 dumper (hdp) - many improvements Next HDF4 Release (Oct. 2000) –Bug fixes –Chunking for General Raster (GR) images –Unlimited dimensions for SDS –GIF HDF4 conversion tools

University of Illinois at Urbana-ChampaignHDF HDF4 Archiving New File Format and Specification Manual –Facilitate long term maintenance of HDF4 –Necessary for bug fixing, adding new features –Decreases learning curve for new developers –Available soon in PDF, HTML and PS formats

University of Illinois at Urbana-ChampaignHDF HDF5

University of Illinois at Urbana-ChampaignHDF The growth of HDF5 Users in 27 countries Users include –Government agencies –National labs –Companies –Universities

University of Illinois at Urbana-ChampaignHDF The growth of HDF5 Scientific fields –Astronomy, astrophysics, aerospace engineering –Geophysics, remote sensing, meteorology, –Oceanography, environmental Science –Informational Science –Medical Research (brain, cancer, biotech) –Product model data

University of Illinois at Urbana-ChampaignHDF Facilitating interoperability between HDF4 and HDF5 HDF4 to HDF5 mapping specification –“Mapping HDF4 Objects to HDF5 Objects” –Rules for mapping high level HDF4 objects to HDF5 How to describe HDF4 objects in HDF5 How to interpret HDF5 objects as HDF4 objects HDF4-to-HDF5 conversion software (later) Space Research, Inc. Explorer 1.1 –reads both HDF4 and HDF5

University of Illinois at Urbana-ChampaignHDF HDF5 Activities in 2000 HDF library release Fortran 90 & C++ API HDF5 Abstract Data Model XML Document Type Definition (DTD) HDF5 tools Support for users and application developers HDF5 Tutorial Searchable, printable documentation

University of Illinois at Urbana-ChampaignHDF Virtual file I/O (C only)  Perform byte-stream I/O operations (open/close, read/write, seek)  User-implementable (I/O to stdio, network, memory, etc.) Library internals Performs data transformations and other prep for I/OPerforms data transformations and other prep for I/O Configurable transformations (compression, etc.)Configurable transformations (compression, etc.) Structure of HDF5 Library Object API (C, Fortran 90, Java, C++)  Specify objects and transformation properties  Invoke data movement operations and data transformations

University of Illinois at Urbana-ChampaignHDF Tools & Utilities NCSA –Java wrapper for HDF5 –H5View: Java browser/Editor for HDF5 –H5gen: XML-to-HDF5 file generator (Java) –H5dump & H5ls Others –VisAD data adapter for HDF5 (Java toolkit) –HDF Inspector/Explorer –Open Data Explorer (IBM) –Ensight

University of Illinois at Urbana-ChampaignHDF H5View

University of Illinois at Urbana-ChampaignHDF H5View Java-based tool for browsing and editingJava-based tool for browsing and editing Display structure of fileDisplay structure of file Display content of objectsDisplay content of objects Create and delete objectsCreate and delete objects Modify values of objects and attributesModify values of objects and attributes

University of Illinois at Urbana-ChampaignHDF HDF5 XML DTD A flexible standard language that can describe an HDF5 file in precise detail –Datasets, dataspaces, datatypes –Groups and links, structure of the file –Values of the attributes and data

University of Illinois at Urbana-ChampaignHDF H5Gen: XML  HDF5 Reads an XML description of an HDF5 file Generates the corresponding HDF5 file Validates XML description vs. HDF5 DTD

University of Illinois at Urbana-ChampaignHDF Some uses of XML View HDF5 files using a web browser XMLtoHTML Java viewers,editors, other tools XML to Java Java to XML Description in XML using HDF5 DTD HDF5 file XML to HDF5 (H5gen) HDF5 to XML Generate, validate, reconstruct HDF5 files Data location services Catalog records in XML Archive XML documentation of metadata & structure

University of Illinois at Urbana-ChampaignHDF Focus areas for 2001 Support Terra & Aqua Get ready for Aura Support HDF-EOS 3 Enhance HDF5View Refine XML DTD & design tools around it HDF5 converters –HDF5  XML –HDF4-to-HDF5 –Others (e.g. GIF  HDF5)

University of Illinois at Urbana-ChampaignHDF Focus areas for 2001 Expand list of applications and users Facilitate access to other tools & software Get vendors on board Performance testing and tuning Extend API with ease-of-use functions Clusters and other new environments Implement a thread-safe version HDF5 advanced tutorial

University of Illinois at Urbana-ChampaignHDF Thank you! HDF website – HDF5 Information Center – HDF Helpdesk HDF users mailing list