The HDF Group July 8, 2014HDF 2014 ESIP Summer Meeting HDF Product Designer Aleksandar Jelenak, H. Joe Lee, Ted Habermann The HDF Group 1

HDF 2014 ESIP Summer Meeting HDF Data Producer’s Conundrum 2 HDF Features Interoperability Mission Requirements HDF Product Designer Datatypes Groups Attributes Dimension scales Compression Chunking Scale/offset Etc. Conventions Metadata Software netCDF Science objectives Data processing, discovery & distribution Data documentation User engagement, preparedness, feedback

HDF 2014 ESIP Summer Meeting HDF Product Designer Background Originated from the HDF5 Earth Science (H5ES) Builder/Creator toolset developed by Jeffrey Lee for the ICESat-2 mission Built using the MySQL, Apache, PHP software stack Runs as a desktop application Under review by The HDF Group to generalize the concept 3

HDF 2014 ESIP Summer Meeting What is HDF Product Designer? 4 Application for organizing HDF5 file content to data and metadata Built-in support for conventions Create and re-use parts of file content Collaborative design process Generate HDF5 template file, or code that produces the designed file for several programming languages HDF5 template has structure and metadata

HDF 2014 ESIP Summer Meeting Design Process 5 Create and organize content: Files, groups, datasets, attributes Many parameters of datasets and attributes can be set Re-use any part of content elsewhere in the same or another file All the work is saved in a database

HDF 2014 ESIP Summer Meeting After a File is Designed… 6 Export description of the file’s content into a tab-delimited text file This description file can be opened with Excel Content description can be edited Description file can be imported back into the Product Designer

HDF 2014 ESIP Summer Meeting After a File is Designed… 7 A template HDF5 file can be generated Code can be produced for reading and writing designed file for: FORTRAN IDL The generated write code can be edited to fill in real data

HDF 2014 ESIP Summer Meeting Typical HDF Product Designer Workflow 8 Design a product file Export description file Stakeholders review/edit content Import description file Generate template file and/or code Testing

HDF 2014 ESIP Summer Meeting Expected Benefits Easy way to design HDF file content Interoperability issues addressed during the design phase End-to-end data flow testing can start earlier Reduced risk for costly redesign of processing pipeline based on stakeholder feedback later Team collaboration Promotion of design best practices across mission teams 9

HDF 2014 ESIP Summer Meeting Future Work 10 Local (desktop) or central (web site) app? Reduce app’s required software footprint MySQL already replaced with SQLite Support Python, MATLAB for template file code Web service for generating HDF5 templates and code Generate portable/free production code using Python and h5py Re-use content info from old products: Import HDF4 Content Map.

HDF 2014 ESIP Summer Meeting Future Work Support for compound datatypes Vdata in HDF4 maps to attribute or dataset? Convention compliance checkers CF, HDF-EOS, etc. Interoperability checkers netCDF-4, visualization tools, etc. 11

HDF 2014 ESIP Summer Meeting Local vs. Central Application? Local: No need for Internet connection Richer/more responsive user interface No user accounts management Requires a developer’s computer Central: Only web browser required Better collaboration workflow Only one place to update the app Dual mode – like software? 12

HDF 2014 ESIP Summer Meeting Looking for Partners New missions Projects upgrading from HDF4 to HDF5 Projects reprocessing their data to increase interoperability Use cases Contact: 13

HDF 2014 ESIP Summer Meeting You can improve it, too! 14 Create your new HDF5 product at / Please submit your brilliant ideas at

HDF 2014 ESIP Summer Meeting Acknowledgement 15 This work was supported by Subcontract number under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.