Aura HDF-EOS File Format Guidelines: Overview and Status Cheryl Craig

Aura HDF-EOS File Format Guidelines: Overview and Status 2 December 2, 2005 What is Aura? NASA atmospheric chemistry satellite launched July 15, 2004 Four instruments with separate instrument teams (HIRDLS, MLS, OMI and TES) Developed a set of Guidelines on top of HDF/HDF-EOS for storing each instrument’s data in a single common format

Aura HDF-EOS File Format Guidelines: Overview and Status 3 December 2, 2005 Why discuss Aura Approach? Aura experience demonstrates that it is possible for developers at diverse locations creating multiple products from several instruments to agree on important common data product formats Common formats substantially simplify end user’s tasks when combining data observed by one or more instruments Common formats simplify the task of developing both the data sets themselves and of tools for manipulating them. Provides lessons learned from a successful effort to develop common standards over multiple instruments at distributed locations Provides insight as to time and effort to produce a set of data guidelines

Aura HDF-EOS File Format Guidelines: Overview and Status 4 December 2, 2005 What are the Aura Guidelines? A set of Guidelines explicitly defining format of similar data products from multiple instruments. A set of naming conventions, units, metadata definitions (via HDF/HDF-EOS attributes) and data organization conventions.

Aura HDF-EOS File Format Guidelines: Overview and Status 5 December 2, 2005 Goals of Standardization Allow easier sharing and use of data –Data products which are the same between instruments have common structure and definitions –Identify and include attributes which aid in developing user software –Allow use of a single HDF/HDF-EOS library for all products –Provide consistent and accessible definition of data formats applicable to multiple instruments and products Reduce development effort and support reuse by enabling application of software to multiple instruments and products

Aura HDF-EOS File Format Guidelines: Overview and Status 6 December 2, 2005 Key Elements for developing a standard Identify a community of data producers and users who could benefit from standardization Get buy-in from every member of community Identify common elements which could benefit from standardization – these are the only fields you need to standardize Develop a consistent structure for describing data formats that can be applied even to unstandardized elements

Aura HDF-EOS File Format Guidelines: Overview and Status 7 December 2, 2005 HDF/HDF-EOS Elements standardized Dimensions and ordering of dimensions for each field Names of fields (includes capitalization and spacing) Data types for each field Attributes for each field, their types and definitions –Title, Units, MissingValue, UniqueFieldDefinition Units for each field (eliminates need to do conversions when using data from multiple instruments)

Aura HDF-EOS File Format Guidelines: Overview and Status 8 December 2, 2005 Aura Guidelines Approval Process Strawman produced based on community requirements Edited via extensive use of –Named authors – required to respond –Silent authors – could respond if desired –Document passed to members for extensive editing Controversial items were brought to individual teams for discussion by Guidelines group member(s) Telecons/DSWG breakout meetings held for items which required discussion Major releases of document voted upon by named authors

Aura HDF-EOS File Format Guidelines: Overview and Status 9 December 2, 2005 Results of Aura Standards We have simplified the use and development process for both data producers and data users Provides defined approaches for software. Once the initial definition effort was complete developers could concentrate on science issues and not worry about how to format the data Assures that features that support subsequent software development are included Users can work with multiple parameter data sets with consistent geolocation, altitude and parameter definitions thus simplifying or eliminating much of the resampling and data conversion steps often required.

Aura HDF-EOS File Format Guidelines: Overview and Status 10 December 2, 2005 Lessons Learned Communicate early, before individual team’s decisions on data files have been made Exchange data sets early on to assure common understanding of the standards Include software engineers and scientists in discussion group Be willing to compromise – will never match your “perfect data set” – remember benefits Group leader must be firm, no issues left unresolved Requires specific commitment of effort by every team and a dedicated coordinator to develop, publish and maintain documents Document needs to be detailed – every item not spelled out is subject to interpretation and alternate implementation

Aura HDF-EOS File Format Guidelines: Overview and Status 11 December 2, 2005 Where is Aura going from here? Extending Aura Guidelines to cover Level 3 data (gridded and zonal mean data) Creating a Technical Note for NASA’s Earth Science Data System Working Group –Details of the Aura standard –Will provide information on how we created the guidelines

Aura HDF-EOS File Format Guidelines: Overview and Status 12 December 2, 2005 Minimal Standardization Encourage basic standards for HDF data files –Would suggest a set of attribute names attached to data fields which users could use or not as they see fit –Candidates would be Units, Scale, Offset, Title, etc. –Scale and Offset would include definition of how to apply them

Aura HDF-EOS File Format Guidelines: Overview and Status 13 December 2, 2005 Aura Document Location Current Authors: Cheryl Craig – NCAR (HIRDLS) David Cuddy – JPL (MLS) Scott Lewicki – JPL (TES) Pepijn Veefkind – KNMI (OMI) Peter Leonard – SSAI (OMI) Al Fleig – PITA (OMI) Paul Wagner – JPL (MLS) On Next Release: Christina Vuu – JPL (MLS)