RFCs for HDF5 and HDF-EOS5 Status Update Richard Ullman Chair ES-DSWG - Standards November 29, 2006

Overview What is the NASA Earth Science Data System Working Group, Standards Process Group? HDF in the SPG Invitation to comment.

Motivation One initiative after another has stressed the need for interoperability standards. Many standards initiatives, both formal and grass roots have put forward specifications or demonstrated various ways to enable access to data. NASA, or NASA funded projects are often in the forefront of these activities. However, NASA participation in a standards development activity does not imply that NASA projects endorse the results of that activity. Need a way to identify the “standards that work” in the context of NASA’s Earth science research and applications data systems under the operational loads that NASA typically experience.

EOSDIS Evolution 2015 Vision Tenets Vision Tenet Vision 2015 Goals Archive Management  NASA will ensure safe stewardship of the data through its lifetime.  The EOS archive holdings are regularly peer reviewed for scientific merit. EOS Data Interoperability  Multiple data and metadata streams can be seamlessly combined.  Research and value added communities use EOS data interoperably with other relevant data and systems.  Processing and data are mobile. Future Data Access and Processing  Data access latency is no longer an impediment.  Physical location of data storage is irrelevant.  Finding data is based on common search engines.  Services invoked by machine-machine interfaces.  Custom processing provides only the data needed, the way needed.  Open interfaces and best practice standard protocols universally employed. Data Pedigree  Mechanisms to collect and preserve the pedigree of derived data products are readily available. Cost Control  Data systems evolve into components that allow a fine-grained control over cost drivers. User Community Support  Expert knowledge is readily accessible to enable researchers to understand and use the data.  Community feedback directly to those responsible for a given system element. IT Currency  Access to all EOS data through services at least as rich as any contemporary science information system.

An international comprehensive, coordinated and sustained Earth observation system Group on Earth bservations A Shared Vision for Earth Observation Articulated by 34 Nations in an Earth Observation Summit (July 31, 2003) Comprehensive: meeting the needs of a variety of science and applications disciplines Coordinated: multinational satellite, suborbital and in situ observing capabilities strategically coordinated via agreed standards and data exchange Sustained: long-term, continued financial and in-kind support from funding authorities

Insights from “SEEDS” Analysis Interoperability does not require homogeneous systems, but rather coordination at the interfaces. Management can judge success based upon program goals rather than dictated solutions. – example: degree of interoperability rather than use of particular data format. Communities of practice have solutions. Published practices that demonstrate benefit can grow … – successful practice in specific community – broader community adoption – community-recognized “standards”

The Request For Comment Process Modeled after example of Internet “IETF RFC”. Tailored for responsiveness to NASA. Proposed standards are documented as specifications acording to SPG guidelines and submitted by practitioners within the NASA community. The Standards Process Group forms a Technical Working Group (TWG) to coordinate evaluation. – What does “implementation” of this specification mean in the context of NASA Earth Science Data Systems? – What constitutes successful “operational” experience? The community is invited by means of announcement to comment on the specification and particularly to address questions formulated by the TWG. The TWG also identifies key stakeholders that are likely to have particular experience with the technology and solicits their opinion. The TWG reports to the SPG and the SPG makes recommendations for final status of the RFC.

Proposed Standard Recommendation SPG Develop review criteria Evaluate known experiences Evaluate community response TWG Respond to TWG Stakeholders Initial Screening Initial review of the RFC Provide RFC submission support Form TWG; set schedule The Review Process Recommended Standard RFC Stakeholder/TWG Review Specification Operational Readiness Suitability for Use

The Endorsement Process SPG will send recommended standards to NASA HQ Program Executive for Data Systems with the following: – Strengths/ Weaknesses – Applicability/ Limitations Endorsement is briefed to HQ Earth Science Steering Committee HQ will disseminate endorsement through NASA CIOs and to general announcement to community “Core” standards will be required of NASA Earth Science programs, projects and awards. “Community” standards will be applied at discretion of program or studies managers.

RFC criteria: Are there components (technologies practices) that if documented and more widely used would promote: – Easier sharing or exchanging of data among distributed partners and users. – Distributed systems development and sharing of software and technical expertise. – Reducing the cost of developing or maintaining a system. – Increasing the use of scientific data products and bringing more funding. – Interoperability and enhancing innovation, collaboration, and computing performance. For identified technologies/practices, Is there a community of use that: – Has experience in implementation and operation. – Has leadership necessary to promote the advantage of wider use.

HDF5 and HDF-EOS RFCs in the NASA SPG Process Related RFCs handled by same TWG One review request for both RFCs Community members asked to comment on either or both of the RFCs Target community includes – NASA instrument teams – NASA data systems managers – Earth science investigators – Developers of software tools for Earth science data Comment period for “implementation experience” April 26 – July 31

HDF RFC “Implementation” Reviewers Total of 17 people commented on one or both RFCs Backgrounds include: – Data providers (AIRS, HIRDLS and MLS instrument teams) – NASA data systems managers (ORNL and NSIDC DAACs, TSDIS/GPM, OMI SIPS ) – Developers of software tools for Earth science data (RSS – developers of IDL, UAH Subsetting team, Unidata, MathWorks, NetCDF, HDF Group (reviewing HDF-EOS5), HDF-EOS team (reviewing HDF5)) – Earth science data users (CERES instrument team, DSCOVR project) All using HDF and/or HDF-EOS software from original developers, perhaps with minor modifications Tools used include FORTRAN, IDL, Mathematica, Scilab, HDFLook, HDFView, Matlab Operating systems include Windows (32 and 64 bit), Linux (32 and 64), Solaris (32 and 64), Mac (PPC & Intel), SGI, IRIX, HP-UX

Summary of comments from Implementation Review phase HDF5 comments from 10 reviewers were generally positive – 9 of 10 recommending endorsement HDF-EOS5 comments from 8 reviewers were somewhat mixed, still overall positive – 4 recommending endorsement – 2 neutral – 2 recommending improvements to RFC before endorsement Primary concerns: – Complexity of data model, API and software libraries – Availability of tools to improve ease of use – Long term support

Other Comments Received Comments on HDF in general from 4 reviewers – General comments, not addressing implementation questions – Not specific to either flavor of HDF being considered Maybe these should be considered for “operations” rather than “implementation” phase 3 of 4 of these comments were negative – Both Science Data Users fell in this group

Summary of Comments So Far from Usability Survey Total of 13 people have commented on HDF5 12 are software tools developers (among other roles); 1 is a data user only; 7 classify themselves as both 5 are international US respondents from military, national labs, high performance computing, NOAA, commercial companies Should HDF be a recommended format for NASA Earth Science data? – 10 selected “strongly agree” – 1 selected “disagree” – 1 selected “neutral” – 1 selected “not applicable”