The NASA Standards Process for Earth Science Data Systems Richard Ullman, NASA Yonsook Enloe, SGT Inc IGARSS 2010

Technology Diffusion 1NASA Earth Science Data Systems Standards - IGARSS - August, % 0% Adoption Time Source: Geoffrey Moore, Crossing the Chasm, 1999 modified after Everett Rodgers, 1962 InnovativePragmatic Technology DemonstrationMission Reliability/Stability InnovatorsEarlyEarlyLateLaggards AdoptorsMajorityMajority NASA’s Chasm

Standards Diffusion Chasm between: early adopter - technology for strategic advantage early majority - pragmatic focus. A practical path across chasm … “Community-led” successful practice in specific community broader community adoption community-recognized “standards” Community-led relies on: trusted endorsements strong leadership “whole product” 2NASA Earth Science Data Systems Standards - IGARSS - August, 2010

ESDS Standards Process – Seeking Pragmatic Recommendations A specification or practice is recommended as a standard … Only after practices have been shown to: (1) have demonstrated implementation and (2) benefit to operation will they be endorsed for preferential use. Ideas come from innovators or mission planners and are tempered by the significant demands of writing an RFC. Review process permits adoption only after “significant” community endorsement. THE RFC PROCESS PROVIDES PRAGMATIC REFERENCES 3NASA Earth Science Data Systems Standards - IGARSS - August, 2010

Crossing our Chasm – ESDS and the Pragmatic Decadal Survey SPG has been successful as a pragmatic voice: NASA HQ has endorsed SPG recommendations for Decadal Survey Mission use. Mission planners have embraced SPG as subject matter experts. NASA HQ recognize the benefit of choosing standards that are already recommended by the community The frustrating thing about standards from a HQ perspective is that everyone seems to want a different one. Program-focused HQ planners want agency investments to have broad return. i.e, for data: effective use for the primary mission and multiple secondary uses Decadal survey mission planners recognize the benefit of choosing standards that are already recommended by NASA. The frustrating thing about standards from a mission is that there are so many to choose from. Mission-focused pragmatic planners want to make the right choice. Standards that work and standards that won’t get them in hot water with their stakeholders. 4NASA Earth Science Data Systems Standards - IGARSS - August, 2010

Decadal Survey [2] SPG process has engendered: Greater confidence that mission products will be interoperable with heritage and current NASA data holdings Stakeholder community benefits because their preferences and opinions are available to the mission planner Lower barriers to entry and use of NASA data by external discipline communities within NASA and outside NASA because community standards are used. NASA Earth Science Data Systems Program is actively evolving the Earth Science data “system of systems” architecture for the Decadal Survey missions and beyond 5NASA Earth Science Data Systems Standards - IGARSS - August, 2010

Tier 1 Decadal Survey Missions Tier 1 Decadal Survey missions are the first group of missions to be formulated since NASA chartered the SPG Soil Moisture Active Passive (SMAP) Mission Ice, Cloud, and Land Elevation Satellite 2 (ICESat 2) Climate Absolute Radiance and Refractivity Observatory (CLARREO) Deformation, Ecosystem Structure, and Dynamics of Ice (DESDynI) The Tier 1 missions are actively working with the SPG in order to explicitly assure that the mission data systems solutions draw on the consensus-built lessons of agency experience. Tier 1 missions have a level 1 requirement to use SPG-endorsed standards 6NASA Earth Science Data Systems Standards - IGARSS - August, 2010

Crossing our Chasm – ESDS and Innovative Technologists It is a challenge to encourage technology demonstration projects to use RFC process. Technologist are often more interested in the technology puzzle rather than in practical dissemination. Early adopters are often more interested in strategic advantage rather than wide use. System Engineering focused on mission requirements makes “outreach” of writing RFC for community good “out of scope” “Whole Product” may be missing. Considerable systems engineering is still required when abstract standard is to be embedded into a mission implementation. 7NASA Earth Science Data Systems Standards - IGARSS - August, 2010

Process Basics 3 types of reviews and all reviews can be done simultaneously – some reviewers perform multiple reviews. Technical specification review Usefulness for purpose review Operational readiness If the proposed standard has been adopted by an another standards organization, then do not need a technical specification review For a mature defacto standard, we will perform a subset of the three types of reviews. May not need an operational readiness review or usefulness for purpose review The SPG will determine which reviews will be conducted for each proposed candidate standard. Can be a subset. Encourage strong community leader – essential to the process 8NASA Earth Science Data Systems Standards - IGARSS - August, 2010

Responsibilities Community Leader Identify someone in their community who will document standard according to SPG guidelines. Work with the community to get an extended review of the proposed standard. SPG Assign “RFC editor” to advise on RFC document. Publish and publicize RFC Assign “TWG”, technical working group to organize community review and evaluate responses. Recommend action to NASA HQ. 9NASA Earth Science Data Systems Standards - IGARSS - August, 2010

Endorsed Standards and Tech Notes Standards Data Access Protocol (DAP) V2.0 OGC Web Map Service V HDF 5 HDF EOS 5 NetCDF Classic GCMD DIF ICARTT Technical Notes OGC Web Map Service V1.3 AURA Guidelines Backtrack Orbit Search Algorithm Interoperability Between OGC CS/W and WCS Protocols Lessons Learned Regarding WCS Server Deisgn and Implementation Creating File Format Guidelines: The Aura Experience 10NASA Earth Science Data Systems Standards - IGARSS - August, 2010

Near Term & Long Term Future RFCs RFCs Under Review CF Metadata Conventions ECHO Metadata Model Mapping HDF5 to DAP2 NetCDF-4/HDF5 File Format Tech Areas for Future RFCs Web Services (e.g. REST, SOAP,..) Catalog services (e.g. OGC CSW,…) Metadata standards (e.g. ISO 19115,…) Visualization standards (e.g. KML,..) File Formats (e.g. NetCDF4,..) Interface standards (e.g. DAP v4, …) Data modeling Data Fusion 11NASA Earth Science Data Systems Standards - IGARSS - August, 2010

BACK-UP 12NASA Earth Science Data Systems Standards - IGARSS - August, 2010

Decadal Survey Background At NASA’s request, the National Research Council (NRC) looks out ten or more years into the future and prioritize research areas, observations, and notional missions to make those observations NRC completed its first decadal survey for Earth science in Jan 2007 at the request of NASA, NOAA, and USGS. For the next decade, the decadal survey identified 15 missions for NASA and 3 for NOAA including 1 joint NASA/NOAA The 17 total (NASA & NOAA) missions are presented as the result of prioritization methodology designed to achieve a robust, integrated program. The 15 missions for NASA are presented in 3 time-phased blocks. The 17 missions form a minimal, yet robust, observation component of an Earth information system that is capable of addressing a broad range of societal need. 13NASA Earth Science Data Systems Standards - IGARSS - August, 2010

Process Diagram 14 Proposed Standard Community Review Recommendation SPG Evaluate Proposed Standard Implementations and Community Response TWG Evaluate Proposed Standard Stakeholders Initial Screening Initial review of the RFC Provide RFC submission support Form TWG; set schedule Recommended Standard RFC Review Questions: Technical Specification Operational Readiness Suitability for Use TWG Technical Note Technical Note Reject NASA Earth Science Data Systems Standards - IGARSS - August, 2010 SPG

NASA ESDS RFC List ESDS-RFCTitleRevClassStatus ESDS-RFC-001Charter of the ESDS Standards Process Group2NoteFinal ESDS-RFC-002The ESDS Standards Process2NoteFinal ESDS-RFC-003Instructions to Authors2NoteFinal ESDS-RFC-004The Data Access Protocol -- DAP StandardFinal ESDS-RFC-005OpenGIS ® Web Map Service Version 1.31NoteFinal ESDS-RFC-006OpenGIS ® Web Map Service Version StandardFinal ESDS-RFC-007HDF 51StandardFinal ESDS-RFC-008HDF EOS 51StandardFinal ESDS-RFC-009Aura Guidelines Technical Note1NoteFinal ESDS-RFC-010Backtrack Orbit Search1NoteFinal ESDS-RFC-011NetCDF Classic1StandardFinal ESDS-RFC-012GCMD Directory Interchange Format (DIF)1StandardFinal 15NASA Earth Science Data Systems Standards - IGARSS - August, 2010

NASA ESDS RFC List ESDS-RFCTitleRevClassStatus ESDS-RFC-013GCMD Service Entry Resource Format (SERF)0.1StandardProposed ESDS-RFC-014Interoperability between OGC CS/W and WCS Protocols0.1NoteFinal ESDS-RFC-015Provenance within Data Interoperability Standards.01NoteProposed ESDS-RFC-016Lessons Learned Regarding WCS Design and Implementation0.1StandardFinal ESDS-RFC-017Mapping HDF5 to DAP20.1NoteProposed ESDS-RFC-018Creating File Format Guidelines - The Aura Experience0.1NoteFinal ESDS-RFC-019ICARTT0.1StandardProposed ESDS- RFC-020 ECHO Metadata Standard0.1StandardProposed ESDS-RFC-021CF Metadata Conventions0.1StandardProposed ESDS-RFC-022NetCDF-4/HDF-5 File Format RFC0.1StandardProposed 16NASA Earth Science Data Systems Standards - IGARSS - August, 2010