Summary of the EOSDIS Reference Architecture ESIP Federation Meeting ESDSWG SPG July 22, 2010

Outline Background EOSDIS Architecture Framework – Science Data System Functions – Cross-cutting Functions Scenarios – Satellite Data – Field Campaign – Airborne Issues

3 EOSDIS Context Tracking & Data Relay Satellite (TDRS) Instrument Teams and Science Investigator-led Processing System (SIPS) EOS Spacecraft Direct Broadcast/ Direct Readout Stations Direct Broadcast (DB) EOS Data and Operations System (EDOS) Data Processing White Sands Complex (WSC) EOS Polar Ground Stations Internet (Search, order, distribution) Research Education Value-Added Providers Interagency Data Centers International Partners Earth System Models Decision Support Systems Distribution and Data Access Data Acquisition Flight Operations, Data Capture, Initial Processing, Backup Archive Data Transport to Data Centers/ SIPSs Science Data Processing, Data Mgmt, Interoperable Data Archive, & Distribution NASA Integrated Services Network (NISN) Mission Services EOS Operations Center (EOC) Mission Control ESMO ESDIS EOSDIS Data Centers

EOSDIS Facilities 4 Data centers, collocated with centers of science discipline expertise, archive and distribute standard data products produced by Science Investigator-led Processing Systems (SIPSs) ASF DAAC SAR Products Sea Ice, Polar Processes SEDAC Human Interactions in Global Change LP DAAC Land Processes & Features PO.DAAC Ocean Circulation Air-Sea Interactions ASDC Radiation Budget, Clouds, Aerosols, Tropo Chemistry ORNL DAAC Biogeochemical Dynamics, EOS Land Validation GES DISC Atmos Composition & Dynamics, Global Modeling, Hydrology, Radiance LAADS/ MODAPS Atmosphere OBPG Ocean Biology & Biogeochemistry GHRC Hydrological Cycle & Severe Weather CDDIS Crustal Dynamics Solid Earth NCAR, U of Col. HIRDLS, MOPITT, SORCE GSFC GLAS, MODIS, OMI, OBPG LaRC CERES, SAGE III GHRC AMSR-E, LIS JPL MLS, TES San Diego ACRIM NSIDC DAAC Cryosphere, Polar Processes SIPSs Key Data Center

Purpose Purpose of the EOSDIS Reference Architecture – Documents the way EOSDIS does business, in all of its variations – Enables discussion with other communities and agencies on earth science data system development – It provides a framework for the insertion of community developed systems into the core system – Informs new missions of the current processes – Serves as a starting point for a future-looking Earth Science Data System Architecture

Discipline and mission specific data systems Only community- specific standards Only directory functions to point and link users to various data systems Multi-site access and data inter-use cumbersome Improved access to data with 90s technology Cross-system search and order access via interoperability data model Site-specific interfaces at DAACs/SIPs Common distribution format (HDF); other formats also supported <1990 mid-1990s Evolution of EOSDIS Data System Features Technology, Funding, Lessons Technology, Funding, Lessons 6

Support for high data volumes, ambitious performance requirements, high availability/reliability Integrated core infrastructure plus loosely coupled elements from community Common data model –Automated metadata creation and ingest –No need for cross-site metadata translation –FGDC standards compliance Expanded set of software tools and services Flexible options for supporting or interoperating with external data sources Coexistence of heterogeneous, distributed data providers/ information partners Optimal set of core standards Support for community- specific standards Reusable software components Exploitation of Service Oriented Architecture On-line archives and cross-system service invocation Ease of innovation and technology infusion More services to meet community demands Late 90s to presentNear-Future Evolution of EOSDIS Data System Features Technology, Funding, Lessons Technology, Funding, Lessons Technology, Funding, Lessons 7

Core and Community Capabilities - Characteristics CORECOMMUNITY Projects Subject to Programmatic Review Projects Competitively Selected Substantive NASA Oversight ‘Light Touch’ Oversight w/Significant Community Involvement Tight Integration of Data System Tools, Services and Functions Community-based Tools and Services Loosely-Coupled Employ Well Established Information Technologies Employ ‘Edgy’ or Emerging Technologies 8

EOSDIS Architecture Framework

Satellite Data Scenario #1 FunctionEOSDIS ApproachDescription 1)Level Zero Processing EDOS Level 0 data handling system Data captured from satellite-based instruments enters EOSDIS via the EOS Data and Operations System (EDOS) Level 0 data handling system 2)Science Processing Science Investigator-led Processing Systems (SIPS) The Science Investigator-led Processing Systems (SIPSs) collect Level 0 data and process these products into higher level products. 3)Archive ECS, Data Center specific systems The EOSDIS data centers, many of which are also Distributed Active Archive Centers (DAACs), perform the functions of ingest, storage, data management and access. 4)Discovery ECHO/WIST/Data Center specific systems The EOSDIS provides a metadata handling function by which the data system keeps track of its vast data collection and data center unique search and order systems. 5)NetworksNISN, Internet2 Effective access to EOSDIS depends on the end-to-end network connectivity between users and geographically distributed EOSDIS Data Centers. 6) Data System Standard Interfaces Interface Definition, Verification and Validation EOSDIS fosters the development of standards used within science data processing systems. The Hierarchical Data Format (HDF), was selected as the data format to be used for archiving and distributing data products for EOS instruments. 7) Performance Monitoring ESDIS Metrics System, Annual User Surveys The EMS supports the ESDIS project management by collecting and organizing a variety of metrics from the distributed data centers and other data providers (such as the GCMD and ESDIS-sponsored websites). 8) Systems and Sustaining Engineering ESDIS Systems Management Process EOSDIS is managed by the ESDIS Project which oversees the planning and funding for the EOSDIS system delegating local authority to individual data centers to implement the system needed to satisfy their discipline community.

AMSR-E Data Flow TLSCF Science Software I&T SIPS-GHCC PO.DAAC JAXA-EOC L1A Product Generation SIPS-RSS L2A Product Generation L2B and L3 Product Generation (science, metadata, qa, ph, subsets, browse, DAPs) L2 and L3 DAPs L2A algorithm NSIDC DAAC Data Archive and Distribution Level-1 Level-2 and Level-3 Level-1 NASA EDOS RBD & PDS Ground Stations L2B and L3 Algorithms Level-0 Product Delivery Record Server (PDRS) Algorithm Teams OceanSea Ice Rain Land Snow Data NASA FDS GBAD

Descriptions of EOSDIS data holdings can be found at the archiving data center. See AMSR-E at the National Snow and Ice Data Center (NSIDC). Select the Order Data option.

Users have a variety of ways to access EOSDIS data. For example select the WIST option.

Warehouse Inventory Search Tool (WIST) The EOS ClearingHouse (ECHO) is a metadata catalog of NASA’s EOS data and a registry for related data services (e.g. reformatting, pattern recognition). ECHO’s catalog contains more than 2800 data sets held at 12 EOSDIS data centers. WIST, the Warehouse Inventory Search Tool, is the primary web-based client for discovering and ordering cross-discipline data from all of ECHO’s metadata holdings. WIST allows users, including those without specific knowledge of the data, to search science data holdings, retrieve high-level descriptions of data sets and detailed descriptions of the data inventory, view browse images, and submit orders via ECHO to the appropriate data providers. The WIST search-and-order tool has the following features: Search capability—Allows a user to explore the list of data sets or granules returned by a search by viewing their temporal coverage, spatial coverage, attributes (metadata), related documents (guide search), and browse images. Order function—Allows the user to select data for ordering, choose packaging information, enter ordering information (such as shipping address), place an order, and view order status. Reference:

Other Scenarios Field Campaign (In situ) scenario Ground and Ocean-based scenario Airborne scenarios Ancillary scenarios Satellite scenarios – US – International

Examples of EOSDIS Data Center Implementations EOSDIS Data Centers Level Zero Processing Science Processing ArchiveDiscovery ASF DAACGAP ECHO/GCMD/URSA CDDISNA IGS/ IGN CDDISECHO/GCMD/Website GHRCEDOSSIPSGHRCECHO/GCMD/HyDRO GES DISCEDOS SIPS/ S4PM S4PA ECHO/GCMD/ MIRADOR/OPeNDAP LP DAACEDOSSIPSECSECHO/GCMD/GloVis ASDCEDOSSIPS ECS/ ANGe ECHO/GCMD/ANGe MODAPS/ LAADS EDOSSIPSLAADSECHO/GCMD/ Website NSIDCEDOSSIPSECSECHO/GCMD OBPG EDOS/ OBPG OCDPSOBPG OBPG/GCMD/SEADAS/SEABAS S OPENDaP, others ORNLNA ORNLECHO/GCMD/ Mercury PO.DAAC SIPSDMAS ECHO/GCMD/POET/OPeNDAP / FTP SEDACNA ECHO/GCMD/ SEDAC Information Gateway

EOSDIS Reference Architecture Plan both a document & website EOSDIS Reference Architecture Draft – Under discussion and drafting components DAACs Committee of NASA data system planners – Scenarios Developed Satellite Data Field Campaign Airborne Basic Website in place However, the reference architecture only documents EOSDIS functions in a very broad approach – Does not take the place of official NPR documentation – Each new Mission requires direct communication and distinct documentation

Example Science Interfaces & Documentation Networks ESDIS Metrics System ESDIS Project Users Flight Project ECHO DMR ADURD SIPSs MRT Mission Readiness Test Plan IPA ICD W/A ICD IRD ICD OA ATBD ICD A subset of interfaces are shown for mission operations ESMO Project ESDIS Project ICD Science Team Data Centers EDOS

Interoperability on Many Levels Process Methodology – Configuration control – Business processes – Policy processes Software Reuse – S4PA, subsetting Standards – Metadata model – Data formats – Interface controls EOSDIS ClearingHouse EOSDIS Metrics components

EOSDIS Business Model Budget preparation and tracking of actual expenses is performed across the Project. Components of EOSDIS submit yearly budgets and work plans Budgets and actuals are tracked monthly against phasing plans and schedules Metrics are collected for both technical and fiscal management Coordination across the architecture allows for cost savings

Specific Areas of cost tracking CategoryDefinition Mission and Science Support Quality assurance and verification, metadata development, data architecture and definition, Science guide documentation and science team interaction, metadata catalog maintenance User ServicesWeb support (content development and maintenance) and outreach (NASA booth), tutorial development, Answering phones and s OperationsCode execution and job scheduling, processing, distribution, Ingest Management, metrics operations & troubleshooting EngineeringWeb and tool development, evolution of existing systems, System development, sustainment, database development InfrastructureConfiguration mgmt, systems admin, DBA IT SecurityApplications and network security, contingency planning AdministrationProject and resource management Administrative assistance

Conclusion The EOSDIS Reference Architecture is still under draft by many staff members – Thanks to all of you!!! Any questions? Comments? Feel free to contact me: – Jeanne Behnke –

AMSR-E – Advanced Microwave Scanning Radiometer - Earth Observing System JAXA – National Space Development Agency (Japan) TLSCF – Team Leader Science Computing Facility (AMSR-E) RSS – Remote Sensing Systems (company) GHCC – Global Hydrology and Climate Center (MSFC, Huntsville, Ala) DAAC – Distributed Active Archive Center PO.DAAC – Physical Oceanography DAAC NSIDC – National Snow and Ice Data Center (NSIDC) DAP – Delivery Algorithm Package GBAD – Ground based attitude determination FDS – Flight Dynamics System RBD – Rate buffered data PDS – Production data set