NASA’s EOSDIS – Long Term Archive Infrastructure and Processes Presented to the CEOS WGISS September 28, 2017 Dawn R. Lowe Hampapuram (Rama) Ramapriyan Chris Lynnes ESDIS Project, NASA/Goddard Space Flight Center

Topics NASA Earth Science Data Archives - overview Archive Infrastructure and processes NASA Data Stewardship Approach to Data Preservation & Access EOSDIS Content Preservation Specification

Typical EOSDIS Archive Hardware Architecture Blade Server Blade Server Controller PostgreSQL Server Metadata Controller Blade Server Disk Storage Area Network Tape Storage Area Network Physical Storage Array Physical Storage Array Backup Storage Small File Backup This shows a typical EOSDIS archive hardware architecture currently hosted on on-premise. File systems are often separated according to the purpose and size of a file. Small files are separated from large data files for performance reasons, and backed up on disk instead of tape. Automated tape libraries are used for backing up large data files. Disk systems are on one Storage Area Network, with Tape Libraries separated on the Tape SAN. A PostGRES database server maintains information about the data entities stored on the system. The metadata controllers maintain low-level (filesystem) information about the files stored in each SAN. Note that this shows only the hardware connectivity, not the software interfaces. Logical Storage Data Pool File Systems Small File Archive Browse Archive Archive Cache Automated Tape Library science files metadata staging

Data Stewardship Ensure that the data and information content are reliable, of high quality, easily accessible, and usable for as long as they are considered to be of value. Involves communications with data producing projects throughout their lifecycles Essential to plan for long term preservation

Preservation Implies… Understandability Bits Discoverability & Accessibility PRESERVE Usability Readability Reproducibility of Results

Preserving Bits Checksums while transferring between subsystems Regular media migration Raw (Level 0) data from satellites held at back-up archive physically distant from DAACs Product generation software held at the DAACs and Science Investigator-led Processing Systems Guards against catastrophic loss Raw data and higher level products are backed-up at the DAACs as well – for efficiency Periodic assessment of risk of data loss and impact on users given the back-up approach being used at the DAACs for different datasets and the expected time for recovery in case of loss of the primary copy

Risk Assessment Code Matrices Study of Data Loss Risk/User Impact conducted in 2012 Periodic updates are made by DAACs Each DAAC assesses its holdings and enters counts that map to Data Loss Risk, User Impact Example of a RAC Matrix shown below Data Loss Risk 5 48 28 4 7 3 463 381 2 300 352 1506 2814 1 719 225 193 860 User Impact 

DAAC Hardware: Technology Refresh Refresh every 3-7 years, with a “rolling wave” process, coordinated with capacity increases Refresh Process: Develop specifications based on projected data volumes, performance requirements, etc. Procure hardware (quotes, purchase orders) Take delivery (receive, tag, inventory) Install hardware* Install and configure system* Decommission and excess old hardware *Requires manufacturer or vendor training if refreshing with a new brand of hardware

Discoverability, Accessibility, Readability Standard metadata are critical for discoverability of data Processing software automatically generates metadata at individual file level Metadata repository is constantly populated Common Metadata Repository (CMR) Unites collection level and file level metadata Provides a source of unified, high-quality and reliable Earth Science metadata across NASA’s Earth science data holdings Standard Data Formats, e.g. HDF, NetCDF HDF is a self-documenting formatting system Flexible structure for data producer to define “profile” HDF library facilitates writing and reading Need to maintain library for future users

Standards-based Packaging of Earth Observation Data Dataset Interoperability Working Group Recommendations EOSDIS Community Best Practices Climate-Forecast Conventions (CF) Recognizable Coordinates network Common Data Form (netCDF) Common Data Model Robust Tool Support Hierarchical Data Format (HDF) Self-contained Metadata

Dataset Interoperability Working Group (DIWG) Recommendations Detailed guidance on use of CF and related conventions Includes rationale for each recommendation Examples: “We recommend that packing attributes (i.e., scale_factor and add_offset) be employed only when data are packed as integers.” We recommend that datasets in grid structures include a Time dimension, even if Time is degenerate (i.e., includes only onevalue) for the cases when the entire grid has one time range or time stamp https://earthdata.nasa.gov/standards/dataset-interoperability-recommendations-for-earth-science

HDF4 Maps for Long Term Preservation Problem: Data Formats that are accessed only via Application Program Interface (API) are vulnerable to eventual deprecation and de-support of the API. Solution: Use the API to generate an external XML file that “maps” out the byte layout of the file; Test by writing read programs without using the API XML Map File HDF Data File

Understandability, Usability & Reproducibility Algorithm Theoretical Basis Documents (ATBDs) Product information pages, guides, answers to frequently asked questions (FAQs), forums Usability Information on fitness for purpose Accuracy assessments, validation and data quality documentation Reproducibility Source code and/or software specification documents Versions of datasets or the means of regenerating them when they result in peer-reviewed publications

Preservation Content Specification (PCS) Covers 8 categories of content plus a checklist: Preflight/Pre-Operations: Instrument/Sensor characteristics including pre-flight/pre-operations performance measurements; calibration method; radiometric and spectral response; noise characteristics; detector offsets Science Data Products: Raw instrument data, Level 0 through Level 4 data products and associated metadata Science Data Product Documentation: Structure and format with definitions of all parameters and metadata fields; algorithm theoretical basis; processing history and product version history; quality assessment information Mission Data Calibration: Instrument/sensor calibration method (in operation) and data; calibration software used to generate lookup tables; instrument and platform events and maneuvers

Preservation Content Categories (cont.) 5. Science Data Product Software: Product generation software and software documentation 6. Science Data Product Algorithm Input: Any ancillary data or other data sets used in generation or calibration of the data or derived product; ancillary data description and documentation 7. Science Data Product Validation: Records, publications and data sets 8. Science Data Software Tools: product access (reader) tools. Checklist: “metadata” about the above 8 categories showing how and where items in each category are preserved https://earthdata.nasa.gov/standards/preservation-content-spec

Sources of Content Calibration Team Mission logs Instrument Teams / PI’s Instrument Developer/ Manufacturer Data gathering project (e.g., flight project) Product Generation Support Teams (SIPSs) DAACs Calibration Team Mission Operations Validation Team Preflight/ Pre-Operations Science Data Products Science Data Product Documentation Mission Data Calibration Science Data Product Software Science Data Product Algorithm Input Science Data Product Validation Science Data Software Tools Ancillary data sources (e.g., NOAA) Level 0 Data Mission logs

Use of PCS in NASA to-date DAACs work with Instrument Teams, with higher priority for instruments at or near end-of-life, using PCS as checklist: UARS (Sept. 1991) Earth Probe/TOMS (July 1996) AMSR-E (EOS Aqua – May 2002) ICESat-1 (Jan. 2003) HIRDLS (EOS Aura – July 2004) LIS (TRMM – Nov. 1997) Artifacts called for in PCS have been gathered for several of the above, organized by categories and archived - e.g., see http://disc.sci.gsfc.nasa.gov/Aura/additional/documentation/hirdls-preservation-documents)

Standards NASA would like to see a broad international standard identifying preservation content: NASA’s PCS is a good starting point as are CEOS Long Term Data Preservation documents ISO 19165 - “Geographic Information - Preservation of digital data and metadata” – Final edits completed in November 2016 Ramapriyan participated on ISO 19165 Project Team as a U.S. expert ISO has approved a New Work Item Proposal for developing 19165-2, a standard specific to Earth Observation data

Back-Up

Backup Strategy vs Data Loss RAC Data Loss Risk Assessment Code Yes 2 3 4 No 5   Offsite Campus Onsite

Restoration vs User Access RAC   User Access Risk Assessment Code Restore User Access Alternate Site none 3 4 5 <1 month 2 <2 weeks 1 <1 week <2 days <3 months >3 months Time to Restore from Backup at your Primary site