1. EO Dataset Preservation Workflow Data Stewardship Interest Group WGISS-37 Meeting Cocoa Beach (Florida-US) - April 14-18, 2014

2. Slide 2 Preserved Dataset consists of: 1.Data Records: these include raw data, Level 0 data and higher-level products, browses, auxiliary and ancillary data, calibration and validation data sets, and Descriptive Information. 2.Processing Software: this includes all the software used in the product generation, quality control, and the product visualization and value adding tools. 3.Representation Information (RepInfo): needed to make the Data Records understandable to the Designated Community. This includes among others mission architecture, products specifications, instruments characteristics, algorithms description, Cal/Val procedures, mission/instruments performances reports, quality related information, etc. It is necessary to ensure data understandability and usability. Preserved Data Set Content (PDSC) document describes in detail the content of a “Preserved Dataset” Definitions (1/2)

3. Definitions (2/2) EO Data Curation E2E Consolidation: aims at the generation of single, consistent, consolidated and validated “Data Records”. Preservation: aims at the creation of a single, consistent, consolidated and validated “Preserved Dataset” and at ensuring its long term integrity, discovery, accessibility and usability. Value Adding : aims at establishing and increasing the value of one or more “Preserved Datasets” over their lifecycle, at favouring their exploitation also through the combination with other Data Records and at extending their user base.

4. E2E Consolidation Aims at the generation of single, consistent, consolidated & validated “Data Records”. It is tailored for each Mission/Sensor according to its specific preservation/curation requirements; Consists of all the activities needed for:

5. Preservation Aims at the creation of a single, consistent, consolidated and validated “Preserved Dataset” and at ensuring its long term integrity, discovery, accessibility and usability. It is focused on an individual or on a multi-mission Dataset and tailored according to its specific preservation/curation requirements It consists of all activities needed to ensure “Preserved Dataset” integrity over time and to optimize its (re)use in the long term (e.g. through metadata and catalogue improvement, processor improvement for algorithm and/or AUX data changes and related (re)processing, linking and improvement of context/provenance information).

6. Value Adding Aims at establishing and increasing the value of “Preserved Datasets” over their lifecycle, at favouring their exploitation also through the combination with other Data Records, and at extending their user base. It includes, among others, activities like primitive features extraction, exploitation improvement, data mining, coordination and management of Data Time Series and Collections (e.g. from similar sensor family) in support to specific applications and in cooperation with international partners. It might lead to an update of the preservation/curation requirements.

7. Slide 7 E2E Consolidation Value Adding Data Records Preservation Preserved Dataset DATA CURATION

8. Slide 8 Preservation Workflow STEPS & Data Lifecycle

9. Introduction The Preservation Workflow defines a recommended set of actions/steps to be implemented for the preservation of a “Dataset” with the objective to ensure and optimize its (re)use in the long term The set of actions/steps need to be tailored for each specific mission according to its specific preservation/curation requirements It can be applied to Historical, Current and Future missions The result of the Preservation Workflow application consists of: Preserved datasets discoverable and accessible to users and for any additional curation/valorisation activity A set of documents describing the selected preservation strategy and implementation plans and activities for each individual mission dataset

10. Slide 10 LTDP Common Guidelines Specific E2E Consolidation process (e.g. EO) PDSC Inventory Table Preservation Workflow Inputs/outputs INPUT OUTPUT Preserved Dataset Data Records SW & Associated knowledge (Representation Information) Preservation Workflow documentation Preservation strategy; Implementation plans; Completeness Analysis; Risk assessment ….

11. Slide 11 Preservation Workflow STEPs PDSC Tailoring and Inventory Table filling (including dependencies: Inventory Data Model) Dataset Appraisal, Definition of Designated Community & Preservation Objective (with preservation/curation requirements) Tailored PDSC Consultation with Designated Community Update of PDSC & Inventory Table Ingestion and Archiving, Master Inventory and Catalogue Population Implementation of E2E Tailored Consolidation process (Data Records) Collection of Representation Information and Processing Software Tailoring of specific E2E Consolidation process (Data Records) Risk & Cost Assessment, Preservation & Cost Planning, Implementation Dissemination & Web Configuration

12. Slide 12 Lifecycle Model for Dataset Curation and Preservation

13. Slide 13 Preservation Workflow Flow chart

14. Slide 14 Data requires Preservation Issue Purge Alert Reject Preservation Request external internal no yes Make Preservation Decision/Recommendation and specify Preservation /Curation Requirements Tailoring of specific E2E Consolidation process (Data Records) PDSC Tailoring and Inventory Table filling (including dependencies: Inventory Data Model) Apply Records Appraisal Dataset ready for Ingest Mission Dataset Preservation Request Definition of Designated Community & Preservation Objective Tailored PDSC Consultation with Designated Community yes no Implementation of E2E Tailored Consolidation process (Data Records) Collection of Rep Info & Processing Software Consolidation Area Update of PDSC & Inventory Table Archive & Ingestion, Master inventory and Catalogue Population Dissemination & Web Publication INGESTION & DATA ACCESS Risk and Cost Assessment, Preservation & Cost Planning Risk Monitoring, Datasets Management and Curation INIZIALIZATION CONSOLIDATION IMPLEMENTATION/ OPERATIONS

15. Slide 15 Data requires Preservation Issue Purge Alert Reject Preservation Request external internal no yes Make Preservation Decision/Recommendation and specify Preservation /Curation Requirements Tailoring of mission specific E2E Consolidation process (Data Records) PDSC Tailoring and Inventory Table filling (including dependencies: Inventory Data Model) Apply Record Appraisal Dataset ready for Ingest Mission Dataset Preservation Request Definition of Designated Community & Preservation Objective Tailored PDSC Consultation with Designated Community yes no Implementation of E2E Tailored Consolidation process (Data Records) Collection of Rep Info & Processing Software Consolidation Area Update of PDSC & Inventory Table Archive & Ingestion, Master inventory and Catalogue Population Dissemination & Web Publication INGESTION & DATA ACCESS ECSSMISSIONSPHASESECSSMISSIONSPHASES Phase A (feasibility) Phase B (Preliminary def) Phase C (detailed def) Phase D (dev) Phase E (ops) Risk Monitoring, Datasets Management and Curation Risk and Cost Assessment, Preservation & Cost Planning

16. Slide 16 Preservation objective; Designated community; Preliminary Definitions; PDSC; Data analysis and appraisal; Configurations; Cost & Risk assessment; Preliminary Agreement; Legal/contractual aspects; Archival plan; … Historical Missions LTDP TEAM Historical Dataset (Various Formats/Media) Initialization (e.g. Appraisal) Master Data Records Documents collection SW & Tools collection LTDP TEAM Consolidation & Implementation

17. Slide 17 Future Missions LTDP TEAM Tailored LTDP Procedures, Standards & Requirements RAW DATA NRT SITES & PRODUCTION FACILITIEs Dissemination Operations Scenario Periodic Adherence Verification for final E2E Certification

18. Slide 18 Current Missions Tailored LTDP Procedures, Standards & Requirements Mission Operations LTDP TEAM Initialization (e.g. Appraisal) Periodic Adherence Verification for final E2E Certification MISSION TEAM RAW DATA NRT SITES & PRODUCTION FACILITIEs Dissemination Preservation objective; Designated community; Preliminary Definitions; PDSC; Data analysis and appraisal; Configurations; Cost & Risk assessment; Preliminary Agreement; Legal/contractual aspects; Archival plan; … LTDP TEAM

19. Slide 19 Thank you for your attention !!! Comments ? Questions ?