“PRESERVATION DOMAIN” Data Stewardship Interest Group WGISS-42 Meeting – ESA-ESRIN, Frascati 19–22 September, 2016 Mirko Albani, European Space Agency ESA UNCLASSIFIED – For Official Use

Background and Purpose WGISS accomplishes its Data Preservation and Curation efforts through the Data Stewardship Interest Group (DSIG) Purpose of the activities: Enable the sharing of agency investigations, developments, experiences and lessons learned relating to EO data stewardship Draft common cross-agency best practices or guidelines of data stewardship for possible adoption by WGISS Sponsor technical exchanges and the conduction of Joint Activities and/or Pilot Projects on specific data stewardship topics Establish and maintain a CEOS “Data Purge Alert” service Contributing to GEO and Standardization activities ESA UNCLASSIFIED – For Official Use

DSIG sessions during this meeting Overview and Updates Associated Knowledge Preservation Best Practice CEOS Maturity Matrix Session: Report on Agencies Stewardship Activities Preservation Topics Discussion Landing Page & DOI for knowledge Glossary Extension Transcription Chains Thesaurus ESA UNCLASSIFIED – For Official Use

Data Stewardship Interest Group Overview & Updates ESA UNCLASSIFIED – For Official Use

Data Management Principles Task Force Feedback from GEO Data Management Principles Task Force (DMP-TF) ESA UNCLASSIFIED – For Official Use

GD-07: GCI Development Overview M.Albani & R. Moreno members of GD-07.03 task team ESA UNCLASSIFIED – For Official Use

Data Management Principles

Status of Data Management Principles Implementation Guidelines Data Management Principles Implementation Guidelines presented as info document at GEO-XII Plenary (Nov15): Description / explanation of each principle Guidance on implementation, with examples Resources implications for implementing each principle Suggested metrics measuring adherence to each principle Contribution from M.Albani and R.Moreno Public review process held in Q1 2016: Online survey and Review process led by IEEE (S. J. S. Khalsa) Document updated on the basis of survey results, reviewed at GEO WP Symposium (2-4 May) and GEO Program Board (July). Draft currently with GEO Secretariat: will be submitted for adoption at GEO-XIII Plenary (Nov) In rosso punto di domanda ESA UNCLASSIFIED – For Official Use

Data Management Principles Implementation Guidelines – Online Survey Harmonised feedback was provided by European Space Agencies in the frame of LTDP WG activities: Improvement of Data Quality Assurance aspects as only monitoring and assessment addressed Include long term planning and knowledge preservation aspects Include appraisal phase ESA UNCLASSIFIED – For Official Use

Data Management Principles Implementation Guidelines – Survey Results ESA UNCLASSIFIED – For Official Use

Data Management Principles Implementation Guidelines – Results DMPs implementation ESA UNCLASSIFIED – For Official Use

Data Management Principles Implementation Guidelines – Results Difficulty in DMP Implementation ESA UNCLASSIFIED – For Official Use

Data Management Principles Implementation Guidelines – Results DMP Implementation Challenges ESA UNCLASSIFIED – For Official Use

GEO GD07.03 3 Year Work Program – In Process Subtask Description: Develop Data Management Principles (DMP) Guidelines and Implementation Process 2017 Proposed Subtask Activities Propose topics, learning outcomes, and opportunities for conducting training and workshops centered around the DMP Principles. Identify additional flagships for demonstrating the DMP Implementation Guidelines (IG) and how existing flagships can implement DMPs. Identify criteria and process for DMP branding of archives implementing DMP IG. 2018 Proposed Subtask Activities Request suggestions for improving and adding to the DMP Implementation Guidelines. Request feedback for improving training and workshops. Revise DMP IG further to reflect feedback, evolving practices, and possibility of reviews. Revise proposed learning outcomes (developed during 2017?) to reflect feedback from participants. Convene training and workshops about the DMP IG ESA UNCLASSIFIED – For Official Use

GEO GD07.03 3 Year Work Program – In Process 2019 Proposed Subtask Activities Revise topics and learning outcomes for training and workshops, based on feedback. Create process to review candidate archives to receive DMP brand for implementing DMP IG Identify and invite candidate archives to apply for DMP brand for implementing DMP IG Initiate process of triennial update of DMP, or process of determining evaluation criteria ESA UNCLASSIFIED – For Official Use

Best Practices/Guidelines Data Stewardship Best Practices/Guidelines ESA UNCLASSIFIED – For Official Use

CEOS Data Stewardship Best Practices Document Tree Policy Documents Individual organizations' policies (stewardship, access, ...) EO Data Stewardship Cooperative Framework Applied to CEOS EO Space Data Sets Data Purge Alert Procedure & White Paper High level framework documents Applied to Technical Documents Preservation Workflow Glossary of Acronyms and Terms General guidelines and best practices Support Guidelines and best practices on specific topics Persistent Identifiers Best Practice EO Data Preservation Guidelines Preserved Data Set Content Associated Knowledge Preservation CEOS Maturity Matrix … Technical implementation procedures EO Data Set Consolidation Process …

Documents “Almost Done” and “Working in Progress” – Data Purge Alert White Paper Final version v1.0 circulated to WGISS-all. Recommended for adoption today

Data Purge Alert Procedure Status Purge Alert Procedure endorsement at CEOS Plenary in Nov 2015 Finalization of Purge Alert White Paper and material needed to ensure visibility of Purge Alert Procedure on CEOS and WGISS Web Sites Define approach to inform Commercial Data Owners of the existence of the procedure (data at highest risk, in particular old ones). Possibility for them to use the procedure. TO BE DONE ESA UNCLASSIFIED – For Official Use

Data Purge Alert Page in WGISS web site: already implemented

Data Purge Alert NEWS: to be implemented In case of Purge Alert notification, the WGISS Chair will inform all CEOS Agencies Principals and publish a ”Purge Alert News” on the CEOS and WGISS web pages. Main CEOS Web page: a flashing “News” link to the WGISS relevant web page WGISS Web Page: specific information on the purge alert notification

PURGE ALERT BY USGS USGS' intent to purge most of their Landsat MSS and TM film and would like to utilize the CEOS Purge Alert to identify organizations that may be interested in obtaining the film Further information awaited by USGS (appraisal information according to Purge Alert procedure) Data Records description List of available associated knowledge Results and documentation of a Data Record Appraisal if available Overview of constraints and conditions if any Recommendations concerning Data Records handling Point of Contact

Documents “Almost Done” and “Working in Progress” – Associated Knowledge Preservation DEDICATED SESSION ESA UNCLASSIFIED – For Official Use

Documents “Almost Done” and “Working in Progress” – CEOS Maturity Matrix DEDICATED SESSION ESA UNCLASSIFIED – For Official Use

Documents for adoption today Assessment of Drafting Status and Discussion Issue to DSIG for Review, comments and feedback Final Presentation and Formal Approval v Purge Data Alert White Paper v Final Issue to WGISS v REVIEW FINALIZATION WGISS#41 WGISS#42 03/16 04/16 05/16 06/16 07/16 08/16 09/16

Future documents timeline proposal Assessment of New Drafting Status and Discussion Issue to DSIG for Review Comments & Feedback Final Presentation and Formal Approval v v v Associated Knowledge Preservation Best Practices Final Issue to WGISS v REVIEW FINALIZATION WGISS#42 WGISS#43 09/16 10/16 11/16 12/16 01/17 02/17 03/17

Future documents timeline proposal Topic Presentation and Discussion Comments & Inputs from DSIG Assessment of New Drafting Status and Discussion Issue to DSIG for Review Comments & Feedback Final Presentation and Formal Approval v v v v v Final Issue to WGISS v WGISS#42 WGISS#43 REVIEW FINALIZATION WGISS#44 09/16 10/16 11/16 12/16 01/17 02/17 03/17 04/17 05/17 06/17 07/17 08/17 09/17 10/17 CEOS Maturity Matrix

Document Management Table Document name Custodian Document ID File name Current version Date Status Planned release date EO Data Preservation Cooperative Framework (DSIG Plan) M. Albani, ESA None Presentations Updated at each WGISS meeting Q3/2016 Preservation Workflow I. Maggio, ESA CEOS/WGISS/DSIG/PW 1.0 03/2015 APPROVED EO Data Stewardship Glossary CEOS/WGISS/DSIG/GLOS EO-DataStewardshipGlossary 1.1 09/2015 EO Data Preservation Guidelines CEOS/WGISS/DSIG/EODPG EO Data Set Consolidation Process R. Cosac, ESA CEOS/WGISS/DSIG/GEODSCP Generic Earth Observation Data Set Consolidation Process Preserved Data Set Content R. Leone, ESA CEOS/WGISS/DSIG/EOPDSC EO Preserved Data Set Content Persistent Identifiers Best Practice K.Molch, DLR CEOS/WGISS/ DSIG/PIDBP CEOS Persistent Identifier Best Practices 01/2016 Data Purge Alert White Paper CEOS/WGISS/DSIG/DPA Presentation; to be published on WGISS web site. White paper 0.2 n/a Update ongoing Associated Knowledge Best Practice CEOS/WGISS/DSIG/AKPBP Associated Knowledge Preservation Best Practices 04/2016 Being drafted Q1/2017

Cooperation with other WGs, VCs, Ad-hoc groups Initiatives ESA UNCLASSIFIED – For Official Use

WGISS initiatives in support to Carbon Action – SIT Tec Workshop 2016 ECVs/CDRs Discovery and Access through WGISS Systems Objective: facilitate discoverability and accessibility of ECV Products and space-born CDRs relevant for the CEOS Carbon Action via WGISS Interoperability Systems & Standards (FedEO/ CWIC/IDN, OpenSearch). WGISS Carbon Portal Objective: development as WGISS project of a CEOS WGISS Carbon Portal prototype similar to the Water portal (http://waterportal.ceos.org/) to allow displaying Carbon datasets and providing assistance to scientists and general users in the development of related services & tools. ESA UNCLASSIFIED – For Official Use

Cooperation with other WGs - Action Action WGISS-41-08: In order to discover how WGISS can assess the needs of other CEOS working groups, virtual constellations, and ad-hoc groups with regards to WGISS capabilities: Action WGISS-41-08a: WGISS members to identify, from within their own agencies, members of said groups. Action WGISS-41-08b: WGISS-Exec to coordinate one-on-one interactions with said contacts. ESA UNCLASSIFIED – For Official Use

Cooperation with other WGs - WGCLimate WGClimate need & announcement: Population of the ECV inventory with the characteristics of Climate Data Records and ECV products made available by CEOS agencies. Rationale: ensure that ECVs and CDRs holdings are widely visible to users. Dedicated website, http://climatemonitoring.info/, to support the population of the inventory; ECV Inventory Questionnaire Guide which provides guidance on how to populate the ECV inventory. This periodic activity (update every 2 years) will provide the basis for a gap analysis,​and subsequent generation of a coordinated action plan to address any detected gaps and/or opportunities for the delivery of additional and improved Climate Data Records. ESA UNCLASSIFIED – For Official Use

Recovery of historical datasets – initial draft list to be investigated Mission(s) Sensor(s) Mission Owner Data Set Location Ozone profiles from 30 to 110 km EURECA ORA ESA Belgian Institute for Space Aeronomy (TBC) Grille vertical profiles Spacelab 1; ATLAS-1 Grille (Infrared Spectrometer) Spacelab 1: NASA-ESA ATLAS-1: NASA Deuterium and Hydrogen atomic concentrations ATLAS-1 ALAE NASA Unknown (Sensor Owner: CNRS) Solar spectral irradiance Spacelab 1; EURECA; ATLAS missions SOLSPEC EURECA: ESA ATLAS missions: NASA Institut Pierre Simon Laplace (TBC) ESMR data sets from the polar regions Nimbus-5 ESMR NASA (NSIDC - National Snow & Ice Data Center) ESMR Imagery of Brightness Temperature on 70 mm Film GSFC - Goddard Earth Science Data and Information Services (GES DISC) SeaSat SMMR data SeaSat SMMR NASA/JPL JPL Physical Oceanography DAAC WGClimate requested to help in identifying additional datasets of interest for ECVs generation (and extension back in time) which were acquired in the past but are not accessible today.

Data Stewardship Related Conferences ESA UNCLASSIFIED – For Official Use

Conference on BIG DATA: Feedback ESA UNCLASSIFIED – For Official Use

BiDS’ 2016 Presentations and proceedings 62 papers selected for oral and 32 for poster presentations Demonstration/industry session (with 10 demos) organised with poster session, to give the possibility to present live demos on big data applications. Among the papers selected for oral presentation, 18 originated from 5 invited sessions devoted to horizontal themes, namely, Organisational Challenges for Big Data Space Missions, Advances in Big Data using OGC Standards, Image Information and Data Mining for Big Data, Big Data Impact on Industry, and Data Science Skills. Presentations and web site: http://congrexprojects.com/2016-events/16m05/introduction Proceedings: http://publications.jrc.ec.europa.eu/repository/handle/JRC100655 Next conference will be held in November 2017 at CNES (Toulouse) ESA UNCLASSIFIED – For Official Use

Living Planet Symposium (LPS) 2016: Feedback http://lps16.esa.int/ ESA UNCLASSIFIED – For Official Use

LPS16 Prague 9-13 May 2016 3100 participants Largest EO conference in the world 2700 abstracts submitted: 1800 posters and 900 orals presented in 11 parallel sessions 300+ students participated to the school Lab activities, organised by ESA and Charles University (Prague) Plenary session - welcome by Czech Prime Minister, Minister of Transport, ESA Director General and ESA Director of EO programmes Industry exhibition with more than 30 companies operating worldwide including Google and Amazon Public exhibition attended by hundreds of Prague citizens

PV2017: 7-9 November 2017 Venue: Rutherford Application Laboratory (RAL), Harwell, UK Dates: 7 – 9 November 2017 (2.5 days) Anticipated attendance: 70 - 200 people ESA UNCLASSIFIED – For Official Use

Thanks you for your attention !!! ESA UNCLASSIFIED – For Official Use