1

SCIDIP-ES Objectives To develop and deploy generic Services and Toolkits for data preservation. Validated in the ES domain ES perspective: providing use cases, environment for S&T deployment, identify test datasets, SW testing, feedback and validation activities. To analyse the state of the art/users needs and propose harmonisation activities for digital data preservation in the Earth Science domain: Paving the way for the set-up of a shared and common approach for Long Term Preservation of Earth Science Data.

SCIDIP-ES partners Earth observation repositories Earth Science repositories Software house specialized in ES preservation systems (acs, cap, gim) R&D in preservation Systems (APA, ENG, Forth, ICT, FTK) University skilled in standards and preservation systems (UTV and JUB)

North pole Ice pack reduction measured by EO satellites between 1978 and 2010 This time series is used to show the progressive reduction of the sea ice in the Arctic area. Sea Ice melting is one of the Essential Climate Variables. Ice melting at the north pole would significantly impact on our global environment (which we need to model and forecast). Moreover it would have an incredible impact on sea routes traffic (90% of the goods are transported on the sea) and therefore on the global economy. It is essential that our scientist have the full access to all information related to this (and other) phenomena. Unfortunately satellites (and/or experiments) are not forever. The video you see was realized unifying over 4 missions (i.e. 4 different satellites and sensor). This poses problems of file conversion, metadata harmonization. Which must be planned and have a cost.

Terabytes

THREATS Changes in: Format Software Hardware IPR Funding Designated Communities THREATS

GOCE turns out to be the first seismometer in orbit Earth Science Change in data user community User community is changing Now scientists working with Earthquakes want this information to study earthquakes propagation in the atmosphere GOCE turns out to be the first seismometer in orbit

LTDP WG LTDP WG under ESA leadership formed in 2008 within the Ground Segment Coordination Body (GSCB) to better Coordinate European (and Canadian) efforts and activities in the domain of digital data preservation Implemented the basic rules of a European LTDP Framework in Earth Observation: “LTDP Common Guidelines and Preserved Data Set Content” Reflecting the consensus of the European EO Data providers Reviewed at international level (GEO, CEOS, NASA) Aligned with EO data quality standards (QA4EO)

State of the art (EO example) Mission managers actually ask for document/software collection for data users. Collection could be not properly organized or different from other mission Not manageable in the long term Not directly connected to catalogued data Users will “Google” for information: not sure they will find the latest and/or original version

Practical problems Mission concept Mission design Mission impl Mission operation Post/Preservation Need to convince mission responsibles to include preservation from the start Need not to change the normal routines/technologies in the archives No technicalities (e.g. no OAIS training to understand the solution) Need to show immediate benefits with almost no effort

THE SCIDIP-ES solution WP 15 T. 33.3 WP 21 WP 12 MERIS Full Resolution- Level 2 MERIS IPF BEAM User Manual Tech Details Other docs Level 1 Level 0

Building a Preserved Dataset Content Defines what, why, how, to be preserved for an EO mission Based on the EO LTDP defined by LTDP WG Analysis of all dependencies (structural, semantics) Extended to Earth Science (generic) MERIS Full Resolution- Level 2 MERIS IPF BEAM User Manual Tech Details Other docs Level 1 Level 0

Phases Mission Preparation Mission Operation Post Mission Framework for ES archivist - SCIDIP-ES developers communication

KNOWLEDGE BROWSER

DLR NDVI - Obtained from MERIS L3 MERIS Full Resolution- Level 3 MERIS L3 IPF Tech Details MERIS Full Resolution- Level 2 MERIS IPF BEAM User Manual Tech Details Other docs Level 1 Level 0

Improvements (from feedbacks) Improve the perception for SCIDIP-ES platform and S&T usability and understandability Taking advantage of achievements in D33.3 (preservation ontology for ES data preservation used in the S&T) De-coupling technology and knowledge of ISO standards details from effective content ingestion and data preservation actions Final users perception would change dramatically: no need to understand OAIS, no need to fill in the registry, all data/knowledge immediately organized (semantically, structurally) “Easy preservation” would facilitate adoption of preservation actions since the beginning of a mission or during its phases

Infrastructure take-up As with any mature infrastructure – internet, mobile phone etc there is a need for Small number of experts who set up the infrastructure software, OAIS, terminology Large number of end users for whom the experience is as painless as possible e.g. Users (archivists etc) do not want to change their habits You have to help them without complicating their life  IN PROGRESS

Functionalities in a nutshell Easy ingestion of ES-PDSC: users are guided through a simple wizard to recovery and upload data Once updated it is easy to browse data-related knowledge Easy to connect SCIDIP-ES registries to existing catalogues SCIDIP-ES registries can be connected one to each other to share preservation effort between agencies All services integrated in a single infrastructure via a common entry point Critical mass of users

Preservation functionalities in a nutshell Allow data managers to know something has changed Facilitate their understanding of the implications of that change Allow them to decide on the best course of action for preservation Understand (and create) the metadata which is needed to fill the gaps (created by someone else or creating a new one) If transformed: how to maintain data authenticity Alternatively: hand it over to another repository (real case here) Make sure data is now usable and close the process

Deployments APA APA will provide, after the project the: “guarantor/exchange” nodes for the Registry etc Build system Interactive platform for users to try things out and provide feedback This will be cleared each day.

Deployment at ESA VM1 VM2 ESA DASHBOARD RepInfo Registry GAP Identif Service Finding Aids Certification ESA DASHBOARD Happi Storage Packaging Orchestration

The Seasat dataset Seasat mission has significant historical value (being the first SAR) but low scientific impact (having completed only few passes). The data has been preserved so far on a best effort basis. Perfect example for archive building These data were on a tape and are being now reprocessed to generate products. All information have been collected and is now used as perfect example here.

The Nimbus-7 CZCS dataset The Nimbus-7 spacecraft was turned off in 1994 after 16 years of service. During its long period of operation, the Nimbus-7 mission was regarded as the single most significant source of experimental data from Earth's orbit relating to atmospheric and oceanic processes.

What we were able to understand Being the ESA registry based on the “Enterprise Version” and the APA based on the “Light Web Version” we (involuntarily) demonstrated we agree to the main principles of service orientation:   standardized service contract: from the external view, the registry are the same service loose coupling: no intra-components dependency service autonomy: independent functional boundary and runtime environment (the switch-over was immediate and painless)

Future architecture at ESA (under consolidation) ESA EO Data catalogue and dissemination system Operational environment ESA Earth Observation Knowledge Management System ESA EO web portal ENVRI Catalogue SCIDIP-ES ESA Dashboard Research environment

ON top of these services ESA is building a Virtual Research Environment e-infrastructure including different Earth Science Virtual Research Communities scenarios Will make use of a subset of SCIDIP-ES services

DLR deployments http://orchestration.eoc.dlr.de/orchestration-service-gui http://scidip.eoc.dlr.de/ScidipRegistry http://scidip-browser.eoc.dlr.de/FindingAidGui http://scidip.eoc.dlr.de/ScidipRepository With this initial deployment, it is possible to achieve semantic connection between the DLR and ESA MERIS datasets

DLR – Screenshots

ISPRA Deployment structure VMWare

BGS/BADC: example client applications RIN’s created for key data types and managed via RIT (e.g. NetCDF, geophysical data, and 3D models) On-line catalogue functionality enhanced by the provision of preservation information (via knowledge browser): Earth Science Academic archive (ESAA -www.bgs.ac.uk/scidip-es ) Strategic Environmental Assessment dataset catalogue (SEA - http://webservices.bgs.ac.uk/scidip-es/sea/search BADC client http://badc.nerc.ac.uk/view/badc.nerc.ac.uk__ATOM__dataent_MST Next steps: Deploy preservation assistant to assist day to day users load their data Orchestration service to help us manage the LTDP of some of our more s/w depended data formats which represent a significant investment in data capture/ingestion (e.g. geophysical data)

INGV Scenarios & Dataset INGV has identified two scenarios, and relative dataset, for using and testing SCIDIP-ES solutions in preservation tasks: MOIST: Multidisciplinary Oceanic Information SysTem (geomarine observation) - http://moist.rm.ingv.it/ DidYouFillIt : Earthquakes Maps - www.haisentitoilterremoto.it

Preservation Virtual Machine

To close the loop opened with the survey for D15.1 Will show SCIDIP-ES results For Data Managers, Data Providers, Data Users Opened from Mid September 2014 Will contain evaluation methods and commitment to adopt our ideas Will reach almost 10K persons in the ES domain Results will form consistent part of final review discussion

Data managers: Data providers: End users: The preservation infrastructure; E-PDSC; Federation model including access; LTDP MM Data providers: Preservation themes; E-PDSC; Federation Model including governance and policies; LTDP maturity model (20 questions) End users: Service federation - end user perspective; E-PDSC; Preservation infrastructure; https://www.surveymonkey.com/s/M6DQYSH