1 The ESA SCIENTIFIC TESTBED in CASPAR S. ALBANI (ESA) CASPAR Workshop, Rome, 15-16 september 2009.

Slides:



Advertisements
Similar presentations
1 NameMatrix Number Francis YeeHT036029M George Goh Alex LimHT052467E Hoe Swee SimHT052560I Vijay.
Advertisements

1 of 18 Information Dissemination New Digital Opportunities IMARK Investing in Information for Development Information Dissemination New Digital Opportunities.
IBM Haifa Research Lab © 2008 IBM Corporation Contacts: Simona Cohen, Michael Factor, Dalit Naor
ASYCUDA Overview … a summary of the objectives of ASYCUDA implementation projects and features of the software for the Customs computer system.
ESA Data Integration Application Open Grid Services for Earth Observation Luigi Fusco, Pedro Gonçalves.
E-Content Service Group Virtual Meeting Digital Preservation: How to Get Started.
ActionDescription 1Decisions about planning and managing the coast are governed by general legal instruments. 2Sectoral stakeholders meet on an ad hoc.
DRIVER Building a worldwide scientific data repository infrastructure in support of scholarly communication 1 JISC/CNI Conference, Belfast, July.
DRIVER Long Term Preservation for Enhanced Publications in the DRIVER Infrastructure 1 WePreserve Workshop, October 2008 Dale Peters, Scientific Technical.
Curating Research: problems and policy Dale Peters Scientific Technical Manager DRIVER II.
The NATURE–SDIplus project Best Practice Network for SDI in Nature Conservation Co-funded by the Community Programme eContentplus ECP-2007-GEO Co-funded.
Introduction to Planets Hans Hofman Nationaal Archief Netherlands Prague, 17 October 2008.
Module N° 7 – Introduction to SMS
1 NECOBELAC Project WORK PACKAGE 3 Cross-national advocacy infrastructure.
Issues and approaches to preservation metadata Michael Day UKOLN: UK Office for Library and Information Networking University of Bath
A centre of expertise in data curation and preservation DigCCur2007 Symposium, Chapel Hill, N.C., April 18-20, 2007 Co-operation for digital preservation.
Configuration management
Software change management
EMS Checklist (ISO model)
Environmental Management Systems Refresher
Project Overview APA Conference 2012 ESA/ESRIN (Frascati), 6-7 November 2012 U.Di Giammatteo (ACS)
ESA UNCLASSIFIED – For Official Use Data Stewardship Interest Group Session LTDP Activities & DSIG Introduction WGISS-37 Meeting Cocoa Beach (Florida-US)
DOSTAG Meeting #51, 3 rd May 2007 Access to ESA Earth Observation data (past and current missions)
CNRIS CNRIS 2.0 Challenges for a new generation of Research Information Systems.
SCIDIP-ES Components Oct ,Brussels. Basic Preservation Strategies Often stated as: “Emulate or Migrate” OAIS concepts change these to: Add Representation.
ISO & OAI-PMH By Neal Harmeyer, Amy Hatfield, and Brandon Beatty PURDUE UNIVERSITY RESEARCH REPOSITORY.
Project Overview APA Conference 2012 ESA/ESRIN (Frascati), 6-7 November 2012 M. Albani (European Space Agency), Project Coordinator.
6-7 May 2002 CEOS GRID Workshop 1 “CEOS GRID Proposals” roundtable: Introduction ESRIN, 6-7 May 2002 CEOS Workshop on GRID.
Lecture 5 Themes in this session Building and managing the data warehouse Data extraction and transformation Technical issues.
Future Access to the Scientific and Cultural Heritage – A shared Responsibility Birte Christensen-Dalsgaard State and University Library.
Preservation and Long-term access through Networked Services Adam Farquhar, The British Library iPres2006 Cornell University, October 2006.
Co-funded by the European Union under FP7-ICT Alliance Permanent Access to the Records of Science in Europe Network Co-ordinated by aparsen.eu #APARSEN.
Data Stewardship Interest Group WGISS-39 Meeting
APA Conference, London, 8-9 November 2011 Mirko Albani (Earth Observation Ground Segment Department, European Space Agency) ESA Long Term Data Preservation.
ESA UNCLASSIFIED – For Official Use Data Stewardship Interest Group WGISS-39 Meeting Data Purge Alert Procedure Tsukuba, Japan – May, 2015 Mirko.
Mid-Michigan Digital Practitioners, March 14, 2014 The National Digital Stewardship Alliance Agenda Mid-Michigan Digital Practitioners Meeting Abigail.
Towards a European network for digital preservation Ideas for a proposal Mariella Guercio, University of Urbino.
® Kick off meeting. February 17th, 2011 QUAlity aware VIsualisation for the Global Earth Observation system of systems GEOVIQUA workshop February, the.
ESA UNCLASSIFIED – For Official Use ESA AVHRR Data Holdings and Ongoing Curation Activities Mirko Albani, Sergio Folco.
ESRIN Earth Observation Program Ground Segment Department 26/09/2015 CEOS-WGISS-40 - Olivier BaroisSlide 1 Open Source Practices.
Archival Workshop on Ingest, Identification, and Certification Standards Certification (Best Practices) Checklist Does the archive have a written plan.
WGISS /09/2015 DATA PRESERVATION – CNES APPROACH B. Chausserie-Laprée.
NIST Data Science SymposiumMarch 4, 2014 NIST Data Science SymposiumMarch 4, Climate Archives in NOAA: Challenges and Opportunities March 4, 2014.
OAIS Rathachai Chawuthai Information Management CSIM / AIT Issued document 1.0.
ATT Contribution to GEO Archive Task Team WGISS – 22 Sep 11 – 15, 2006 Annapolis, USA.
ESA UNCLASSIFIED – For Official Use Data Stewardship Interest Group WGISS-40 Meeting Preservation of SW & Documents at CEOS Agencies Approaches and Lessons.
Global Forest Observations Initiative Simon Eggleston GFOI SDCG 5, Rome, Italy, 25 Feb 2014,
EO Dataset Preservation Workflow Data Stewardship Interest Group WGISS-37 Meeting Cocoa Beach (Florida-US) - April 14-18, 2014.
EC Review – 01/03/2002 – WP9 – Earth Observation Applications – n° 1 WP9 Earth Observation Applications 1st Annual Review Report to the EU ESA, KNMI, IPSL,
2. WP9 – Earth Observation Applications ESA DataGrid Review Frascati, 10 June Welcome and introduction (15m) 2.WP9 – Earth Observation Applications.
Managing Access at the University of Oregon : a Case Study of Scholars’ Bank by Carol Hixson Head, Metadata and Digital Library Services
Lifecycle Metadata for Digital Objects November 15, 2004 Preservation Metadata.
Data Preservation at Rutherford Lab David Corney 9 th July 2010 KEK.
Cloud-based e-science drivers for ESAs Sentinel Collaborative Ground Segment Kostas Koumandaros Greek Research & Technology Network Open Science retreat.
Grant Writing for Digital Projects September 2012 IODE Project Office IODE Project Office Oostende, Belgium Oostende, Belgium Sustainability and.
Fulvio Marelli - ESA and future An example of data lifecycle: sensed data need to be acquired…
BNSC Agency Report David Giaretta Colorado Springs 16 Jan 2007.
ESA UNCLASSIFIED – For Official Use Data Stewardship Interest Group EO Data Associated Knowledge Preservation WGISS-41 Meeting - Canberra, (AUS) 14–18.
Exploitation of ISS Scientific data EGI-Aparsen Workshop March Science Park– Amsterdam – The Netherlands Cooperative ISS Research data Conservation.
A Shared Commitment to Digital Preservation and Access.
CEOS Working Group on Information System and Services (WGISS) Data Access Infrastructure and Interoperability Standards Andrew Mitchell - NASA Goddard.
COBIT. The Control Objectives for Information and related Technology (COBIT) A set of best practices (framework) for information technology (IT) management.
An introduction to MEDIN Data Guidelines September 2016
The ESA CASPAR Scientific Testbed and the combined approach with GENESI-DR S. ALBANI (ACS c/o ESA-ESRIN) PV2009, 1-3/12/2009, Madrid.
Data Stewardship Interest Group WGISS-42 Meeting
Joseph JaJa, Mike Smorul, and Sangchul Song
Exploitation of ISS Scientific data - sustainability
Pier Giorgio Marchetti, Philippe Mougnaud European Space Agency
Open Archival Information System
The ESA Earth Observation Long Term Data Preservation (LTDP) Programme
Presentation transcript:

1 The ESA SCIENTIFIC TESTBED in CASPAR S. ALBANI (ESA) CASPAR Workshop, Rome, september 2009

2 SUMMARY The importance of Earth Observation data LTDP: ESA benefits, risks and policies The ESA scientific testbed in CASPAR Conclusions

3 ESA EO Data Handling ESA, through worldwide receiving ground stations, acquires data from Earth Observation (EO) satellites and archives/processes them at Processing and Archiving Facilities. ESA-ESRIN is the largest European EO data provider and operates as the reference European centre for EO Payload Data Exploitation. At present, several thousands ESA worldwide users have online access to EO missions related metadata (10 million references), data (about 4 PB) and derived information.

4 The EO data avalanche… ESA ESA EO data archives include data from ESA missions (ERS and ENVISAT) and Third Party missions (Landsat, SPOT, ALOS, TIROS, AQUA&TERRA, etc.). The plans of new missions indicates 5-10 times more data to be archived in next years Similar trend for other national space agencies

5 The importance of EO data EO archives ensure a global coverage of the Earth: multi-sensor data (from optical to active radar sensors); long time series (time-span from a few years to decades); variable geographical coverage (local, regional, global); variable geometrical resolution (from few to few hundreds meters); variable temporal resolution (from few days up to months).

6 The requirements for accessing ESA historical archives are strongly increased in the last ten years and the trend is likely to increase in the future mainly for long term science and environmental monitoring. Therefore, the prospect of losing the digital records of science (and with the specific unique data, information and publications managed by ESA) is very alarming. We have to preserve data against changes in: –hardware, –software, –environment, –knowledge base of the scientific community… LTDP: Benefits, risks and policies (1)

7 Issues for the near future concern the identification of: –type and amount of data to be preserved; –location of archives and their replication for security reasons; –technical choices (e.g. formats, media); –availability of adequate funds. Decisions should be taken in coordination with other data owners and with the support/advice of the user community. LTDP: Benefits, risks and policies (2)

8 Preservation of all European EO data for an unlimited time-span ensuring accessibility and usability Cooperative and harmonized collective approach among the EO data owners (European LTDP Framework) Application of European LTDP Common Guidelines Member States ESA LTDP Working Group formed (Jan 2008) within the GSCB to define LTDP Common Guidelines (with the European EO data stakeholders) and to promote them in CEOS/GEO. Earth Observation Long Term Data Preservation Strategy LTDP Workshop ESA-ESRIN May 2008 ESA is going to apply the European LTDP Common Guidelines to its own missions LTDP is now included in the ESA general budget under basic activities LTDP: Benefits, risks and policies (3)

9 Meanwhile ESA is active in developing appropriate techniques and strategies by promoting and participating to projects related to long term data preservation ESA participates to LTDP related projects to: –evaluate new technical solutions and procedures to adopt when applicable to maintain leadership in using emerging services in EO; –share knowledge with other entities, also outside of the scientific domain; –extend the results/outputs of these cooperative projects in other EO and ESA communities (e.g. Record of Science office). LTDP: Benefits, risks and policies (4)

10 ESA role in CASPAR ESA participation to CASPAR is mainly driven by the interest in: –consolidating and extending the validity of the OAIS reference model, already adopted in several internal initiatives (e.g. SAFE); –developing preservation techniques/tools covering not only the data but also the knowledge associated with them. In CASPAR ESA is playing the role of both user and infrastructure provider for the scientific data testbed.

11 ESA TESTBED ACTIVITIES The ESA testbed has covered: the setup of the framework in ESA-ESRIN; the definition and collection of a significant sample of a whole processing chain dataset (viewers, converters, processors and data of different level, revision and format); the conversion of the data from the native format to a OAIS compliant format; the generation of appropriate Representation Information, Descriptive Information, Knowledge Modules and Scientific Community profiles; the analysis of ontologies to describe and preserve scientific workflows (e.g. the applicability of CIDOC CRM on scientific data); the ingestion of data in a 100% CASPAR-based system; the coping with some long term data preservation problems by using only CASPAR components, methodology and tools.

12 The ESA selected dataset as a suitable demonstration case in the framework of the scientific testbed of the CASPAR project consists of data from GOME (Global Ozone Monitoring Experiment), a sensor on board the ESA ERS-2 (European Remote Sensing) satellite. The GOME dataset: –has a big total amount of information distributed with a high level of complexity; –is unique because it provides more than 11 years global worldwide coverage; –is very important for the scientific community and the Principal Investigators (e.g. KNMI and DLR) that on a routine basis receive GOME data for their research projects (e.g. concerning ozone depletion or climate change); –is just a test-case because similar issues involve many other Earth Observation instrument datasets. Testbed dataset (1)

13 Level 0: raw data Data viewers Level 1: radiances/reflectances Level 1C: fully calibrated L1 data Level 2: geophysical data as trace gas amounts Level 3: a mosaic composed by several level 2 data with interpolation of data values to fill the satellite gaps Level processors Auxiliary data Documents and methods Examples of GOME science applications Format converters Testbed dataset (2)

14 Preservation of the ability to process data from one level to another –preservation of GOME data and of all components that enables the operational processing for generating products at higher levels. DEMO: preservation of the ability to produce GOME Level 1C data starting from Level 1 data –now the ESA testbed is able to demonstrate the preservation of this GOME processing chain at least against changes of operating system or compilers/libraries/drivers affecting the ability to run the GOME Data Processor. Testbed preservation scenario (1)

15 L1 L1C source code GOME L1 products L1 L1C processor PSD.pdf license.doc disclaimer.pdf readme_1st.doc readme.doc release_l01.doc user_manual.pdf howtouse_l01.doc The Ozone The ERS-2 satellite The GOME sensor ERS-Products.pdf ProductSpecification.pdf The C Bible The OS Bible We have to preserve the GOME L1 products, processors, reference manuals, etc. L1->L1C processing dataset

16 OAIS compliant dataset SAFE AIP Content Information Preservation Description Information Descriptive Information Data ObjectRepresentation Information FixityContext Reference Empty. Packaging Information No packaging restrictions. Metadata is extracted by the product and contained in the SAFE manifest. The filename itself. Provenance The principal investigator who recorded the data and the information concerning its storage, handling and migration. A Cyclical Redundancy Check (CRC) code for a file. The RepInfo provided are contained in the manifest file and in the schemas.

17 After the ingestion in the CASPAR system of a complete and OAIS-compliant GOME L1 processing dataset, something (e.g. OS or gLib version) changes and a new L1->L1C processor has to be developed/ingested to preserve the ability to process data from L1 to L1C. We can cope with these changes related to the processing by managing a correct information flow through the system, the system administrators and the users, using a framework developed using only the CASPAR components. Moreover we can return to an user asking for L1C data not only the related L1 data plus the processor needed to generate them but also all the information needed to perform this process depending on the user needs and knowledge. Testbed: preservation scenario (2)

18 CONCLUSIONS The preservation of EO data is of vital importance for the scientific and operational user communities. ESA is pursuing the objective to ensure the perpetual preservation of these data in coordination with institutions of its member states. ESA data preservation initiatives will benefit of the results of initiatives carried out in collaboration with European data owners/providers, entities and institutions, with the objective to guarantee long term data preservation (and other similar EU sponsored projects) by adopting, when applicable, technical solutions and procedures developed in the framework of these cooperative projects. The scientific testbed implemented in ESA-ESRIN provides a good proof of the effectiveness of the CASPAR preservation framework in the Earth Observation domain focusing on methodologies and techniques to guarantee the preservation of the knowledge associated to data.

19 THANKS!