1. Overview of the Data Archive Plan by Luc Joudrier & Dave Heather
ESWT #7 Data Archive
Overview of the Data Archive Plan by Luc Joudrier & Dave Heather
ExoMars ESWT# Dec 2014

2. Data Archive Plan 2018 Data Archive Plan ROCC ROCC – ESAC ICD ESAC PSA
The Data Archive Plan describes the process of archiving the data for the 2018 ExoMars Rover Mission. It is derived from the 2016 ExoMars Data Archive Plan.
It governs the interface between ROCC and Planetary Science Archive at ESAC.
The creation of the data to be archived is handled by ROCC.
Mirroring of the data to the Russian PSA will be handled by ESAC-PSA.
2018 Data Archive Plan
ROCC
ROCC – ESAC ICD
ESAC PSA
ROCC – PPL ICDs
Science Teams
IKI Mirror Archive
ExoMars Project <10 Dec 2014>
< ESWT#7 >

3. Which PDS Format ?
New missions, starting from ExoMars 2016 mission will be compliant to PDS4, version 4 of the NASA’s Planetary Data System.
ExoMars 2016 SOC developed at ESAC is gaining experience with this new format by developing PDS4 raw data pipeline, reusing the concepts of GDP. This experience is being shared with ALTEC and ultimately fed into the ROCC to PPL ICDs discussions.
Whether data are produced at ROCC or not (advanced products may be produced with support of the PI’s home base), they must be delivered to ROCC that is the entity who will deliver them to ESAC-PSA.
ExoMars Project <10 Dec 2014>
< ESWT#7 >

4. Science Data Deliverables
Science data deliverables are divided into primary products and supplementary products
Primary products contain the scientific data resulting from observations, including housekeeping and calibration data if applicable
Supplementary products include all non-observational data required directly or indirectly in the processing and analysis of the observational data.
Science data are delivered within a PDS bundle (PDS4 naming convention, former PDS3 dataset)
From: ExoMars 2016 Data Archiving |
N. Manaud | DHAWG#1 |
6 Feb | ExoMars 2016 | Pag. 4
ExoMars Project <10 Dec 2014>
< ESWT#7 >

5. Primary Products - Data Processing Levels
Telemetry data (ROCC)
Original binary stream from an instrument encoded in telemetry packets.
Raw data (produced at ROCC with PI’s pipelines)
Original data from an instrument. If compression, reformatting, packetization, or other translation has been applied to facilitate data transmission or storage, those processes will be reversed so that the archived data are in a PDS approved archive format.
Partially processed data (produced at ROCC with PI’s tools)
Data that have been processed beyond the raw stage but which have not yet reached calibrated status.
Calibrated data (produced at ROCC with PI’s pipelines)
Data converted to physical units where all of the calibration based on the instrument has been done.
Derived data (PI teams for scientific products, ROCC for Engineering products)
Results that have been generated from one or more calibrated products (for example: mosaics, maps, resampled data, fits to data models, temperature profiles etc…).
Calibration tables or tables of viewing geometry, used to interpret observational data should also be classified as ‘derived’ data if not easily matched to one of the other three categories. Engineering products (mobility, Drilling, SPDS) will be classified as Derived.
Preliminary to be discussed
From: ExoMars 2016 Data Archiving |
N. Manaud | DHAWG#1 |
6 Feb | ExoMars 2016 | Pag. 5
ExoMars Project <10 Dec 2014>
< ESWT#7 >

6. Supplementary Products - Types
Calibration
Calibration products include all data used at any stage of the calibration process. They can include, for example, observations of an on-board hardware calibrator or an external target used for calibration, or instrument response curves and bad pixel maps.
Geometry
Geometry products include pointing, orientation and positioning information. These pieces of information are linked to the localisation (position and attitude) of the rover.
Browse
Browse products are browse representations (thumbnails) or quick-look plots/reports of the observational products.
Document
Document products include all documentation or supporting information to assist in understanding and using the Primary Products. The specific contribution of each instruments to the ROCC to ESAC ICD is a required document product – This contribution is initially formulated in the ROCC to PPL ICDs (equivalent to Experimenter to Archive Interface Control Document (EAICD) naming convention of the 2016 Data Archive Plan).
From: ExoMars 2016 Data Archiving |
N. Manaud | DHAWG#1 |
6 Feb | ExoMars 2016 | Pag. 6
ExoMars Project <10 Dec 2014>
< ESWT#7 >

7. Supplementary Data Types
Miscellaneous
Miscellaneous products include any additional information and documentation products generated (e.g. tables to track planned and executed observations, spacecraft manoeuvres, pointing modes, ground-station communication passes, target identification, spacecraft and hardware anomalies), if not easily classified as one of the above-described Supplementary products. Anything that might affect, influence or offer an insight into the interpretation or understanding of the science data can be included in the archive.
Context
Context products include descriptions of the mission, spacecraft, instrumentation and targets in sufficient detail to give the user an insight into the nature of the data returned and the calibration processes used.
From: ExoMars 2016 Data Archiving |
N. Manaud | DHAWG#1 |
6 Feb | ExoMars 2016 | Pag. 7
ExoMars Project <10 Dec 2014>
< ESWT#7 >

8. Which process? 1/2 Short Term
Define which data are generated for science and support of operations (Draft ROCC to PPL ICDs)
Medium Term
Definition of data types, data structures and the data set organization for each instrument -> Refine ROCC to PPL ICDs so to prepare the ROCC to ESAC ICD
Agreement of the format of the generated products -> freezing of ROCC to ESAC ICD
Implementation of the data pipelines generating the agreed products (part of the archive to be delivered to PSA)
Readiness of ROCC: Verification of the generation of the data products.
Archive Working Group will be organized among ALTEC – PPL Teams – ESAC-PSA - ESA as mean to support and monitor the above process.
Schedule to be agreed.
ExoMars Project <10 Dec 2014>
< ESWT#7 >

9. Which process? 2/2 At the time of operations
Use of the data pipelines and generation of the products in the agreed format
Regular Peer Reviews of the data products generated over a period of 1 (TBC) month. The Peer Reviews are organized by the Project Scientist at ROCC.
Regular Release of the Peer- Reviewed data to ESAC PSA
Mirroring of data to IKI
Release to public of the data products after the 6 months of proprietary period.
ExoMars Project <10 Dec 2014>
< ESWT#7 >

10. Questions ?
ExoMars Project <12 Feb 2013>
< ESWT#5 >

11. ExoMars Project <12 Feb 2013>
< ESWT#5 >

12. COPY OF THE ESWT #6 PRESENTATION ABOUT 2016 ARCHIVE PLAN.
FOR INFORMATION ONLY
ExoMars Project <10 Dec 2014>
< ESWT#7 >

13. ExoMars 2016 Data Handling and Archiving Working Group Meeting #1
6th Feb 2014
Nicolas Manaud, Dave Heather, Olivier Witasse

14. Agenda Introduction (N. Manaud)
Working Group / points of contact
ExoMars 2016 Science Archive
Activities and Process
Quality Control Process
Data Rights and Release Policy
Introduction to the PSA/PDS4 (D. Heather)
Data Handling and Archiving Concept (N. Manaud)
Science Data Deliverables
Data Flow Overview
Scoping Data Processing Effort
Instrument Required Support (PI teams)
Combined theme-based data products (O. Witasse)
Upcoming work schedule (N. Manaud)
Discussions/AOB (All)
ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 14

15. DHA Working Group
This working group is the forum for discussion and agreement on activities related to science data processing and archiving, including associated services.
Composition
Archive Scientists (Chairman)
Nicolas Manaud (ESAC)
Oleg Batanov (IKI)
PI representatives
PSA Coordinator
David Heather (ESAC)
Project Scientists
Olivier Witasse (ESTEC)
Daniel Rodionov (IKI)
ESAC SOC S/W Engineers
Antonio Vilacorta (ESAC)
Jon Brumfitt (ESAC)
Frequency of meetings
combined with SWT/SOWG
regular DHAWG telecons.
ad-hoc meetings (as needed)
ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 15

16. Points of Contact Archive Scientist Nicolas Manaud (ESAC)
Instrument Teams
TGO
ACS
Alexey Shakun (IKI)
CaSSIS
Nick Thomas (UBE)
FREND
Alexey Malakhov (IKI)
NOMAD
Eddy Neefs (IR)
Jon Mason (UVIS)
EDM
AMELIA
Francesca Ferri (UNIPD-CISAS, Padova)
DECA
Detlef Koschny (?)
DREAMS
Pietro Schipani (INAF-OAC, Naples)
ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 16

17. ExoMars 2016 Science Archive
ExoMars TGO and EDM data will be archived into the ESA’s Planetary Science Archive (PSA) following existing process and standards from previous ESA’s planetary missions.
ExoMars data will be compliant to PDS4, version 4 of the NASA’s Planetary Data System.
ESA/PSA is the archiving authority
Agreements at management level between IKI and ESA
Russian Deep Science Archive will contain a duplicate of the ESA’s ExoMars Science Archive
ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 17

18. DHA Activities and Process Overview
Pre-Launch Preparation
Planning: Science Data Generation, Validation and Archiving Plan (“Archive Plan”)
Plan for producing, validating, archiving and disseminating all EM16 science data
Design: Experiment to Archive Interface Control Document (EAICD)
Definition of data types, data structures and the data set organisation for each instrument
Development
Development and/or monitoring of data processing pipeline development
Routine Operations
Data Delivery and Validation
Quality Control
Data Access and Exploitation
Post-Operations and Legacy
ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 18

19. Quality Control Process Overview
Formal reviews of the science data ingested in the archive will be organised by the ESAC SOC, in coordination with the IKI SOC and the Project Scientist(s). The review is a three-step process consisting of:
Review of the Science Data Generation, Validation and Archiving Plan and the individual Experiment-to-Archive Interface Control Documents (EAICDs), including (if possible) sample data and processing pipelines documentation (Archiving Documentation Review).
Review of the first data delivery (Initial Peer Review).
Review of all data together after the final delivery (Final Peer Review)
After successful Initial Peer Review, successive data deliveries will be syntactically validated prior to ingestion into the PSA to guarantee PDS compliancy.
Scientific quality of observational data will be assessed by the Instrument Team and reported in data products meta-data
ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 19

20. Data Rights and Release Policy
Instrument teams are granted a proprietary period of 6 months for the generation, validation and analysis of their data. Once data proprietary period is over, science data products will be publicly released through the PSA, and IKI Deep Science Archive.
A data collection period of 1 month is defined to facilitate the communication with the Instrument Teams for notification/confirmation before science data products are made publicly available. The data collection period defines the content of each data delivery to the PSA.
ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 20

21. Science Data Deliverables
Science data deliverables are divided into primary products and supplementary products
Primary products contain the scientific data resulting from observations, including housekeeping and calibration data if applicable
Supplementary products include all non-observational data required directly or indirectly in the processing and analysis of the observational data.
Science data are delivered within a PDS bundle (PDS4 naming convention, former PDS3 dataset)
ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 21

22. Primary Products - Data Processing Levels
Telemetry data (SOC)
Original binary stream from an instrument encoded in telemetry packets.
Raw data (SOCs and/or PI teams)
Original data from an instrument. If compression, reformatting, packetization, or other translation has been applied to facilitate data transmission or storage, those processes will be reversed so that the archived data are in a PDS approved archive format.
Partially processed data (PI teams)
Data that have been processed beyond the raw stage but which have not yet reached calibrated status.
Calibrated data (PI teams)
Data converted to physical units where all of the calibration based on the instrument has been done.
Derived data (PI teams for scientific products, SOC for SPICE)
Results that have been generated from one or more calibrated products (for example: mosaics, maps, resampled data, fits to data models, temperature profiles, gravity or magnetic fields, or ring particle size distributions).
Calibration tables or tables of viewing geometry, used to interpret observational data should also be classified as ‘derived’ data if not easily matched to one of the other three categories. Navigation products (including SPICE kernels) will be classified as Derived.
ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 22

23. Supplementary Products - Types
Calibration
Calibration products include all data used at any stage of the calibration process. They can include, for example, observations of an on-board hardware calibrator or an external target used for calibration, or instrument response curves and bad pixel maps.
Geometry
Geometry products include pointing, orientation and positioning information. Geometry information can be provided in two formats (1) as calculated values (e.g. pixel-by-pixel geometry parameters computed for each image/product in the data set), and (2) as SPICE kernels, which can be used with the NAIF SPICE software toolkit to calculate the appropriate values.
Browse
Browse products are browse representations (thumbnails) or quick-look plots/reports of the observational products.
Document
Document products include all documentation or supporting information to assist in understanding and using the Primary Products. The Experimenter to Archive Interface Control Document (EAICD) is a required document product.
ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 23

24. Supplementary Data Types
Miscellaneous
Miscellaneous products include any additional information and documentation products generated (e.g. tables to track planned and executed observations, spacecraft manoeuvres, pointing modes, ground-station communication passes, target identification, spacecraft and hardware anomalies), if not easily classified as one of the above-described Supplementary products. Anything that might affect, influence or offer an insight into the interpretation or understanding of the science data can be included in the archive.
Context
Context products include descriptions of the mission, spacecraft, instrumentation and targets in sufficient detail to give the user an insight into the nature of the data returned and the calibration processes used.
ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 24

25. Data Flow Overview
ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 25

26. Scoping Data Processing Effort
Your Raw Data Generation Primary Pipelines options:
Instrument SOC
ESAC SOC
In this case we also offer to generate PDS bundles.
Russian SOC
Current understanding
ACS using Russian SOC Pipeline (TBC)
CaSSIS using ESAC SOC Pipeline (confirmed)
FREND using ESAC or Russian SOC Pipeline (TBD)
NOMAD using Instrument SOC (confirmed)
IR channels: existing TM processing pipeline (PDS4 adaptation needed)
Other channels will implement the same approach
ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 26