All rights reserved © Altec ExoMars 2018 Rover Operations Control Centre Science instruments data pipeline G. Martucci

All rights reserved © Altec Ref. Nr.  Scientific pipeline introduction  CODMAC Levels  Data Products definition  Science pipeline scope  Data Pipeline  Pipeline example Presentation overview Date 10/12/2014 Page 2

All rights reserved © Altec Ref. Nr. Scientific pipeline introduction Date 10/12/2014 Page 3 Scientific pipeline can be summarised as follows:  it is constituted by a set of processes aimed at producing exploitable data for both scientific and engineering teams  it is constituted by a set of processors, properly configured, that permit the processing of received Rover TM to obtain higer level CODMAC data (see next slides)  Processors are typically handled automatically at ROCC for critical data production  Generated products are then passed to next level processors, handled directly by scientific teams (this is due to the required interaction with data under analysis)  Thanks to this pipeline, products may also be produced in PDS4 (this is highly encouraged)

All rights reserved © Altec Ref. Nr.  Data products are incrementally produced at different levels. Incrementing the product level results in a refined version of the original datum  The Committee On Data Management, Archiving and Computing (CODMAC) has provided a description for the data processing levels on ground  This has been reflected on ROCC-to-PPL ICD and shall be taken as a reference for the scientific data processing within ROCC CODMAC Levels (1 / 2) Date 10/12/2014 Page 4

All rights reserved © Altec Ref. Nr. CODMAC Levels (2 / 2) Date 10/12/2014 Page 5 LevelTypeData Processing Level Description 1Raw DataTelemetry data with data embedded 2Edited Data Corrected for telemetry errors and split or decommutated into a data set for a given instrument. Sometimes called Experimental Data Record. Data are also tagged with time and location of acquisition. Correspond to NASA Level 0 data. 3Calibrated Data Edited data that are still in units produced by instrument, but that have been corrected so that values are expressed in or are proportional to some physical unit such as radiance. No resampling, so edited data can be reconstructed. NASA level 1A. 4Resampled Data Data that have been resampled in time or space domains in such a way that the original edited data cannot be reconstructed. Could be calibrated in addition to being resampled. NASA level 1B. 5Derived DataDerived results, as maps, reports, graphics, etc. NASA level 2 through 5. 6Ancillary Data Non science data needed to generate calibrated or resampled data sets. Consist of instruments gains, offsets, pointing information for scan platforms, etc. 7Correlative Data Other science data needed to interpret space-based data sets. May include ground-based data observations such as soil type or ocean buoy measurements of wind drift. 8User Description Description of way the data were required, any peculiarities associated with the data sets, and enough documentation to allow secondary user to extract information from the data.

All rights reserved © Altec Ref. Nr. Data Products definition Date 10/12/2014 Page 6 In the ROCC-to-PPL ICD two sets of data products are defined:  Primary Products:  Essential for tactical operation support  Available to ROCC operations staff as a batch processing pipeline  Completely generated in automatic mode  Secondary Products:  Used for strategic purposes  Used for scientific exploitation  Could need specific expertise, interactive intervention, specific SW configuration  Can be either produced within the automatic pipeline or at assessment level, with operator interaction  Primary Products are not associated to specific levels of CODMAC, but can belong to any level.

All rights reserved © Altec Ref. Nr. Science Pipeline scope Date 10/12/2014 Page 7  Many of the primary products generated throughout the science pipeline process are considered critical for tactical planning  This is due to the fact that their assesment is a decision point for planning activities  ROCS has «in-house» means for performing basic pre-processing of data, up to L2, this also includes decompression. This could be however integrated as a tool / application provided by science teams  Higher level data processing is expected to be performed through Science Teams provided processors. These will be integrated within ROCS

All rights reserved © Altec Ref. Nr. Data Pipeline (1 / 2) Date 10/12/2014 Page 8  The data pipeline is constituted by a frame process that manages the actual science and HK processors  Within the ‘ROCC-to-PPL’ ICDs a set of information have to be exchanged in order to be able to correctly «call» from this frame process the specific libraries / tools provided by the science teams, such as:  Running environment  Executable name  List and order of expected inputs  List and order of expected outputs  Configurations files ...  As such an executable script is called by the manager and started for processing  The results are then archived in the DAR and made available to the ROCC users, both locally and remotely located

All rights reserved © Altec Ref. Nr. Science teams assessment Data Pipeline (2 / 2) Date 10/12/2014 Page 9  ROCS will acquire TM through SCOS2000 in the form of TM frames  Then it will extract packets for science data (and process rover data to obtain engineering parameters)  Data are then passed to the processing pipeline to obtain proper CODMAC level products  Scientific team are then able to assess data and to provide input(s) for tactical planning TM Acquisition (and pre-processing) (SCOS2000) TM Frame data acquisition Extract science packets (with decompression algorithms) Extract rover packets and parameters (calibration of HK TM) TM processing Prepare TM packet for processing Prepare TM parameters for processing TM Processing packets parameters Data Products

All rights reserved © Altec Ref. Nr. Pipeline example (1 / 2) Date 10/12/2014 Page 10  The following scheme (courtesy of ISEM team) is a real example of a data pipeline process for Primary Products production: L1L2 Auto SW ISEM L3 Auto SW ISEM TM Data structure Preflight calibration Temperature calibration Dispersion curve  Data is provided as L1 and is then processed in the internal pipeline through an automatic processor provided by ISEM  L2 Data is further processed by a second processor, still integrated in the automatic pipeline

All rights reserved © Altec Ref. Nr. Pipeline example (2 / 2) Date 10/12/2014 Page 11  For what concerns Secondary Products the following scheme shows an example (courtesy if ISEM team):  Starting from L2 data a set of higher CODMAC levels secondary products is generated  This happens through interactive processors, that are not automatically run  In this example L5 CODMAC level is reached L2L3 SW ISEM L4L5 Calibrations Preflight On-board Additional data Position Surface 3D-model Atmosphere data Manual data analysis Matching with other instruments