UTG4 – Archiving and processing planetological data Simone Silvestro INAF Osservatorio Astronomico di Capodimonte, Napoli

Introduction: INAF instrument & missions Data processing: USGS ISIS & INAF pipelines Archciving: NASA PDS/ESA PSA & others

15 Instruments (11 missions) Ongoing Terminated Future Introduction 15 Instruments (11 missions) Ongoing Terminated Future 1) NOMAD 2) CaSSIS 3) MARSIS 4) PFS 5) EFD 6) JIRAM 1) VIR 2) VIRTIS (1) 3) VIRTIS (2) 1) MAJIS 2) JANUS 3) Ma-MISS 4) Micro-MED 5) SYMBIO-SYS ExoMars 2016: NOMAD, CaSSIS Mars Express: PFS, MARSIS ExoMars 2020: Ma-MISS, Micro-MED Cassini: VIMS JUICE: MAJIS, JANUS Venus Express: VIRTIS (1) Rosetta: VIRTIS (2) Juno: JIRAM BEPI-Colombo: S-SYS DAWN: VIR CSES: EFD

Introduction Different bodies 1) NOMAD 2) CaSSIS 3) PFS 4) MARSIS Jupiter & Satellites Saturn & Satellites Venus Asteroids & Comets Earth Mercury 1) NOMAD 2) CaSSIS 3) PFS 4) MARSIS 5) Ma-MISS 6) MicroMED 1) JIRAM 2) MAJIS 3) JANUS 1) VIMS 1) VIRTIS (1) 1) VIR 2) VIRTIS (2) 1) EFD 1) SYMBIOSYS

Instrument types Camera/Spectrometer Radar GPR Others Introduction Instrument types Camera/Spectrometer Radar GPR Others 1) NOMAD 2) CaSSIS (stereo) 3) PFS 4) Ma-MISS 5) JIRAM 6) MAJIS 7) JANUS (stereo) 8) VIRTIS (1) 9) VIRTIS (2) 10) VIR 11) SYMBIO-SYS (stereo) 12) VIMS 1) MARSIS 1) Micro-MED 2) EFD

Introduction: INAF instrument & missions Data processing: USGS ISIS & INAF pipelines

*USGS ISIS/Propietary pipeline ONLY propietary pipeline/ISIS TBD Processing Processing pipelines *USGS ISIS/Propietary pipeline ONLY propietary pipeline/ISIS TBD 1) VIRTIS (1) – Venus Exp. 2) VIRTIS (2) – Rosetta 3) VIR – DAWN 4) CaSSIS – TGO ExoMars 2016 1) MAJIS - JUICE 2) JANUS - JUICE 3) JIRAM - Juno 4) NOMAD – TGO ExoMars 2016 5) Ma-Miss – ExoMars 2020 6) Micro-MED 7) PFS – Mars Express 8) MARSIS – Mars Express 9) SIMBIO-SYS – Bepi Colombo 10) EFD - CSES *Integrated Software for Imagers and Spectrometers

Processing Propietary pipelines INAF/IAPS: NOMAD (internal Python scipts) PFS (internal tools) MARSIS (in-house proc. pipeline – Matlab scripts) VIR (internal propietary calibration pipeline) VIRTIS (1) (in-house proc. pipeline, Python reading software freely available) VIRTIS (2) (IDL processing pipeline) INAF/OAPD: CASSIS (high-level products – digital terrain models – GIS interface for request/targeting) SIMBIO-SYS (high-level products – digital terrain models)

Processing USGS ISIS

Processing USGS ISIS Example of a processing pipeline in USGS ISIS Three processing steps valid for most cameras/spectrometers: Level 0 – Data ingestion Acquire and convert image data files (PDS format) to the ISIS3 image format (cubes) Add information to the ISIS3 image in order to compute geometric properties such as latitude/longitude range and illumination angles of the image Level 1 (A and B) – Radiometric Calibration and Noise Removal Convert raw pixel numbers to reflectance (irradiance/solar flux or I/F) Remove noise Level 2, 3 – Projection Geometrically rectify to a map projection

NASA processing levels 0 – reconstructed, unprocessed instrument and payload data at full resolution, with artifacts removed 1A – reconstructed, unprocessed instrument data at full resolution, time-referenced, and annotated with ancillary information, including radiometric and geometric calibration coefficients and georeferencing parameters (e.g., platform ephemeris) computed and appended but not applied to Level 0 data 1B – level 1A data that have been processed to sensor units 2 – derived geophysical variables at the same resolution and location as Level 1 source data 3 – variables mapped on uniform space-time grid scales, usually with some completeness and consistency 4 – model output or results from analyses of lower-level data (e.g., variables derived from multiple measurements)

Introduction: INAF instrument & missions Data processing: USGS ISIS & INAF pipelines Archciving: NASA PDS/ESA PSA & others

Archiving Global repositories NASA PDS ESA PSA Local repositories ASI SSDC INAF Others CNSA IKI ROSCOSMOS All the data produced by the instruments listed in this ppt should be NASA PDS/ESA PSA compliant

Archiving NASA PDS – 8 main nodes 6 science discipline nodes 2 support nodes

Archiving PDS Standards PDS Data product = Data object + label file describing the object Data files: ASCII or binary format Primary data object: set of results from a scientific observation (TABLE, SPREADSHEET, IMAGE, SERIES, SPECTRUM, QUBE) Secondary data object: any data used for processing or interpreting the primary data object(s) (HISTOGRAM, PALETTE, HEADER) Label files: written in Object Description Language (ODL) in the format «keyword = value», defines both the physical and logical structure of the constituent data object(s)

PDS label example Primary data object Secondary data object /* File Format and Length */ RECORD_TYPE = FIXED_LENGTH RECORD_BYTES = 800 FILE_RECORDS = 860 /* Pointer to First Record of Major Objects in File */ ^IMAGE = 40 ^IMAGE_HISTOGRAM = 840 ^ANCILLARY_TABLE = 842 /* Image Description */ SPACECRAFT_NAME = VOYAGER_2 TARGET_NAME = IO IMAGE_ID = "0514J2-00" IMAGE_TIME = 1979-07-08T05:19:11Z INSTRUMENT_NAME = NARROW_ANGLE_CAMERA EXPOSURE_DURATION = 1.9200 NOTE = "Routine multispectral longitude coverage, 1 of 7 frames" /* Description of the Objects Contained in the File */ OBJECT = IMAGE LINES = 800 LINE_SAMPLES = 800 SAMPLE_TYPE = UNSIGNED_INTEGER SAMPLE_BITS = 8 END_OBJECT = IMAGE OBJECT = IMAGE_HISTOGRAM ITEMS = 25 ITEM_TYPE = INTEGER ITEM_BITS = 32 END_OBJECT = IMAGE_HISTOGRAM OBJECT = ANCILLARY_TABLE ^STRUCTURE = "TABLE.FMT" END_OBJECT = ANCILLARY_TABLE END PDS label example Primary data object Secondary data object

Archiving PDS Standards Processing levels 1 – raw data: data with telemetry embedded 2 – edited data: corrected for telemetry error, tagged with time and location of acquisition, expressed in instrument units, also called Experimental Data Record (EDR) correpsonds to NASA level 0 data 3 – calibrated data: data values expressed in physical units, no resampling, edited data can still be reconstructed (NASA level 1A) 4 – resampled data: data resampled in space/time and calibrated, also called Reduced Data Record (RDR), edited data cannot be reconstructed (NASA level 1B) 5 – derived data: derived results (maps, reports, graphics, etc. NASA Levels 2 through 4). 6, 7, 8 – ancullary/correlative data and user descriptor needed to fully carachterize and process the data product(s)

ESA Planetary Science Archive (PSA) Archiving ESA Planetary Science Archive (PSA)

ESA Planetary Science Archive (PSA) Archiving ESA Planetary Science Archive (PSA)

ASI Space Science Data Center (SSDC) Archiving ASI Space Science Data Center (SSDC)

Archiving ASI SSDC - MATISSE Web based tool to access and visualize data from planetary exploration instruments

Archiving Summary Future missions SIMBIO-SYS (Bepi-Colombo): raw and calibrated data on ESA PSA, high-level product(s) on ASI SSDC Ma-MISS: raw and calibrated data on ESA PSA, NASA PDS and IKI ROSCOSMOS MicroMed: TBD MAJIS/JANUS (JUICE): calibrated data on ESA PSA, raw and high-level data product(s) on local rep. (INAF/IAPS or ASI SSDC) Ongoing missions NOMAD (ExoMars TGO): non-calibrated data on local rep. (INAF/IAPS), calibrated data on ESA PSA CASSIS (ExoMars TGO): raw and calibrated data on ESA PSA/NASA PDS, high-level product(s) (DTMs) o local rep. (INAF/OAPD) PFS (MarsExpress): raw data on local rep. (INAF/IAPS), calibrated data on ESA PSA/NASA PDS MARSIS (MarsExpress): EDR and RDR files on ESA PSA/NASA PDS, raw data at INAF/IAPS JIRAM (JUNO): EDR/RDR files on the NASA PDS, high-level producs on ASI SSDC EFD (CSES): raw/processed data at ASI SSDC downloaded from CNSA Terminated missions VIR (DAWN): processed data at NASA PDS, raw at INAF/IAPS VIRTIS (1) (Venus Express): raw/calibrated data at ESA PSA VIRTIS (1) (Rosetta): raw/calibrated/high-level product(s) at ESA PSA and NASA PDS (ESA requested the availability of high-level products – spectral and temp. maps – on the PSA) VIMS (Cassini): raw data on the NASA PDS with documentation. Calibrated data not available

Archiving Summary Future missions SIMBIO-SYS (Bepi-Colombo): raw and calibrated data on ESA PSA, high-level product(s) on ASI SSDC Ma-MISS: raw and calibrated data on ESA PSA, NASA PDS and IKI ROSCOSMOS MicroMed: TBD MAJIS/JANUS (JUICE): calibrated data on ESA PSA, raw and high-level data product(s) on local rep. (INAF/IAPS or ASI SSDC) Ongoing missions NOMAD (ExoMars TGO): non-calibrated data on local rep. (INAF/IAPS), calibrated data on ESA PSA CASSIS (ExoMars TGO): raw and calibrated data on ESA PSA/NASA PDS, high-level product(s) (DTMs) o local rep. (INAF/OAPD) PFS (MarsExpress): raw data on local rep. (INAF/IAPS), calibrated data on ESA PSA/NASA PDS MARSIS (MarsExpress): EDR and RDR files on ESA PSA/NASA PDS, raw data at INAF/IAPS JIRAM (JUNO): EDR/RDR files on the NASA PDS, high-level producs on ASI SSDC EFD (CSES): raw/processed data at ASI SSDC downloaded from CNSA Terminated missions VIR (DAWN): processed data at NASA PDS, raw at INAF/IAPS VIRTIS (1) (Venus Express): raw/calibrated data at ESA PSA VIRTIS (1) (Rosetta): raw/calibrated/high-level product(s) at ESA PSA and NASA PDS (ESA requested the availability of high-level products – spectral and temp. maps – on the PSA) VIMS (Cassini): raw data on the NASA PDS with documentation. Calibrated data not available

High-level data products Archiving High-level data products DTMs Spectral maps Geological maps

Archiving Propietary pipelines INAF/IAPS: NOMAD (internal Python scipts) PFS (internal tools) MARSIS (in-house proc. pipeline – Matlab scripts) VIR (internal propietary calibration pipeline) VIRTIS (1) (in-house proc. pipeline, Python reading software freely available) VIRTIS (2) (IDL processing pipeline) INAF/OAPD: CASSIS (high-level products – digital terrain models – GIS interface for request/targeting) SIMBIO-SYS (high-level products – digital terrain models)

CaSSIS – Data repository @ OAPD Archiving CaSSIS – Data repository @ OAPD Each Monday data download from mirror in UaBern Automatic shape files generation of the acquired FoVs

Data extraction with GIS tools Archiving CaSSIS – Data repository @ OAPD Data extraction with GIS tools GIS software (ArcGIS, QGIS) allows very fast data visualization, extraction and exploitation In the context of the CaSSIS mission, there are a large amount of data which will be impossible to exploit without the aid of GIS-based tools White polygons are some of the footprints of all the CaSSIS observations, which have been collected in a shapefile (i.e a database of observations) The shapefile can be queried, in order to rapidly extract observations at a given location (red dot) Queries can be very specific (i.e. filters, incidence, emission, phase, local time), in order to adapt to the scientific requirements The shapefile gathers all the CaSSIS data in one visualization, which can be integrate with data from other instrument on Mars, thus allowing the science team to fully exploit all existing data

Data extraction with GIS tools Archiving CaSSIS – Data repository @ OAPD Data extraction with GIS tools GIS software (ArcGIS, QGIS) allows very fast data visualization, extraction and exploitation In the context of the CaSSIS mission, there are a large amount of data which will be impossible to exploit without the aid of GIS-based tools White polygons are some of the footprints of all the CaSSIS observations, which have been collected in a shapefile (i.e a database of observations) The shapefile can be queried, in order to rapidly extract observations at a given location (red dot) Queries can be very specific (i.e. filters, incidence, emission, phase, local time), in order to adapt to the scientific requirements The shapefile gathers all the CaSSIS data in one visualization, which can be integrate with data from other instrument on Mars, thus allowing the science team to fully exploit all existing data

Conclusions Most of the instruments are cameras/spectrometers, each institute has developed different processing pipelines. Documentation on how processing the data can be found on PDS/PSA There is not a common policy on how to release high-level data producst (different missions = different policies) High-level data products are stored on different repositories (in INAF or at the ASI SSDC) My personal opinion... It would be nice to have, at least for the high-level data products a common repositoriy (in INAF) where data could be easily accessed A user-friendly (GIS-based?) interface (NASA Giovanni, JMARS, Mars Dune/Crater Database) would help to disseminate the usage and outcome from the space missions where INAF is involved

NASA Giovanni

JMARS: Java Mission-planning and Analysis for Remote Sensing (ASU) Available for Earth, Ceres and Vesta too!

Mars Global Digital Dune Database (MGD3)