1. UTG4 – Archiving and processing planetological data
Simone Silvestro
INAF Osservatorio Astronomico di Capodimonte, Napoli

2. Introduction: INAF instrument & missions
Data processing: USGS ISIS & INAF pipelines
Archciving: NASA PDS/ESA PSA & others

3. 15 Instruments (11 missions) Ongoing Terminated Future
Introduction
15 Instruments
(11 missions)
Ongoing
Terminated
Future
1) NOMAD
2) CaSSIS
3) MARSIS
4) PFS
5) EFD
6) JIRAM
1) VIR
2) VIRTIS (1)
3) VIRTIS (2)
1) MAJIS
2) JANUS
3) Ma-MISS
4) Micro-MED
5) SYMBIO-SYS
ExoMars 2016: NOMAD, CaSSIS
Mars Express: PFS, MARSIS
ExoMars 2020: Ma-MISS, Micro-MED
Cassini: VIMS
JUICE: MAJIS, JANUS
Venus Express: VIRTIS (1)
Rosetta: VIRTIS (2)
Juno: JIRAM
BEPI-Colombo: S-SYS
DAWN: VIR
CSES: EFD

4. Introduction Different bodies 1) NOMAD 2) CaSSIS 3) PFS 4) MARSIS
Jupiter
&
Satellites
Saturn
&
Satellites
Venus
Asteroids
&
Comets
Earth
Mercury
1) NOMAD
2) CaSSIS
3) PFS
4) MARSIS
5) Ma-MISS
6) MicroMED
1) JIRAM
2) MAJIS
3) JANUS
1) VIMS
1) VIRTIS (1)
1) VIR
2) VIRTIS (2)
1) EFD
1) SYMBIOSYS

5. Instrument types Camera/Spectrometer Radar GPR Others
Introduction
Instrument types
Camera/Spectrometer
Radar GPR
Others
1) NOMAD
2) CaSSIS (stereo)
3) PFS
4) Ma-MISS
5) JIRAM
6) MAJIS
7) JANUS (stereo)
8) VIRTIS (1)
9) VIRTIS (2)
10) VIR
11) SYMBIO-SYS (stereo)
12) VIMS
1) MARSIS
1) Micro-MED
2) EFD

6. Introduction: INAF instrument & missions
Data processing: USGS ISIS & INAF pipelines

7. *USGS ISIS/Propietary pipeline ONLY propietary pipeline/ISIS TBD
Processing
Processing pipelines
*USGS ISIS/Propietary pipeline
ONLY propietary pipeline/ISIS TBD
1) VIRTIS (1) – Venus Exp.
2) VIRTIS (2) – Rosetta
3) VIR – DAWN
4) CaSSIS – TGO ExoMars 2016
1) MAJIS - JUICE
2) JANUS - JUICE
3) JIRAM - Juno
4) NOMAD – TGO ExoMars 2016
5) Ma-Miss – ExoMars 2020
6) Micro-MED
7) PFS – Mars Express
8) MARSIS – Mars Express
9) SIMBIO-SYS – Bepi Colombo
10) EFD - CSES
*Integrated Software for Imagers and Spectrometers

8. Processing
Propietary pipelines
INAF/IAPS:
NOMAD (internal Python scipts)
PFS (internal tools)
MARSIS (in-house proc. pipeline – Matlab scripts)
VIR (internal propietary calibration pipeline)
VIRTIS (1) (in-house proc. pipeline, Python reading software freely available)
VIRTIS (2) (IDL processing pipeline)
INAF/OAPD:
CASSIS (high-level products – digital terrain models – GIS interface for request/targeting)
SIMBIO-SYS (high-level products – digital terrain models)

9. Processing
USGS ISIS

10. Processing USGS ISIS Example of a processing pipeline in USGS ISIS
Three processing steps valid for most cameras/spectrometers:
Level 0 – Data ingestion
Acquire and convert image data files (PDS format) to the ISIS3 image format (cubes)
Add information to the ISIS3 image in order to compute geometric properties such as latitude/longitude range and illumination angles of the image
Level 1 (A and B) – Radiometric Calibration and Noise Removal
Convert raw pixel numbers to reflectance (irradiance/solar flux or I/F)
Remove noise
Level 2, 3 – Projection
Geometrically rectify to a map projection

11. NASA processing levels
0 – reconstructed, unprocessed instrument and payload data at full resolution, with artifacts removed
1A – reconstructed, unprocessed instrument data at full resolution, time-referenced, and annotated with ancillary information, including radiometric and geometric calibration coefficients and georeferencing parameters (e.g., platform ephemeris) computed and appended but not applied to Level 0 data
1B – level 1A data that have been processed to sensor units
2 – derived geophysical variables at the same resolution and location as Level 1 source data
3 – variables mapped on uniform space-time grid scales, usually with some completeness and consistency
4 – model output or results from analyses of lower-level data (e.g., variables derived from multiple measurements)

12. Introduction: INAF instrument & missions
Data processing: USGS ISIS & INAF pipelines
Archciving: NASA PDS/ESA PSA & others

13. Archiving
Global repositories
NASA PDS
ESA PSA
Local repositories
ASI SSDC
INAF
Others
CNSA
IKI ROSCOSMOS
All the data produced by the instruments listed in this ppt should be NASA PDS/ESA PSA compliant

15. Archiving NASA PDS – 8 main nodes
6 science discipline nodes
2 support nodes

18. Archiving
PDS Standards
PDS Data product = Data object + label file describing the object
Data files: ASCII or binary format
Primary data object: set of results from a scientific observation (TABLE, SPREADSHEET, IMAGE, SERIES, SPECTRUM, QUBE)
Secondary data object: any data used for processing or interpreting the primary data object(s) (HISTOGRAM, PALETTE, HEADER)
Label files: written in Object Description Language (ODL) in the format «keyword = value», defines both the physical and logical structure of the constituent data object(s)

19. PDS label example Primary data object Secondary data object
/* File Format and Length */
RECORD_TYPE = FIXED_LENGTH
RECORD_BYTES = 800
FILE_RECORDS = 860
/* Pointer to First Record of Major Objects in File */
^IMAGE = 40
^IMAGE_HISTOGRAM = 840
^ANCILLARY_TABLE = 842
/* Image Description */
SPACECRAFT_NAME = VOYAGER_2
TARGET_NAME = IO
IMAGE_ID = "0514J2-00"
IMAGE_TIME = T05:19:11Z
INSTRUMENT_NAME = NARROW_ANGLE_CAMERA EXPOSURE_DURATION =
NOTE = "Routine multispectral longitude coverage, 1 of 7 frames"
/* Description of the Objects Contained in the File */
OBJECT = IMAGE
LINES = 800
LINE_SAMPLES = 800
SAMPLE_TYPE = UNSIGNED_INTEGER
SAMPLE_BITS = 8
END_OBJECT = IMAGE
OBJECT = IMAGE_HISTOGRAM
ITEMS = 25
ITEM_TYPE = INTEGER
ITEM_BITS = 32
END_OBJECT = IMAGE_HISTOGRAM
OBJECT = ANCILLARY_TABLE
^STRUCTURE = "TABLE.FMT"
END_OBJECT = ANCILLARY_TABLE
END
PDS label example
Primary data object
Secondary data object

20. Archiving
PDS Standards
Processing levels
1 – raw data: data with telemetry embedded
2 – edited data: corrected for telemetry error, tagged with time and location of acquisition, expressed in instrument units, also called Experimental Data Record (EDR) correpsonds to NASA level 0 data
3 – calibrated data: data values expressed in physical units, no resampling, edited data can still be reconstructed (NASA level 1A)
4 – resampled data: data resampled in space/time and calibrated, also called Reduced Data Record (RDR), edited data cannot be reconstructed (NASA level 1B)
5 – derived data: derived results (maps, reports, graphics, etc. NASA Levels 2 through 4).
6, 7, 8 – ancullary/correlative data and user descriptor needed to fully carachterize and process the data product(s)

21. ESA Planetary Science Archive (PSA)
Archiving
ESA Planetary Science Archive (PSA)

22. ESA Planetary Science Archive (PSA)
Archiving
ESA Planetary Science Archive (PSA)

23. ASI Space Science Data Center (SSDC)
Archiving
ASI Space Science Data Center (SSDC)

24. Archiving ASI SSDC - MATISSE
Web based tool to access and visualize data from planetary exploration instruments

25. Archiving Summary Future missions
SIMBIO-SYS (Bepi-Colombo): raw and calibrated data on ESA PSA, high-level product(s) on ASI SSDC
Ma-MISS: raw and calibrated data on ESA PSA, NASA PDS and IKI ROSCOSMOS
MicroMed: TBD
MAJIS/JANUS (JUICE): calibrated data on ESA PSA, raw and high-level data product(s) on local rep. (INAF/IAPS or ASI SSDC)
Ongoing missions
NOMAD (ExoMars TGO): non-calibrated data on local rep. (INAF/IAPS), calibrated data on ESA PSA
CASSIS (ExoMars TGO): raw and calibrated data on ESA PSA/NASA PDS, high-level product(s) (DTMs) o local rep. (INAF/OAPD)
PFS (MarsExpress): raw data on local rep. (INAF/IAPS), calibrated data on ESA PSA/NASA PDS
MARSIS (MarsExpress): EDR and RDR files on ESA PSA/NASA PDS, raw data at INAF/IAPS
JIRAM (JUNO): EDR/RDR files on the NASA PDS, high-level producs on ASI SSDC
EFD (CSES): raw/processed data at ASI SSDC downloaded from CNSA
Terminated missions
VIR (DAWN): processed data at NASA PDS, raw at INAF/IAPS
VIRTIS (1) (Venus Express): raw/calibrated data at ESA PSA
VIRTIS (1) (Rosetta): raw/calibrated/high-level product(s) at ESA PSA and NASA PDS (ESA requested the availability of high-level products – spectral and temp. maps – on the PSA)
VIMS (Cassini): raw data on the NASA PDS with documentation. Calibrated data not available

26. Archiving Summary Future missions
SIMBIO-SYS (Bepi-Colombo): raw and calibrated data on ESA PSA, high-level product(s) on ASI SSDC
Ma-MISS: raw and calibrated data on ESA PSA, NASA PDS and IKI ROSCOSMOS
MicroMed: TBD
MAJIS/JANUS (JUICE): calibrated data on ESA PSA, raw and high-level data product(s) on local rep. (INAF/IAPS or ASI SSDC)
Ongoing missions
NOMAD (ExoMars TGO): non-calibrated data on local rep. (INAF/IAPS), calibrated data on ESA PSA
CASSIS (ExoMars TGO): raw and calibrated data on ESA PSA/NASA PDS, high-level product(s) (DTMs) o local rep. (INAF/OAPD)
PFS (MarsExpress): raw data on local rep. (INAF/IAPS), calibrated data on ESA PSA/NASA PDS
MARSIS (MarsExpress): EDR and RDR files on ESA PSA/NASA PDS, raw data at INAF/IAPS
JIRAM (JUNO): EDR/RDR files on the NASA PDS, high-level producs on ASI SSDC
EFD (CSES): raw/processed data at ASI SSDC downloaded from CNSA
Terminated missions
VIR (DAWN): processed data at NASA PDS, raw at INAF/IAPS
VIRTIS (1) (Venus Express): raw/calibrated data at ESA PSA
VIRTIS (1) (Rosetta): raw/calibrated/high-level product(s) at ESA PSA and NASA PDS (ESA requested the availability of high-level products – spectral and temp. maps – on the PSA)
VIMS (Cassini): raw data on the NASA PDS with documentation. Calibrated data not available

27. High-level data products
Archiving
High-level data products
DTMs
Spectral maps
Geological maps

28. Archiving
Propietary pipelines
INAF/IAPS:
NOMAD (internal Python scipts)
PFS (internal tools)
MARSIS (in-house proc. pipeline – Matlab scripts)
VIR (internal propietary calibration pipeline)
VIRTIS (1) (in-house proc. pipeline, Python reading software freely available)
VIRTIS (2) (IDL processing pipeline)
INAF/OAPD:
CASSIS (high-level products – digital terrain models – GIS interface for request/targeting)
SIMBIO-SYS (high-level products – digital terrain models)

29. CaSSIS – Data repository @ OAPD
Archiving
CaSSIS – Data OAPD
Each Monday data download from mirror in UaBern
Automatic shape files generation of the acquired FoVs

30. Data extraction with GIS tools
Archiving
CaSSIS – Data OAPD
Data extraction with GIS tools
GIS software (ArcGIS, QGIS) allows very fast data visualization, extraction and exploitation
In the context of the CaSSIS mission, there are a large amount of data which will be impossible to exploit without the aid of GIS-based tools
White polygons are some of the footprints of all the CaSSIS observations, which have been collected in a shapefile (i.e a database of observations)
The shapefile can be queried, in order to rapidly extract observations at a given location (red dot)
Queries can be very specific (i.e. filters, incidence, emission, phase, local time), in order to adapt to the scientific requirements
The shapefile gathers all the CaSSIS data in one visualization, which can be integrate with data from other instrument on Mars, thus allowing the science team to fully exploit all existing data

31. Data extraction with GIS tools
Archiving
CaSSIS – Data OAPD
Data extraction with GIS tools
GIS software (ArcGIS, QGIS) allows very fast data visualization, extraction and exploitation
In the context of the CaSSIS mission, there are a large amount of data which will be impossible to exploit without the aid of GIS-based tools
White polygons are some of the footprints of all the CaSSIS observations, which have been collected in a shapefile (i.e a database of observations)
The shapefile can be queried, in order to rapidly extract observations at a given location (red dot)
Queries can be very specific (i.e. filters, incidence, emission, phase, local time), in order to adapt to the scientific requirements
The shapefile gathers all the CaSSIS data in one visualization, which can be integrate with data from other instrument on Mars, thus allowing the science team to fully exploit all existing data

32. Conclusions
Most of the instruments are cameras/spectrometers, each institute has developed different processing pipelines. Documentation on how processing the data can be found on PDS/PSA
There is not a common policy on how to release high-level data producst (different missions = different policies)
High-level data products are stored on different repositories (in INAF or at the ASI SSDC)
My personal opinion...
It would be nice to have, at least for the high-level data products a common repositoriy (in INAF) where data could be easily accessed
A user-friendly (GIS-based?) interface (NASA Giovanni, JMARS, Mars Dune/Crater Database) would help to disseminate the usage and outcome from the space missions where INAF is involved

33. NASA Giovanni

34. JMARS: Java Mission-planning and Analysis for Remote Sensing (ASU)
Available for Earth, Ceres and Vesta too!

35. Mars Global Digital Dune Database (MGD3)