CCSDS Meeting data Archive Ingest - January 2007 CNES 1 CCSDS - MOIMS Area Data Archive Ingest WG CNES Report Colorado Springs meeting – January 2007 Claude Huc

CCSDS Meeting data Archive Ingest - January 2007 CNES 2 Contents  CNES standards, evolution of data engineering domain (RNC)  A new data standard format for the CDPP  Current work on the ‘Producer Archive Interface Specification’

CCSDS Meeting data Archive Ingest - January 2007 CNES 3 CNES standards (RNC) current content of the data engineering domain At the highest level v An introductory document justifying its existence v The OAIS reference model On a technical level v A document defining Criteria for evaluating data formats v The DEDSL abstract standard v The DEDSL XML/DTD syntax v The EAST specification v The PAIMAS v Applicable rules and recommendations for projects producing data New document planned in 2007 : v Long term archiving of data: Applicable rules and recommendations for archive services

CCSDS Meeting data Archive Ingest - January 2007 CNES 4 A new data standard format for the CDPP - Plasma Physics data Center - Main existing data standard formats in space physics community :  CDF – Common data Format A very good format for data analysis v Number of general graphic, statistic and analysis tools available Efficient for access and file size  however Not really powerful for the metadata definition and management Representation information apparently not fully public and quite complex Used more for summary data than for high resolution data  Seems not really appropriate for long term preservation

CCSDS Meeting data Archive Ingest - January 2007 CNES 5 A new data standard format for the CDPP Main existing data standard formats in space physics community  Cluster Exchange Format Almost the high resolution data in the Cluster Data Archive at ESTEC are CEF conformant A significant number of the space physics area are covered  However Specific syntax for the metadata Ascii files with impact v On file sizes v On access performances  Other used formats : NetCDF, FITs…  In practice : no standard format in the plasma physics community  Consequences for the CDPP : about 200 high resolution data sets and about 150 different data format

CCSDS Meeting data Archive Ingest - January 2007 CNES 6 Main objectives for a new CDPP format  A Format for the long term preservation  A format allowing multiple and generic added value services  To make easier the data access  To make easier the data analysis  Allowing simple field extraction  Make easier the creation of composed parameters ( alfven velocity,…)  …

CCSDS Meeting data Archive Ingest - January 2007 CNES 7 The metadata  A data set conformant with the CDPP format specification includes A set of files One XML Metadata  These metadata are base on the CDPP and CAA (Cluster Active Archive) dictionaries  The XML metadata file describes the parameter level metadata (semantics and syntactic features) General parameters of the files in this data set : type of coding (binary/ascii), type of date/time…)  The metadata can be extended without impact on the data files  Metadata block structure : syntax, parent-child relation, scientific interpretation…

CCSDS Meeting data Archive Ingest - January 2007 CNES 8 Main features of the data files  A very simple structure under the form of a temporal series Standard (10 -6 sec) or precise (until sec) dates and time  Binary or ASCII files (defined in the XML metadata file)  Management of scalars, vectors, multidimensional arrays.  Management of variable size arrays

CCSDS Meeting data Archive Ingest - January 2007 CNES 9 Associated tools  Read and write Java and C libraries  ASCII  Binary transformation  Toof for creation new data sets conformpant with CDPP format from existing data  Tools currently under development  translation CDF  CDPP.  translation CEF  CDPP.  Translation NetCDF  CDPP  Tool to make easier the metadata file creation Les outils en cours de développement  Convertisseur CDF  CDPP.  Convertisseur CEF  CDPP.  Outil d’aide à la constitution du fichier de métadonnées.

CCSDS Meeting data Archive Ingest - January 2007 CNES 10 Ongoing work for Producer Archive Interface Specification  The document is currently being thoroughly revised Progressive convergence between NASA and CNES for the concepts and the terminology See Daniele’s presentation

CCSDS Meeting data Archive Ingest - January 2007 CNES 11 The end…