September 2003ERPANET - Marburg 2003View # 1 Data preservation Using CCSDS and ISO Standards Denis MINGUILLON

September 2003ERPANET - Marburg 2003View # 2 The problem to solve Data is often expensive to produce –Bits must not be lost (preserved medium) –Bits must remain accessible (known syntax) –Bits must remain understandable (known semantics) This presentation focuses on point 2 and 3

September 2003ERPANET - Marburg 2003View # 3 The syntactic aspect How to master the bit organisation –2 possible solutions Define exchange formats –Better done by user communities (ex : FITS for astronomy) –Not generic enough Define a standard way for describing any data –Was the CNES proposal to CCSDS –Was entrusted to CNES (definition and tools) by CCSDS

September 2003ERPANET - Marburg 2003View # 4 History CNES proposed a description language –EAST for data syntax –CCSDS recommendation since May 1997 CCSDS B-2 Issue 2 November 2000 –ISO standard since 2000 ISO CCSDS recommendations

September 2003ERPANET - Marburg 2003View # 5 EAST principles EAST is a standard language –Enhanced Ada SubseT inherits from a huge standardisation work enhanced with a few features missing for data description EAST has not to be mastered to be used thanks to the tools Data syntax description

September 2003ERPANET - Marburg 2003View # 6 EAST principles (2) Assumption : any kind* of digital data (available as an ordered set of bits) can be described as a tree. The minimal tree is a rake * note : optimal for binary data, interesting for raw ASCII format, of no use for XML data To describe such a tree, EAST proposes : composite types for the branches elementary types for the leaves Data syntax description

September 2003ERPANET - Marburg 2003View # 7 EAST data types Composite types are : records arrays lists Elementary types are : integers and reals enumerated characters Data syntax description

September 2003ERPANET - Marburg 2003View # 8 Optionality Some branches or leaves of the data tree may be optional. –EAST offers a way to indicate which data entities are optional and under which condition they exist –Example : Housekeeping data exists if telemetry mode is set to « housekeeping » Data syntax description

September 2003ERPANET - Marburg 2003View # 9 Variable size Some repetitions of data entities may have a variable number of elements –EAST offers a means to indicate how many of those elements are present in the current data Data syntax description

September 2003ERPANET - Marburg 2003View # 10 EAST use conclusion All the features described in the previous slides have been sufficient to describe almost every data encountered since 1997 –Exceptions : variable size not carried by the data variable size provided after the variable size field Data syntax description

September 2003ERPANET - Marburg 2003View # 11 History (2) CNES has also been in charge of writing the recommendation for data semantics description : –DEDSL (D ata E ntity D ictionary S pecification L anguage ) –XML/DTD syntax –CCSDS recommendation since January 2002 CCSDS B-1 Issue 1 January 2002 –ISO standard since 2003 ISO : 2003 CCSDS recommendations

September 2003ERPANET - Marburg 2003View # 12 DEDSL principles The data semantics description can be kept as a set of attribute values. The recommendation defines standard attributes : name, definition, units…. The recommendation gives a standard way to define user attributes to be added to the standard ones if needed Data semantics description

September 2003ERPANET - Marburg 2003View # 13 DEDSL principles (2) A data description consists in the tree organisation (as seen on the sysntactic aspect) completed by all the fields (nodes or leaves) caracteristics given as attributes values. Example of the descripon of a leaf of such a tree description.Example

September 2003ERPANET - Marburg 2003View # 14 OASIS OASIS is the tool to describe data (syntax and semantics) in a friendly way (without any requested knowledge of the EAST or DEDSL syntax) A dedicated Man-Machine interface offers the requested features (those listed previously) to describe data A tool to describe data

September 2003ERPANET - Marburg 2003View # 15 OASIS (2) Example :Example a data tree as built and shown by OASIS A tool to describe data

September 2003ERPANET - Marburg 2003View # 16 OASIS (3) Once the tree and each of its nodes and leaves described, the descriptions can be produced just by a click ( EAST and DEDSL-XML files ) Once saved, the EAST description can be used by the EAST tools to read and write the described data Once saved, the XML dictionary can be published as an html or word document A tool to describe data

September 2003ERPANET - Marburg 2003View # 17 EAST tools Application Programming Interface (API) GENERATOR : for data writing INTERPRETER : for data reading –Both available for C, C++, FORTRAN and Ada Standalone tools (based on the API) data_checker : data vs description compliance ascii_dump : display any data in ascii values DUW : interactive data generator other utilities… Tools for EAST described data

September 2003ERPANET - Marburg 2003View # 18 EAST tools (2) EAST tools are available on : –PC (DOS, NT, Linux), –HP-UX, –SUN-Solaris, –Dec true64 (OSF1), –VAX-VMS –other on request…. Tools for EAST described data

September 2003ERPANET - Marburg 2003View # 19 Interface Control Documents The DEDS-XML OASIS output can be transformed in an html or Word document A tool (named SCRIBE) is being developed for that purpose. Documenting the interfaces

September 2003ERPANET - Marburg 2003View # 20 Synthesis Data dictionary DEDSL/XML XML Data EAST Apis Html or Word document DCI generation File descriptions XML schema Read Write Generate Check ‘RAW’ Data OASIS tool File descriptions EAST Read Write Generate Check XML applications

September 2003ERPANET - Marburg 2003View # 21 Main advantages Only one consistent reference for syntactic description used for input/output and for semantic description (documentation for understanding) Reusable and reliable generic input/output software available User application easier to maintain due to less impact from changes of data formats Help in describing data and producing documentation The project conclusions

September 2003ERPANET - Marburg 2003View # 22 EAST/DEDSL-XML in action Many projects at CNES –SPOT, SSALTO, SIPAD, HELIOS, PHARAO… ESA projects –AMS, DEBAT NASA –Science data rehabilitation Use in other projects

September 2003ERPANET - Marburg 2003View # 23 EAST/DEDSL-XML feed back PROs : Easier sub-system integration –description accuracy (even after long time!!!) –early data simulation –assisted data checking Easier ICD evolutions –full top-down process Helpful in heterogeneous platforms context –automatic conversions Helpful in multi-missions context Projects feedback

September 2003ERPANET - Marburg 2003View # 24 EAST/DEDSL-XML feed back (2) CONs : Performances to be again improved –reading in time constrained context –writing in any case User type sharing to be added –to facilitate evolutions Documentation production to be improved –project customisation –add as much filters as possible to allow adaptation of the document to its target Projects feedback

September 2003ERPANET - Marburg 2003View # 25 Current actions Documentation tool in progress generic tool to transform XML-DED in Word or html documents 2nd prototype delivered Main lacks taken into account by the DEBAT project (ESA/ESRIN) Support to NASA tests on data migration - good first results Support to XML schema production On going activity

September 2003ERPANET - Marburg 2003View # 26 CNES policy for the EAST tools OASIS and EAST tools are freely available They can be downloaded from a CNES server Tools are maintained by a CNES team and a sub-contractor team (CS-SI) Policy : if CNES stops founding the maintenance the source code would be made available Information

September 2003ERPANET - Marburg 2003View # 27 Links For any support request : ( address) For information and documentation east.cnes.fr (web server) To download the software : logiciels.cnes.fr (web server) To get the standards (web server) –then under publications, recommendations and reports, information access and interchange systems. Information