C. Huc/CNES, D. Boucon/CNES-SILOGIC Producer-Archive Interface Specification

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 2 Main changes  Main changes between V01 and V02 Documents sent on Descriptor Definition (end May 2005) and SIPs (July 2005). Concepts have changed: suppression of Slips, Generic Descriptor Model. Implementation parts (sections 4 and 5) Completed and rewritten document.

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 3 Contents  Introduction Objectives Main steps General framework Example  General principles for implementing the Producer-Archive Interface Process description Plan of Objects to be Transferred (POT) SIPs Sequencing constraints  Formal Phase Specification  Transfer Phase and Validation Phase Specification  Managing modifications  Tools  Conclusion

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 4  Provide a standard method to formally define the digital information objects to be transferred by a Producer to an Archive effectively transfer these objects in the form of SIPs validate the SIPs Introduction Objectives (reminder)

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 5 Preliminary Phase Formal Definition Phase Transfer Phase Validation Phase Introduction General framework: PAIMAS Phases Initial contacts, feasibility study, draft of the SIPs, Preliminary Agreement Precise definition of the digital objects, SIP design, Submission Agreement Actual transfer of the SIPs Actual validation processing of the SIPs by the Archive and Required follow-up action with the Producer Organisational sub-phase Formal definition sub-phase Complete and precise definition of the data to be transferred and the way they are organised in the form of a SIP Complementary points (contractual and legal aspects, …) Submission Agreement implementing tests sub-phase performing the transfer sub-phase (technical means –XFDU …)

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 6  How can we define an object (and have a sufficiently precise description without creating the objects)?  How can we define the organisation between the objects (« Plan of Objects to be Transferred » )?  How can we define a complete packaging description for an object and a group of objects?  How can we adapt the general framework to the domain specificities and project needs?  How can we specify the constraints (granularity and sequencing) applying to these objects?  How can we validate the transferred objects?  How can we have a unique visual representation of the objects to transfer and their constraints? Introduction General framework: needs to meet (reminder)

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 7 Introduction Example WIND WAVES Experiment WIND WAVES TNR L2 Data set WIND WAVES TNR L2 Data (1..N) Complementary Data Objects EAST Description (1) Document (1) Space Physics domain WIND mission WAVES experiment WIND WAVES TNR L2 data set Complementary Data Objects: EAST syntactic descriptor Mission description document

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 8 Contents  Introduction Objectives Main steps General framework Example  General principles for implementing the Producer-Archive Interface Process description Plan of Objects to be Transferred (POT) SIPs Sequencing constraints  Formal Phase Specification  Transfer Phase and Validation Phase Specification  Managing modifications  Tools  Conclusion

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 9 General principles Process, from Object description to SIP Model instances Object content description SIP constraints SIP content description Time Formal Definition Phase Transfer and validation Phases POT SIP Models Sequencing constraints Submission Agreement Descriptor instances SIP Model instances Descriptor Models Digital Objects Data Rules Models and POT creation

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 10 General Principles Process, from SIP creation to SIP acceptance Transfer management Anomaly management Time SIP reception SIP validation Producer SIP creation SIP transfer Archive Control Ok Yes AIPs No Transfer and validation Phases Reception acknowledgement Anomaly acknowledgement Data SIP Model instances

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 11 Plan of Objects to be Transferred (POT)  Gives a complete and overall view of all digital objects to be transferred: described by Descriptors. Descriptor : This is an information unit for describing a set of characteristics for a given Data Object. A Descriptor may come in the form of a model; in this case, it is used to identify the object category which can then be described as well as the attributes for this description. The Descriptor may have the state of an instance; in this case, it describes an object instance and supplies the attribute values defined in the Descriptor model. –POT creation: dictionary for project Descriptor Models, SIP Models, Descriptor instances –POT constraint: All the Objects defined in a Descriptor must be delivered together in the same SIP. –POT representation General principles Plan of Objects to be Transferred

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 12  Process –Generic Descriptor Model Structure –Specialisation Phase –Instantiation Phase of the Project Descriptor Models General principles Plan of Objects to be Transferred: Dictionary

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 13  Descriptor Model, attributes for ( see section page 16 ): –Descriptor identification: identifying the type of object described. –Content description: describing the object content. –Relationships within the POT: defining the relations between the object described and the other Model objects. –Object content: describes all the Data and Metadata included in this Descriptor. –Specialisation section. –SIP Model reference. General principles Plan of Objects to be Transferred: the Descriptors

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 14  Identification (M, 1..1) –descriptor_type (M, 1..1): defines the type of Object Model and is constant for a given type of Descriptor. –descriptor_ID (M, 1..1): Descriptor Identifier, used to identify the nodes in the POT. –object_occurrence (M, 1..1): number of Data Objects described by this descriptor_ID. This Descriptor instance may describe a unique Object (this is often the case for a Data Collection, or a syntactical description of a Data Collection), or a representative of a category containing a certain number of Objects. This number may not be known ahead of time (value 1..n). –version (M, 1..1). General principles Plan of Objects to be Transferred: the Descriptors

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 15  Content description (M, 1..1) –title (M, 1..1): extensive name of the Object. –content (M, 1..1): explanatory text specifying the content and main characteristics of the described Object. –size (O, 0..1): estimated volume of an object and unit. –Non definite attribute (O, 0..1): specialization attribute (to be defined if necessary). The name and form of this attribute depends on the future implementation. General principles Plan of Objects to be Transferred: the Descriptors

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 16  Relationships within the POT (M, 1..1) –parent_collection (M, 1..1): identifier of Collection Descriptor to which this object belongs. –association (O, 0..N): this is used when the Descriptor describes a Complementary Data Object or a Complementary Collection (transversal links). related_descriptor_ID (M, 1..1) : Descriptor identifier for the Data Object Collection or Data Object to which this Object refers relation_with_DO (O, 0..1): type of relation between the Object in question and the related_descriptor_ID. General principles Plan of Objects to be Transferred: the Descriptors

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 17  Object content (O, 0..N). –content_ID (O, 0..1): the Data section describes the Data Object itself, while the Metadata section describes the Metadata attached to the Data. –data (O, 0..1): data_ID (M, 1..1): Data Identifier. data_content (O, 0..1): Explanatory text specifying the content and main characteristics of the described Data data_format (O, 0..1): Format of objects in the collection (PNG, PDF, CDF, Flat Binary, Flat ASCII, …) –Metadata (O, 0..1) metadata_ID (M, 1..1): Metadata Identifier. metadata_content (O, 0..1): Explanatory text specifying the content and main characteristics of the described Metadata. metadata_format (O, 0..1): Used to specify the metadata standard to be applied. metadata_schema (O, 0..1): Used to specify the ‘catalogue’ data characterising the object: for instance in the form of an XML document specified by an XML Schema. General principles Plan of Objects to be Transferred: the Descriptors

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 18  Non definite section: optional section in which can be included any attributes to specialise the Descriptor.  SIP Model reference: SIP Model Identifier. For a collection of collections, there may not be a corresponding delivery during the Transfer Phase, and hence no applicable SIP Model: the attribute value is then NONE. General principles Plan of Objects to be Transferred: the Descriptors

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 19  Addition of an attribute by using the: –“Non definite attribute” (if related to information description). –“Non definite section”.  Elimination of an optional attribute.  Modification of existing attributes: –by occurrence restriction, by modification of status attribute (e.g. optional changed in mandatory). General principles Plan of Objects to be Transferred: specialisation Generic DescriptorModelDomain Descriptor Models Producer-Archive Project Descriptor Models

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 20 General principles Plan of Objects to be Transferred: instances  Most crucial task during the Formal Definition Phase  definition of the actual objects to be transferred: –Identifying all of the project data Collections, –Organising these Collections in a hierarchical tree structure, –Defining a Descriptor instance for each of the Collections, –Defining a Data Object Descriptor for the objects in each Collection, –… Building the POT

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 21 General principles Plan of Objects to be Transferred: instances The SIPTemplate describes the organisation of the Unit Package This part contains everything that is known about the object at the time of the Formal Definition Phase (characteristics of the object and its relations to other objects)

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 22 General principles Plan of Objects to be Transferred: example ROOT WIND_WAVES WA_TNR_L2_CO METADATA_STANDARD=DIF WA_TNR_L2_DO (N) METADATA_MODEL=CDPP_DO.xsl WIND_WAVES_CC EAST_DESCRIPTION (1) WAVES_DOCUMENTATION (1) Object Descriptor ID SPACE_PHYSICS_SIP Wind_waves_tnr_l2_aaammjj.dat Wind_waves_tnr_l2_aaammjj.xml EAST_SIP Wind_waves_tnr_l2.east EXPERIMENT_DESCRIPTION_SIP Wind_waves_22000_0387.pdf SPACE_PHYSICS_COLLECTION_SIP Wind_waves_tnr.dif N Files to be transferred

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 23 Contents  Introduction Objectives Main steps General framework Example  General principles for implementing the Producer-Archive Interface Process description Plan of Objects to be Transferred (POT) SIPs Sequencing constraints  Formal Phase Specification  Transfer Phase and Validation Phase Specification  Managing modifications  Tools  Conclusion

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 24  Coherent group of Digital Objects in a package to be transferred (objects of the same or different types). –May contain pointer towards the Digital Objects.  Defined during the Formal Definition Phase: –Rules for grouping objects. –SIP categories for the project General principles SIPs

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 25 General principles Links between SIP and Descriptor Descriptor Model Attribute = descriptor_ID … Attribute = SIP Model ID descriptor_ID = A … SIP Model ID = sipC Instantiated Descriptor descriptor_ID = B … SIP Model ID = sipC … Descriptor_ID = A Data ________________ Descriptor_ID = B Data SIP Model ID = sipC …. Attribute = descriptor_ID … Delivered SIP SIP Model Formal Definition PhaseTransfer Phase

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 26 General principles SIP Models Generic SIP Model Domain SIP Models Producer-Archive Project SIP Models Process for the development of the Project SIP Models

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 27 General principles SIP Models Producer ID Project ID Sip Model ID SIP Model location descriptor_ID Descriptor location last_object updated_object Other SIP attributes Digital Object Each sub-package is associated to 1 Descriptor instance  SIP structure (see page 22):  Global information (M, 1..1)  Information associated with each embedded object (M, 1..N)  Other information (M, 1..1)  Digital Object (O,0..N)

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 28  A given digital object may have to be transmitted before or after another.  Constraint group: set of objects related to each other by a set of dependent sequencing constraints. General principles Sequencing Constraints Attribute nameMeaningValue syntaxOccurrence TIME_CONSTRAINT_GROUPdefine the groupsidentifier0..1 SERIAL_NUMBER_IN_CONSTRAINTdefine the constraints within the same group integer0..1

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 29 General principles Sequencing Constraints SIP1: Collection WIND_WAVES_TNR_L2 SIP2: Complementary Objects EAST file and document: is delivered before any delivery of SIP3. SIP3: Data Objects Example of SIP3 instance AttributeValue SIP_MODEL_IDSIP3 SIP_CONTENTWIND_WAVES_TNR_L2_DATA TIME_CONSTRAINT_GROUP Group_1 SERIAL_NUMBER_IN_CONSTRAINT 2 Example of SIP2 instance AttributeValue SIP_MODEL_IDSIP2 DESCRIPTOR_IDEAST_DESCRIPTION DESCRIPTOR_IDDOCUMENTATION TIME_CONSTRAINT_GROUP Group_1 SERIAL_NUMBER_IN_CONSTRAINT 1 1 2

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 30 Contents  Introduction Objectives Main steps General framework Example  General principles for implementing the Producer-Archive Interface Process description Plan of Objects to be Transferred (POT) SIPs Sequencing constraints  Formal Phase Specification  Transfer Phase and Validation Phase Specification  Managing modifications  Tools  Conclusion

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 31 Formal Phase Specification Global view project_ID Descriptor instances Data dictionary creation Anomaly management POT creation Sequencing constraints SIP templates creation Producer descriptor_ID descriptor_type object_occurrence title parent_collection content_ID data_ID metadata_ID … sip_template_ID time_constraint_group serial_number_in_constraint descriptor_ID (1..n) anomaly_ID date … Archive POT message Validation SIP templates Ingest Base

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 32 Formal Phase Specification POT creation  Proposed implementation based on XML Schema and XML files.  Process –Generic Descriptor Model: complete enough to cover the needs for the description of different object categories (Data Objects or Complementary Data Objects), flexible enough to adapt to the specialisations required. Descriptor instances Data Dictionary POT Validation

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 33 Formal Phase Specification Generic Descriptor Model Implementation

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 34 Formal Phase Specification Generic Descriptor Specialisation  The generic Descriptor can be specialized by: –suppression of optional elements ; –modification of occurrence numbers (for example instead of 1..n); –addition of new elements (or sequence of elements) by the use of the groups « other » or « any »; –definition of a list (enumerate) for the elements « descriptor_type » or « sip_template_ID »; –definition of patterns for identifiers (imposed nomenclature) or restriction of string length.

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 35 Formal Phase Specification Descriptor instances and POT creation  XML files  Based on the previous specialised schemas  Instantiated in several times during the Formal Definition Phase  Use of an XML editor or forms  Base of the information contained in the Descriptor instances  Validation (XML schemas, coherence of the nodes, …) POT visualisation (mock-up)

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 36 Formal Phase Specification SIP implementation  based on XFDU Generic SIP Model Generic SIP Model Specialization SIP templates implementation

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 37 Formal Phase Specification SIP implementation  List of the specific SIP elements in an XFDU: Global information: sip_ID; producer_ID; project_ID; sip_template_ID; sip_template_location. Information associated with each embedded object: descriptor_ID; descriptor_location; last_object; updated_object.

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 38 Contents  Introduction Objectives Main steps General framework Example  General principles for implementing the Producer-Archive Interface Process description Plan of Objects to be Transferred (POT) SIPs Sequencing constraints  Formal Phase Specification  Transfer Phase and Validation Phase Specification  Managing modifications  Tools  Conclusion

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 39 Transfer/Validation Phase Specification Functions  SIP instances  Transmission sessions  SIP reception and validation  Transfer Management  Anomaly management

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 40 Transfer/Validation Phase Specification Global view project_ID SIPs Transmission session Anomaly management SIP reception SIP validation Transfer management Producer session_ID producer_ID project_ID transfer_date sip_list transfer_repository session_ID sip_list transfer_date descriptor_ID status occurrence_number number_validated_objects sip_template_ID time_constraint_group serial_number_in_constraint descriptor_ID (1..n) content_ID (0..n) data_ID (0..n) metadata_ID (0..n) session_ID anomaly_ID error_level date … Archive POT log message Ingest Base

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 41 Contents  Introduction Objectives Main steps General framework Example  General principles for implementing the Producer-Archive Interface Process description Plan of Objects to be Transferred (POT) SIPs Sequencing constraints  Formal Phase Specification  Transfer Phase and Validation Phase Specification  Managing modifications  Tools  Conclusion

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 42 Managing modifications  POT modification –Impact on the data to be delivered –Technical impact on the current functions and tools –Possible re-negociation of the Submission Agreement  Object modification –« Updated » element –The new object replaces the older one

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 43 Tools  Formal Definition Phase –Tool for creating specialised dictionaries from the generic Model. –Tool for generating Descriptor instances. –Tool for creating and validating the POT.  Transfer and Validation Phases –Tool for generating SIP instances (prototype to create XFDU instances ?). –Tool for the transfer validation and management: Validate the content of each received SIP according to the initial validation plan (object expected and conforms to the associated Descriptor). Respect of the sequencing constraints. Monitoring project progress for the Producer and for the Archive. Propose a follow-up graphical view by using, if possible, the POT graphical view.

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 44  How can we define an object (and have a sufficiently precise description without creating the objects)? –Descriptor and attributes  How can we define the organisation between the objects (« Plan of Objects to be Transferred « )? –Descriptors and relation definitions between objects  How can we define a complete packaging description for an object and a group of objects? –SIPs  How can we adapt the general framework to the domain specificities and project needs? –Specialisation process  How can we specify the constraints (granularity and sequencing) applying to these objects? –TIME_CONSTRAINT_GROUP, SERIAL_NUMBER_IN_CONSTRAINT attributes  How can we validate the transferred objects? –Validation process  How can we have a unique visual representation of the objects to deliver and their constraints? –See the mock-up Conclusion Needs to meet: some answers

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 45 Conclusion  Reach an Agreement on –Terminology, –Concepts (Descriptors, SIPs, sequencing constraints), –Process, –XFDU attributes, –Implementation (Descriptor schema, SIP schema, …)

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 46