SEA-1 20 Nov 2014 CCSDS System Engineering Area (SEA): XSG – XML Standards & Guidelines Peter Shames, SEA AD 20 Nov 2014

SEA-2 20 Nov 2014 Motivation for this XML Guidelines Project CCSDS currently has several different XML based data formats: DEDSL DTD NAV NDM XML SM Service Package SM&C XTCE & others SOIS XTEDS Apologies if I missed any … and others are coming These have been developed during the last 8-12 years of CCSDS standards, and they were largely in disjoint domains As CCSDS has grown there are now interfaces and overlaps both in content and in terminology, but no standards for namespaces, common terms, or schema construction, nor defined means for establishing and maintaining consistency This project proposes to tackle this issue head on by: Adopt or develop an agreed set of guidelines for XML standards Derive initial ontology terms from the existing XML (type) schema Identifying common schema for QUDV and related topics Using the new common core Ontology and domain extensions for all terms Doing the work to regularize and formalize the use of terms Propose, as needed, other new registries in SANA, values, units Propose a process for managing the evolution and versioning of the set of XML schemas in the future Working any issues that arise with the affected WGs

SEA-3 More on XML Issues There is no standard for construction and use of names, common terms, or references There are no commonly defined registries for: quantities, units, time, and value terminology, agencies, ground stations & antennas, frequency bands, and other common identifiers The XML formats may be DTD, or Schema, or other forms, they may be qualified or unqualified We are finally establishing CCSDS namespace standard and do have an XML repository in the SANA, but this is just a beginning 20 Nov 2014

SEA-4 Example: Nav NDM XML Schema Based on existing CCSDS Nav ASCII document formats All element names are upper case Uses XML Schema and qualified namespace Little in-line documentation, no in-line constraint checking Defines a set of NAVWG common terms and a set of “CCSDS” common terms in separate schema ndmxml-R1.5-ccsds-common ndmxml-R1.5-navwg-common Several different schema used to define TDM, OEM, OPM, ADM, AEM, APM document types NOTE: CCSDS common terms could be merged with those defined in SmSchemaCommonTypes 20 Nov 2014

SEA-5 Example: CSS SM Schema Developed for CCSDS Cross Support Service Management Uses XML Schema and qualified namespace References NDM XML schema for their content Little in-line documentation, no in-line constraint checking Defines SmSchemaCommonTypes that is included in other SM schema SmSchemaCommonTypes-v1.0.0 Defines a lot of CCSDS common types such as CCSDS 401, Space link, and also SM common terms Several different schema used to define config profiles, service agreement, service package, and trajectory prediction Common type definitions could be split into several sets SM specific common terms SLE / CSTS specific common terms Space link, RF, Mod, Coding common terms NOTE: CCSDS common terms could be merged with those defined in ndmxml-R1.5-ccsds-common NOTE: Identify a set of agency / service providers that could be a separate common registry 20 Nov 2014

SEA-6 Example: DEDSL DTD Developed for CCSDS Data Entity Dictionary exchange Uses older XML DTD format, not a Schema References DTD abstract spec for its content Little in-line documentation, no in-line constraint checking Defines some common types that are shared with other XML standards: integer, real, text, identifier, One DTD used to define dictionary name, version, language, and related attribute specifications NOTE: Not clear that any of these DTD specifications are in use, need to verify Could probably be replaced by ISO/IEC : Nov 2014

SEA-7 XML Schema Guideline Sources Google XML Document Format Style Guide, 2008 Topics: design or re-use, schema style (and what to do with DTDs), namespaces, names & enumerated values, elements, attributes, values, key-value pairs, binary data, processing instructions, representation of XML document instances, elements vs. attributes Defines requirements and provides rationales ESA Design & Style Guide for XML Data and Schema, 2005 Topics: Project guidelines on use of XML, overview of XML, XML tools and development environments (IDE), naming, design and style, security & encoding Design & style topic covers: design attributes, type, namespace, extensibility, constraints, versioning, global vs local Defines rules and recommendations with rationales XML Schema Best Practices, From XML-DEV, compiled by U of Cambridge, 2001 Topics: namespace, element vs type, extending schema, extensible content, global vs local, hide vs expose, variable content containers Analyzes different approaches, provides recommendations 20 Nov 2014

SEA-8 XML Schema Guideline – Common Ground (Order & text from ESA) Rule 1: Use meaningful, familiar and consistent names. Rule 2: Avoid making abbreviations for names, removing vowels. Rule 8: Make two identical copies of all your schemas, where the copies differ only in the value of elementFormDefault (in one copy set elementFormDefault="qualified", in the other copy set elementFormDefault="unqualified"). Rule 9: Uniquely identify all schema components with the id attribute. Rule 10: If the item is not intended to be an element in instance documents, then define it as a type. Rule 11: If the item ́s content is to be reused by other items then define it as a type. Rule 12: If the item is intended to be used as an element in instance documents, and it is required that sometimes it be nillable and other times not, then it must be defined as a type. Rule 13: If the item is intended to be used as an element in instance documents and other elements are to be allowed to substitute for the element, then it must be declared as an ele- ment. 20 Nov 2014

SEA-9 XML Schema Guideline – Common Ground, contd (Order & text from ESA) Rule 17: If you have a targetNamespace, make it the default namespace. Rule 18: Do not hard code the identity of an imported schema. Rule 19: Capture the schema version somewhere in the XML schema. Rule 20: Include information in the instance data files, that makes it possible to determine which version (or versions) of the schema they are compatible with. Rule 21: Make older versions of your XML schema available. Rule 22: In situations where a new version of a schema makes backwards incompatible changes (e.g., a construct that was valid and meaningful for the previous schema does not validate against the new schema), make sure that older instances will not be accidentally validated against the new version. 20 Nov 2014

SEA-10 XML Schema Guideline – Common Ground (Order & text from Google) Attempt to reuse existing XML formats whenever possible, especially those which allow extensions. If you are reusing or extending an existing format, make sensible use of the prescribed elements and attributes, especially any that are required. Document formats SHOULD be expressed using a schema language. [Rationale: Clarity and machine-checkability.] Schemas SHOULD use the "Salami Slice" style (one rule per element). Schemas MAY use the "Russian Doll" style (schema resembles document) if they are short and simple. Element names MUST be in a namespace, except when extending pre-existing document types that do not use namespaces. A default namespace SHOULD be used. [Rationale: Namespace-free documents are obsolete] Attribute names SHOULD NOT be in a namespace unless they are drawn from a foreign document type or are meant to be used in foreign document types. Namespace names are HTTP URIs. Namespace names SHOULD take the form where whatever is a unique value based on the name of the document type, and year is the year the namespace was created. 20 Nov 2014

SEA-11 XML Schema Guideline – Common Ground, cont (Order & text from Google) Namespaces MUST NOT be changed unless the semantics of particular elements or attributes has changed in drastically incompatible ways. Namespace prefixes SHOULD be short (but not so short that they are likely to be conflict with another project). Single-letter prefixes MUST NOT be used. Prefixes SHOULD contain only lower-case ASCII letters. All names MUST use lowerCamelCase. That is, they start with an initial lower-case letter, then each new word within the name starts with an initial capital letter. Names MUST contain only ASCII letters and digits. Published standard abbreviations, if sufficiently well-known, MAY be employed in constructing names. Ad hoc abbreviations MUST NOT be used. All elements MUST contain either nothing, character content, or child elements. Mixed content MUST NOT be used. Document formats MUST NOT depend on the order of attributes in a start-tag. Elements SHOULD NOT be overloaded with too many attributes (no more than 10 as a rule of thumb). Instead, use child elements to encapsulate closely related attributes. 20 Nov 2014

SEA-12 Numeric values SHOULD be 32-bit signed integers, 64-bit signed integers, or 64-bit IEEE doubles, all expressed in base 10. These correspond to the XML Schema types xsd:int, xsd:long, and xsd:double respectively. If required in particular cases, xsd:integer (unlimited-precision integer) values MAY also be used. Boolean values SHOULD NOT be used (use enumerations instead). If they must be used, they MUST be expressed as true or false, corresponding to a subset of the XML Schema type xsd:boolean. Embedded syntax in character content and attribute values SHOULD NOT be used. Syntax in values means XML tools are largely useless. Syntaxes such as dates, URIs, and XPath expressions are exceptions. Simple key-value pairs SHOULD be represented with an empty element whose name represents the key, with the value attribute containing the value. Elements that have a value attribute MAY also have a unit attribute to specify the unit of a measured value. For physical measurements, the SI system SHOULD be used. 20 Nov 2014 XML Schema Guideline – Common Ground, cont (Order & text from Google)

SEA-13 XML Schema Guideline – Common Ground, cont (Order & text from Google) Binary data MUST NOT be included directly as-is in XML documents, but MUST be encoded using Base64 encoding. An attribute named xsi:type with value xs:base64Binary MAY be attached to this element to signal that the Base64 format is in use. [Rationale: Opaque blobs should have decoding instructions attached.] Use standard XML parsers, not hand-rolled hacks. The character encoding used SHOULD be UTF-8. Exceptions should require extremely compelling circumstances. Namespaces SHOULD be declared in the root element of a document wherever possible. The mapping of namespace URIs to prefixes SHOULD remain constant throughout the document, and SHOULD also be used in documentation of the design. Well-known prefixes such as html: (for XHTML), dc: (for Dublin Core metadata), and xs: (for XML Schema) should be used for standard namespaces. 20 Nov 2014

SEA-14 XML Schema Guideline – Common Ground, cont (Order & text from Google) Attributes are more restrictive than elements, and all designs have some elements, so an all-element design is simplest -- which is not the same as best. If something might appear more than once in a data model, use an element rather than introducing attributes with names like foo1, foo2, foo3.... Use elements to represent a piece of information that can be considered an independent object and when the information is related via a parent/child relationship to another piece of information. Use elements when data incorporates strict typing or relationship rules. If order matters between two pieces of data, use elements for them: attributes are inherently unordered. Use common sense and BE CONSISTENT. Design for extensibility. You are gonna need it. [Rationale: Long and painful experience.] Break ANY OR ALL of these rules (yes, even the ones that say MUST) rather than create a crude, arbitrary, disgusting mess of a design if that's what following them slavishly would give you. 20 Nov 2014

SEA-15 XML Schema Guideline – Discussion Topics (Order & text from ESA) Rule 23: Where minimizing size and coupling of components is of utmost concern then use the so-called Russian Doll design. Rule 24: Where your task requires that you make available to instance document authors the option to use element substitution, then use the so-called Salami Slice design. Rule 25: The so-called Venetian Blind design is the one to choose when your schemas require the flexibility to turn namespace exposure on or off with a simple switch, and where component reuse is important. 20 Nov 2014

SEA-16 XML Schema Guideline – Discussion topics (Order & text from Google) The schema language SHOULD be RELAX NG compact syntax. Embedded Schematron rules MAY be added to the schema for additional fine control. [Rationale: RELAX NG is the most flexible schema language, with very few arbitrary restrictions on designs.] Dates should be represented using RFC 3339 format, a subset of both ISO 8601 format and XML Schema xsd:dateTime format. UTC times SHOULD be used rather than local times. 20 Nov 2014

SEA-17 Simple RelaxNG Example (From IBM.com Developers: XML Matters) Listing 3: A context-free compact syntax schema # A library patron example default namespace = " namespace foo = " datatypes xsd = " ## Annotation here start = patron patron = name & id-num & book name = element name { xsd:string { pattern = "\w{,10}" } } id-num = element id-num { xsd:string } book = element book { ( attribute isbn { text } | attribute title { text } | attribute anonymous { empty }) }* 20 Nov 2014

SEA-18 XML Schema version of Simple RelaxNG Example (From IBM.com Developers: XML Matters) <grammar xmlns= ns=" datatypeLibrary=" xmlns:a=" xmlns:foo=" Annotation here \w{,10} 20 Nov 2014

SEA-19 XML Schema version of Simple RelaxNG Example contd (From IBM.com Developers,XML Matters) 20 Nov 2014

SEA-20 Relax NG and Schematron RelaxNG may embed Schematron rules Schematron is a: Way to test XML documents Rules-based validation language Way to specify and test statements about your XML document: elements, attributes, content Schematron can: do validation, run reports, find patterns, check elements against a vocabulary A Schematron “program” is a well-formed XML document, run by a processor Schematron specifies, it does not perform A Schematron “schema” specifies tests to be made on your XML A set of declarations for a process (“test this; tell me that”) 20 Nov 2014

SEA-21 RelaxNG and “normal” Schema RelaxNG has two forms, “normal” and “compact” Compact form is very straightforward, easy to read, and avoids all of the “ ” clutter Most tools handle RelaxNG and there are a number of translators from RelaxNG to Schema to DTD and back again (as long as compatible features are used) But … RelaxNG is not in use in CCSDS today and there is little familiarity with it 20 Nov 2014

SEA-22 Observations Existing CCSDS XML standards were all developed independently and with different formats and styles. Different levels of care have been taken to define common terms in any given set of documents. There are instances where the same terms are defined in more than one spec and others where the same concept is defined differently. There are more and more instances where XML formats defined in one WG are used in another WG, and this is likely to increase. Trying to do this work in a SIG, on a shoestring, just isn’t working, it needs to be part of a WG Propose making it be a part of the SEA SAWG re-start, along with RASDS and ontologies 20 Nov 2014

SEA-23 XML Notes Consider inserting a guideline for users to perform adequate validation of the schema between the source and the sink Content Structure Potentially validity and authentication Adhere to Postel’s rule: Be liberal in what you accept, be strict in what you send Provide other transformations than just XML, such as HTML, JSON, other styles … Consideration of XSD, RelaxNG, ABNF, ASN.1, or other alternate abstract forms Not about tooling, about compliance, transformability and validation Check with Francisco Martinez, John Pietras, qual vs unqual issues 20 Nov 2014

SEA-24 Proposed Phased Approach Identify all current and planned XML related standards. Pick a key subset of the XML standards and develop schema structure and content validators, potentially build using COTS tools and Schematron. Vet the validation approach with the WGs and do V&V of the capability. Make the result available to the community on the SANA web site (TBC). Establish a process for developing similar validators for all CCSDS XML standards. Analyze current XML standards to identify places where there are common terms defined, such as NDM XML and CSSM. Develop an evaluation suite of common terms, suitable for referencing and a parallel set of modified schema. Vet the common terminology approach with the WGs and do V&V of the capability. Make the result available to the community on the SANA web site (TBC). Establish a process for developing similar adaptations for all CCSDS XML standards. Develop a process for integrating the terms defined in the CCSDS ontology into the updated schema definitions. 20 Nov 2014

SEA-25 Backup The following slides are from: SysML 1.4: Quantities, Units, Dimensions & Values (QUDV) & ISO Library Nicolas Rouquette Jet Propulsion Laboratory California Institute of Technology March 2014 Copyright 2013, 2014, California Institute of Technology (“Caltech”) U.S. Government sponsorship acknowledged. All rights reserved. 20 Nov 2014

SEA-26 Using VIM 3 rd edition in SysML 1.4 & QUDV Normative SysMLNon-normative QUDV & ISO Quantities and units (30 definitions) language concept modeling pattern language concept modeling pattern ISO Library 1.1 quantity ✔ 1.2 kind of quantity ✔ 1.3 system of quantities ✔ 1.4 base quantity ✔ 1.5 derived quantity ✔ 1.6 International System of Quantities (ISQ) ✔ 1.7 quantity dimension ✔ 1.8 quantity of dimension one ✔ 1.9 measurement unit ✔ 1.10 base unit ✔ 1.11 derived unit ✔ 1.12 coherent derived unit ✔ 1.13 system of units ✔ 1.14 coherent system of units ✔ 1.15 off-system measurement unit ✔ 1.16 International System of Units (SI) ✔ 1.17 multiple of a unit ✔ 1.18 submultiple of a unit ✔ 1.19 quantity value ✔ 1.20 numerical quantity value ✔ 1.21 quantity calculus ✔ 1.22 quantity equation ✔ 1.23 unit equation ✔ 1.24 conversion factor between units ✔ 1.25 numerical value equation ✔ 1.26 ordinal quantity ✔ 1.27 quantity-value scale 1.28 ordinal quantity-value scale ✔ ✔ 1.29 conventional reference scale ✔ ✔ 1.30 nominal property ✔ No Coverage in QUDV or SysML 2Measurement (53 definitions) 3Devices for measurement (12 definitions) 4Properties of measuring devices (31 definitions) 5Measurement standards (etalons) (18 definitions) 20 Nov 2014

SEA-27 Systems of Units & Quantity Kinds Normative SysMLNon-normative QUDV & ISO Def.Quantities and units (30 definitions) language concept modeling pattern language concept modeling pattern ISO Library 1.3 system of quantities ✔ 1.6 International System of Quantities (ISQ) ✔ 1.13 system of units ✔ 1.16 International System of Units (SI) ✔ 20 Nov 2014

SEA-28 Practical Considerations for Modeling Values in SysML #1: Recognize the distinct levels of modeling involved A Level depends on those above i Vi depends on Vj V4 < V5 See VIM3 & ISO foreword : This first edition of ISO cancels and replaces ISO 31-0:1992 and ISO 1000:1992. The major technical changes from the previous standard are the following: the structure has been changed to emphasize that quantities come first and units then follow; V3 < V4, V5 Per SysML ValueType V2 < V3 Per UML TypedElement/Type relationship V1 < V2 Per UML ValueSpecification/Slot/ StructuralFeature & defaultValue relationships QuantityKinds Units ValueTypes (QuantityKind, Unit) Value Properties (typed by ValueTypes) Value Specifications (typed by ValueTypes) V1 V2 V3 V4 V5 Runtime Values V0 20 Nov 2014

SEA-29 Practical Considerations for Modeling Values in SysML #2: Recognize the alignment with UML QuantityKinds Units ValueTypes (QuantityKind, Unit) Value Properties (typed by ValueTypes) Value Specifications (typed by ValueTypes) V1 V2 V3 V4 V5 Runtime Values V0 UML InstanceSpecifications (classifier = SysML or QUDV Unit or QuantityKind) UML DataTypes (stereotype = SysML::ValueType & stereotype tags: quantityKind, unit) UML Properties (type = > UML DataType) UML ValueSpecifications (type = > UML DataType) Varies by methodology 20 Nov 2014

SEA-30 Practical Considerations for Modeling Values in SysML #3: Recognize the implications of DataType modeling in UML ValueTypes (QuantityKind, Unit) Value Properties (typed by ValueTypes) Value Specifications (typed by ValueTypes) V1 V2 V3 Runtime Values V0 UML DataTypes UML ValueSpecifications To Be Decided -Scalar? -Structured? -Literals? -Expressions? -Intervals? -Instance Specifications? Choices made about datatype modeling… … have implications for modeling values… … and type checking ! The conformance of a ValueSpecification to a ValueType depends on these choices! 20 Nov 2014