Quick Guide to CAM Blueprints Intelligence Commanders Training Quick Guide to CAM Blueprints Using blueprints to develop XML exchange templates and schema Author: David RR Webber Chair OASIS CAM TC January, 2010 http://www.oasis-open.org/committees/cam (CAM – Content Assembly Mechanism Specification) Foundations of Intelligence 1

Introduction Configuring formal information exchanges using XML transactions today requires a set of exchange schema that describe those interactions An exchange schema provides the structural layout for the XML transaction along with rules detailing the content expected Exchange schema development requires knowledge of XML best practices, industry specifications, data modelling techniques and guidelines for exchanging XML transactions How does the CAM blueprint approach enable a shift away from a purely manual process today to one that can be highly automated, consistent and repeatable? Introducing concepts and overall needs

Solution Approach Top Down development Allows developers to quickly outline the information exchange Provides XML dictionaries of canonical reusable components CAM tool then expands the details and fill-in the blanks Review in visual editor to confirm or extend results Iterative approach until complete Automated exchange package generation Schemas, XML, documentation, mapping crosswalk Test generation of example XML with rules validation Principles and Rules evaluator Provides best practices for interoperability Checks for specific schema techniques and details Dictionary driven reuse Provides industry components / Enterprise Data Model Ensures consistency of definition and use Roll your own dictionaries Quick sketch of main aspects of blueprinting approach

Agenda Blueprints and Dictionaries Generating Exchange Artifacts Exchange Assembly Process Steps Overview Leveraging re-use – canonical XML dictionaries Introducing Exchange Assembly Blueprints Modelling information structures in XML Using namespaces and prefixes Constructing your Blueprint templates Running the Expander Finishing your Exchange Template Generating Exchange Artifacts NDR evaluation, Exchange schema, mapping crosswalk, XML instances, realistic data use, business rules documentation Summary Use this slide to format text lists with bullets

Exchange Assembly Process Overview 1 Business Modelling Requirements Charter Work Group Project Inception CAM Enabled Dictionary Generation / Selection Model Exchange Blueprints Dictionary 2 Web based Desktop Excel Search tools Import XSD Naming / NDR / spell check Generate Dictionary Import EDM Exchange Content Modelling 3 Expand and Evaluate Results Expand, Review, Refine Completed Exchange Structures NDR Evalu- ation 4 dictionaries Review / Tailor Exchange Structures Each swim lane represents sets of tasks and actions in the process of developing an information exchange. Complete package of artifacts required are generated by CAM tools, also testing and verification reports. This then is ready to publish so that implementers can align their application systems to the specifications and business requirements. In NIEM parlance this is the IEPD – Information Exchange Package Definitions. Notice also the EDM – Enterprise Data Model components can be presented as dictionaries. Completed Exchange artifacts XML examples XSD schemas Subset Schema Crosswalk Mapping Spreadsheet Rules Report Rendering Stylesheet Test Examples and Rules 5 Test Cases Validation Engine Schema Generation Generate Package Artifacts and Test them XSD Schema Writer Cross Reference Mapping 6 Exchange Package

Leveraging Re-use : Canonical XML Dictionaries Intelligence Commanders Training Leveraging Re-use : Canonical XML Dictionaries What is a canonical dictionary Styles of dictionary How do I obtain a dictionary? How do I use it with CAM? Foundations of Intelligence 6

Canonical XML dictionary A collection of distinct components that represent discreet business information for an application domain Includes singleton components and combinations of related components together as sub-assemblies Information is represented in a simple neutral conceptual data format that captures the critical concepts about the data e.g. name, description, content type, contextual usage pattern, hierarchy Wikipedia definition: http://en.wikipedia.org/wiki/Canonical#Computer_science

XSD Schema v CAM Dictionary There are generally two “styles” of XSD schema Distinct information exchange schema structures Collections of schema model components, possibly abstract, and possibly recursive CAM dictionaries built from distinct exchange structures generally mimic those components directly Parent / Child sets are visible in dictionary as ABIE/BBIE pairings “children” column in dictionary contains direct descendents “context” column reflects component relationships CAM dictionaries built from models with abstract and recursive parts may have fragments of components only Assembly of exchange structures using references to piece parts, rather than complete structure assemblies Reuse is then around these extension part collections Context may need to be overridden to meet desired use pattern Examples of two styles of schema – exchange schema: OASIS standard specifications; model schema: NIEM model schema. Notice that dictionary components are to facilitate assembly – so may have duplicates, or merged components sets (e.g. ABIE may appear more than once, or BBIEs only after first ABIE occurrence to use them. This reflects how the dictionary was built – e.g. single step from one CAM template, or multiple merging steps from many CAM templates.

Available XML Dictionaries LEXS 3.1.4 dictionary OASIS EDXL dictionary OASIS EML dictionary NIEM 2.1 dictionaries CBRN dictionary Emergency dictionary Family dictionary Immigration dictionary Infrastructure dictionary Intelligence dictionary Justice dictionary Maritime dictionary Screening dictionary Trade dictionary NIEM core dictionary Immigration blueprint Packaged with CAM editor see dictionary folder of install + spreadsheet + blueprint samples XML XML XML Note: Those marked in bold are model style dictionaries with recursive components. Available from download site direct link: http://sourceforge.net/projects/camprocessor/files + includes spreadsheets and sample blueprint Sampling of available dictionaries. Those marked in bold are model style dictionaries with recursive components. XML XML XML XML XML XML

Drag and Drop dictionary into Spreadsheet Item names Parent item indicator List of Child items Drag and Drop XML dictionary format into Microsoft Excel Child item indicator This shows the components and concepts that underpin the dictionary, viewed as a spreadsheet. This provides a familiar interface for business data analysts to work with. Simply take the XML dictionary file and drag and drop it into Microsoft Excel, click “OK”, and it will open as a workbook. Item Properties Unique ID

Creating your own Dictionaries The CAM toolkit includes dictionary generation tools can harvest components from existing exchange schema will import model components from ERwin modelling tool can ingest and use subset schema or component schema A separate companion Quick Guide describes the dictionary creation processes and tools Look for existing dictionary packs from Sourceforge files location of CAM download http://www.sourceforge.net/projects/camprocessor/files

Using Dictionaries Determine your business information exchange components at conceptual level Search and locate candidate components from appropriate domain dictionary collections Catalogue the parts to be used Dictionary components can be referenced individually or as collections by an assembly blueprint that puts them all together to create a complete information exchange Components can be selected from multiple dictionaries Note any new extension pieces as needed A control file configures which physical dictionary files to reference and associates one or more particular namespace prefixes with each dictionary The download packs available from Sourceforge contain everything you need to get started; dictionaries, control files, sample blueprints and instructions.

Dictionary Control File Associates physical dictionary files to namespace prefix details This is only required if you are setting up your own new dictionaries from scratch. You can of course review control files to see what dictionaries are referenced. Control files should be manually edited in a XML editor tool to make configuration changes as needed (e.g. arrowed text areas).

Introducing Exchange Assembly Blueprints Intelligence Commanders Training Introducing Exchange Assembly Blueprints What is a blueprint template Example blueprints Blueprint tools overview How do I create a blueprint? How do I use it with CAM? Foundations of Intelligence 14

Introducing Blueprints A blueprint is the outline of your information exchange and the top level sets of components needed Allows rapid iterative exchange structure design Start with a basic idea of what is needed, extend and refine as you clarify groups of components and their relationships Can import components from one or more domain dictionary collections Allows you to focus on the conceptual and hierarchical information model components and their purpose The low level specific information details will be filled in for you by the expander tool from the matching dictionary definitions Review results and add local extensions / exclusions as needed Blueprints themselves also have high re-use value Can be sub-assemblies and patterns not just exchange models

Example Blueprints OASIS EDXL HAVE message LEXS messaging blueprint Business functional components LEXS messaging blueprint Reusable messaging envelope constructs Individual component Message handling, delivery and control Payload goes here Top level sets of business information components these examples available from CAM editor install package ~ CAMeditor\eclipse\workspace\CAMEditor\dictionary\blueprints\

Blueprint Tools Overview Excel Domain dictionary Web tool 1 Blueprint Designer Component Definitions Industry dictionary 2 Component Definitions Search Tools Expander Tool Insert Dictionary Parent Components 3 4 5 Completed Exchange Template

How do I create a blueprint? Use CAM Eclipse visual editor create basic blueprint from scratch or from XML instance Clone and edit an existing blueprint use a XML editor to manipulate raw XML of blueprint (e.g. search and replace namespace details) modify blueprint using CAM Eclipse visual editor Beware of namespaces! You need to pre-declare namespaces in the blueprint – that may appear later in the finished expanded exchange Use CAM Eclipse visual editor root node menu to add namespace definitions Tip: use your XML editor to cut and paste namespaces into raw blueprint XML file directly A blueprint is nothing more than a regular CAM template stripped of everything except the structure section and a top level outline.

Blueprint Structure Overview Short Name for Exchange Root node name and optional namespace XML container elements to facilitate use patterns Dictionary component references A blueprint is nothing more than a regular CAM template stripped of everything except the structure section and a top level outline. Dictionary Type annotation reference (note red paper clip icon) Namespace declarations

Adding Components from a Dictionary Understanding the 4 basic types of component referencing available Direct component (BBIE) Direct collection (ABIE) Indirect type (extension) Indirect sub-component (contextual re-use) Control of the XML structure hierarchy Grouping collections of components Extending an existing component Building new extension from existing components Namespaces – when to use them

Example XML Techniques (Immigration Alien Alert example blueprint) Direct collections referencing (ABIE) Grouping collection of components Direct component referencing (BBIE) Building new component from existing ones Indirect type extension via annotation Type (red paper clip icon denotes annotation) Namespaces – root namespace optional (aaa) Dictionary component namespaces are required

Making New Blueprint from scratch Assign root element Open new template 2 1 Complete structure by adding elements and dictionary components 3

Running Blueprint Expander Intelligence Commanders Training Running Blueprint Expander Blueprint check list Using Expander tool dialogue Reviewing results Hints and Tips Foundations of Intelligence 23

Using Your Blueprints What you need - Getting Started CAM Eclipse visual editor Blueprint template Dictionary or dictionaries of components Dictionary control file Contains linkage of namespace prefix to each dictionary Contains physical filename path of each dictionary Getting Started Install CAM editor tool from Sourceforge.net Open example blueprint from EDXL folder ~ CAMeditor\eclipse\workspace\CAMEditor\dictionary\blueprints\EDXL Try blueprint expander with control file EDXL-dictionary-control-file.xml

Running Blueprint Expander This example is available in the NIEM dictionaries download pack from Sourceforge Select Dictionary Control File Pick output result filename (default is -expanded.cam) Determine rule generation selections from dictionary definitions Note: entries are “sticky” so are remembered for next time

Blueprint Expander Example COMPLETED EXCHANGE TEMPLATE 2 Expander Tool Structure Details Expanded 1 Dictionary Lookups Exchange components outline 3 EXCHANGE BLUEPRINT Blueprint follows model Console Log detail

Reviewing Results Exchange Structure items 2 1 Rules for each item Domain Components Namespaces 3

Hints and Tips Blueprint Remedy Node not expanded, or removed! Override default dictionary structure context definition using an annotation Type reference to the node instead – see Alien alert example blueprint – aaa:OfficerDetails Added parent and child combination. All children added instead of just one. Expander inserts all children of parent. Either delete unwanted children from expanded result (see next section), or create custom parent node, and pick children components directly – see Alien alert blueprint example. Node not expanded Make sure namespace prefix and spelling exactly matches dictionary and dictionary control file prefixes. Undefined namespace prefix error when opening expanded template in CAM editor Expander added dictionary components from another namespace. Declare that namespace prefix back in blueprint and re-run expander again Expanded result missing all desired sub-components Create your own blueprint extension parent and combine parts to create desired result – see Alien alert example – aaa:FurtherDetails Expanded result has too many sub-components In expanded result mark unneeded parts with “excludeTree” rules, then run File / Export / Compress to strip out; or use extension technique above to pick only few child elements instead of whole parent Some of these techniques are blueprint ones, others involve the expanded result template. Next section covers editing in more details.

Completing the Exchange Template Intelligence Commanders Training Completing the Exchange Template Adding nodes Deleting nodes Editing Rules Adding annotations Foundations of Intelligence 29

Exchange Template Editor 2 1 Rules for each item Can add new custom domain Items 3 May need to add a namespace for new items 4

Deleting Exchange Components Select item to be removed 1 Right click for popup menu 2 From Rules exclude item or tree 3 4 After marking all items to be excluded, save template, then run Files / Export / Compress template to physically remove them.

Editing Rules 1 Select item to add new rule Right click for popup menu 2 Use rule wizard to complete 3

Adding annotations 1 Select item to add new annotation Right click for popup menu 2 Use annotation wizard to complete 3

Generating Exchange Artifacts Intelligence Commanders Training Generating Exchange Artifacts Dictionary crosswalk mapping Exchange Schema XML instances Business rules documentation Foundations of Intelligence 34

Exchange Generation Steps EXCHANGE TEMPLATE Structure Rules Documentation Toolkit for exchange artifacts generation Run template Evaluation Report Compare to industry dictionary - create mapping spreadsheet - create crosswalk xml Generate business rules report Generate exchange XSD schema Generate XML test instance(s) 1 2 3 Suite of menu options and tools in desktop CAM toolkit editor (each run custom xslt scripts on the exchange CAM template xml) 4 5 CAM Template

Using NDR Evaluation Report Provides scoring, alerts, warnings and potential issues including: Naming and content model conventions Naming and restriction consistency checks Interoperability enablers/inhibitors checks Rules integrity and duplicates Statistics on exchange size Spell checking on component names Access from Tools menu, Evaluate Template option

Evaluation Report (NDR) example Part of an example report for LEXS getDataItemRequest template displayed in toolkit HTML viewer

Compare to industry dictionary References industry dictionary of names and properties Matches on physical names Reports mapping details Compatible with Microsoft Excel Report can be used to do spell checking Generates crosswalk xml file When NIEM-dictionary selected: generates NIEM SSGT wantlist.xml file provides % re-use statistics for NIEM components ( note NIEM-dictionary.xml is located in install directory ~ CAMeditor\eclipse\workspace\CAMEditor\dictionary folder )

Example cross-reference spreadsheet Formatted view in Microsoft Excel of import of cross-reference report details (from generated XML file)

Generate Documentation and Schema Create HTML report of exchange schema details and associated content and business rules Report layout and content designed to be reviewed by business analysts Schema: Generate XSD schema for exchange Customizable exchange folder layout management by namespace for extension, subset and exchange schema components Writes XSD schema in syntax that is clear, simple and compatible with deployment tooling environments

Business Rules Documentation Part of the example rules for LEXS getDataItemRequest template displayed in toolkit HTML viewer

Export Exchange to XSD Schema Export Template to Exchange XSD: Completed Exchange Structure Set Exchange Output Option Complete set of exchange schemas generated

Exchange Schema Generated Each namespace file is import for those specific type definitions Set of XSD files with filename and namespace suffix Reviewing XSD results in a schema editor tool

XML Testing Examples Generation Support for software development testing process Designed to allow creation of concrete realistic examples not just random value based Hinting system allows insertion of actual test system values into XML examples Can create both valid and invalid examples to support unit testing of application software Exclude capability allows generator to create examples that contain only a portion of the entire exchange Control over random seed value used allows re- generation of identical test cases

XML example generation wizard View of CAM toolkit with LEXS getDataItemRequest and dialogue for XML test example generator tool

Running validation rules tests Built-in CAM validation engine allows testing of XML instances against actual exchange rules (CAMV). Critical to ensure that the exchange validates actual live production example scenarios correctly Allows deployed solution to match exchange schema details Errors can be reviewed interactively in exchange visual interface Post-processing of validation results allows unit regression tests to be created with reporting of errors, warnings and information level notes

Run Exchange Template Pick XML test case to validate Run validation Review results in visual editor

Example Exchange Packaging Details Package Artefact Description CAM Generates Files NDR Evaluation Report Evaluation of exchange template compared to set of Naming and Design Rules (NDR) and interoperability factors / principles Crosswalk XML Itemized list of each dictionary component element and attribute included in the exchange. Exchange Schema Base document schema that defines the full XML structure for the exchange and is generally named after the exchange itself. Extension Schema Specification for extended components—separate local name-spaces of components not contained in dictionary Sample XML Instance Example instance(s) Documentation Master Documentation The Master Document is the main document for which all of the context and details around the exchange are explained. This document includes, the overview, as well as details surrounding the exchange, business drivers and requirements Business rules Business rules in plain or structured English, generated by CAM development tool. Mapping to Dictionary Mapping of domain components, tagged with constraints (i.e., cardinality, etc.) to dictionary components as a spreadsheet. Extended components Components created because they were not in dictionary—may be part of mapping spreadsheet and include structure and definitions of new components.

Summary Dictionary driven exchanges Blueprint enabled reuse Intelligence Commanders Training Summary Dictionary driven exchanges Blueprint enabled reuse Automated exchange package generation Alignment to NDR Principles and Rules Testing and validation support Foundations of Intelligence 49

Review Top Down development NDR Principles and Rules Reference dictionary components Create exchange blueprint Run Expander tool Refine desired structure in visual editor NDR Principles and Rules Best practices for interoperability and schema techniques Dictionary driven reuse Select Enterprise Data Model and industry components Ensures consistency of definition and use Automated exchange package generation Schemas, XML, documentation, mapping crosswalk Test generated example XML with rules validation

Links and Resources CAM Toolkit download http://sourceforge.net/projects/camprocessor NIEM dictionaries and sample blueprint http://sourceforge.net/projects/camprocessor/files NIEM tools and IEPD tools tips and hints http://wiki.oasis-open.org/cam/NIEM_Tools_and_CAM