Delivering NIEM-aligned data services David Webber, Information Architect, Oracle Public Sector NIEM and XML for Architects and Developers

The following is intended to outline Oracle general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. Disclaimer Notice Slide 2

© 2012/13 Oracle Corporation – Restricted Distribution3 Agenda Background - What are the business drivers and needs? NIEM today – Technical Introduction The Delivery Life Cycle – Lessons Learned Software Delivery - Anatomy of a NIEM “IEPD” Tooling to the Rescue! Key Pillar of Dictionaries Summary

BACKGROUND Business Drivers and Needs

© 2012/13 Oracle Corporation – Restricted Distribution5 XML is everywhere… PMIX – Prescription Drug information sharing Elections Results Reporting Pharmacy and Vaccines locating COBie Buildings Information sharing PESC – Student Post-Secondary Education City 311 Services Vehicle spare parts inventory Public Safety information sharing Healthcare services delivery coordination eDevice control templates

© 2012/13 Oracle Corporation – Restricted Distribution6 Public Sector Challenges Cost of building information exchanges – Reduction of development sunk costs through minimizing coding tasks – Allowing rapid prototyping and proof of concepts (agile development) – Flexibility - dynamic adaptable templates instead of rigid fixed code – Reuse – templates for common systems / solutions easily shared and adapted Government transparency and public information sharing – Structured content from data stores – e.g. election results reporting, monthly cost reports, scheduled road maintenance; obvious metadata and semantics – Simple search requests on data – e.g. part numbers, codes, licenses lookup – Not intended for unstructured and textual content document delivery Practical real world data sharing anywhere – Direct simple tools with short learning curve and plug and play deployment

© 2012/13 Oracle Corporation – Restricted Distribution7 Technology Capability Requirements Delivering on the Open Data Digital Strategy Vision for government Open Standards and Open Source based Plug and play with code-free templates and rapid development Supports NIEM and Open XML exchanges Leverages deployed technology today Workflow integration Secure messaging delivery and partner management Delivered across-platform and device End-to-end security & governance

NIEM TODAY Technical Introduction

The NIEM Framework NIEM connects communities of people who share a common need to exchange information in order to advance their missions, and provides a foundation for seamless information exchange between federal, state, local, and tribal agencies. Much more than a data model, NIEM offers an active user community as well as a technical and support framework. Formal Governance Processes Online Repositories Mission-Oriented Domains Self-Managing Domain Stewards Data Model XML Design Rules Development Methodology Predefined Deliverables (IEPD) Tools for Development and Discovery Established Training Program Implementation Support Help Desk & Knowledge Center

The NIEM Data Model NIEM’s data model is a set of common, controlled, and approved XML data structures and definitions vetted through the Federal, State, Local, Tribal and Private Sectors. Data elements are organized into core and domain-specific components Core components are used by multiple domains and can be described by structure, semantics, and definition universally Domain-specific components are continually updated by subject matter experts that are actual NIEM participants and industry experts for their particular domain NIEM Naming and Design Rules (NDR) specify how each of these components are defined and utilized

NIEM’s Domains View NEWEST DOMAINS NIEM Government Resource Management Domain NIEM Health Domain NIEM Human Services Domain DOMAINS IN PLANNING NIEM Agriculture NIEM Education NIEM Transportation 11

© 2012/13 Oracle Corporation – Restricted Distribution12 Available XML Dictionaries NIEM 2.1 dictionaries NIEM 2.1 dictionaries – CBRN dictionary – Emergency dictionary – Family dictionary – Immigration dictionary – Infrastructure dictionary – Intelligence dictionary – Justice dictionary – Maritime dictionary – Screening dictionary – Trade dictionary – Immigration blueprint – NIEM core dictionary Available from download site direct link: XML + includes spreadsheets and sample models Note: Those marked in bold are model style dictionaries with recursive components.

© 2012/13 Oracle Corporation – Restricted Distribution13 NIEM uses W3C Schema Collection XSD Collection *Code list xsd Namespace * Optional XML Schema xsd subset *Constraint xsd *Extension xsd *Restriction xsd Industry Domain schemas Code list xsd LEGEND: Exchange xsd NIEM Exchange Schema

© 2012/13 Oracle Corporation – Restricted Distribution14 NIEM Past and Present Challenges Vertical domain vocabulary for DHS and DOJ One dozen domain dictionaries Technology limited - built using XSD schema Core components highly contextual to DHS/DOJ Surprisingly – no actual logical models of information! Gap with semantic technologies integration Limited data content rules and code lists Mapping automation support missing Multi-year development life cycles

© 2012/13 Oracle Corporation – Restricted Distribution15 NIEM is still improving… Collections of complex XML Schema Verbose components Embedded context in names Currently facing significant scaling challenges – Inconsistencies; too much manual management; slow lifecycles Dictionary technology incubating Enhanced code lists mechanism incubating UML profile is evolving initiative with OMG – Attempt to marry modelling techniques and XSD Schema syntax Policy and Security mechanisms incubating Need to embrace enabling tools for rapid expansion of domains and community collaboration

DELIVERY LIFECYCLE How to build NIEM solutions Lessons learned Foundation Pillars

© 2012/13 Oracle Corporation – Restricted Distribution17 The 8 “D”s and NIEM Design Develop Deploy Document Dictionaries Discovery Differentiate Diagnose Repeatable, Reusable Process (Exchange Specification Lifecycle) NIEM IEPD Process *IEPD - Information Exchange Package Documentation

©2012 Oracle Corporation 18 NIEM IEPD Reports DictionariesDictionaries DiscoveryDiscovery NIEM IEPD / Exchange Delivery Lifecycle DiagnoseDiagnose DifferentiateDifferentiate DesignDesign DevelopDevelop Validated Templates / Schema Validated Templates / Schema XSD Schema XML Samples XMI / UML Models Production Results Production Results Requirements Updates Requirements Updates Document ation Drag and Drop Visual Designer DocumentDocument DeployDeploy Exchange Templates Exchange Templates Required IEPD artifacts

©2012 Oracle Corporation 19 Integrating into your existing processes and control Critical to manage and steer use of NIEM Ensure business outcomes match requirements Lessons learned with NIEM The earlier in the software development process that NIEM is introduced the better the overall outcomes Each step has deliverables – business plan, delivery architecture, data model, alignment reports, schema, test results that can be monitored in your SDLC* Monitoring products reinforces reuse as best practice Tools essential in review processes Collaboration tools can coordinate artifacts across projects and participants Test bed tools and open APIs reduce integration alignment costs *SDLC – Software Development Life Cycle

SOFTWARE DELIVERY Anatomy of a NIEM “IEPD” It is not just all about the XSD schema

©2012 Oracle Corporation 21 IEPD Components & Requirements IEPD IEM IEPD MPD NIEM Core Schema(s) Domain Schema(s) Main Document Catalog Change Log Sample XML Instance In order to be NIEM-conformant, the IEPD must adhere to: 1.NIEM Conformance Document 2.NIEM Naming and Design Rules (NDR) v1.3 3.NIEM Model Package Description (MPD) Specification v1.0

Formal NIEM IEPD Packaging Details IEPD ArtefactDescription Exchange Files Subset Schema (SSGT) Subset of the full NIEM schema—a compressed directory of schemas to distinguish from other schema sets. Wantlist User requirements -distinguishes user-required data components from components they depend on for conformance. Generated by and up-loaded to the Schema Subset Generation Tool (SSGT). This is an open specification and the SSGT is not required to create a wantlist, though it is easier. Exchange Schema Base document schema that defines the XML root element and is generally named after the IEPD itself. Also known as the document schema, reference schema, or root schema. Constraint Schema Constraints for separate constraint validation path—a compressed directory of schemas to distinguish from other schema sets. Extension Schema Specification for extended components—separate local name-spaces of components not contained in NIEM. Sample XML Instance Example instance(s) – may reference optional stylesheet. Stylesheet (generic) Example stylesheet for display of instance(s). Documentation Master Documentation The Master Document is the main document for which all of the context and details around the exchange are explained. This document includes, the overview, as well as details surrounding the exchange, business drivers and requirements Exchange model XMI Exchange model in standard open format (xmi, vsd, zargo) and standard open graphic (jpg, pdf, etc.) preferably a Unified Modeling Language (UML) model. Business Rules Business rules in one of the following formats: (1) plain or structured English, (2) written into master documentation, (3) Schematron or other formal business rule language, or (4) generated by a development tool. Mapping to NIEM Mapping of domain components, tagged with constraints (i.e., cardinality, etc.) to NIEM components; preferably the Component Mapping Tool (CMT). Extended Components Components created because they were not in NIEM—may be part of mapping spreadsheet and include structure and definitions of new components. Prefer CMT. Change log Record of cumulative changes from previous IEPD versions. The initial IEPD simple records its creation date. Catalog Catalog XML file A machine-readable list of artifacts in the IEPD in an open, portable and browser-displayable format (simple manual edit). Metadata XML file All metadata of owner and domain to be registered with the IEPD (simple manual edit). Generated using CAM Never Used Word Document Manual Entry

©2012 Oracle Corporation 23 XML Technology Components NIEM: need to manage complete aspects of exchange delivery, not just the structure and content data model

©2012 Oracle Corporation 24 Swim Lane View UML Technologies XML Technologies W3C Schema This is what NIEM exchanges are delivered with on-the-wire! UML XMI XSD Namespace Enumerations XSLT XQuery XPath XML Patterns Associations Cardinality Stereotypes Inheritance Facets Elements Attributes Content Complex Types Audience: Data Modellers / Designers Audience: Software Developers and Middleware Engineers Audience: General content delivery and applications; content engineers; rules representation systems; multimedia delivery; geospatial systems Objects WebWeb HTML JavaScript JSON REST SQL SOAP Audience: Web application developers Open-XDX 24

TOOLING TO THE RESCUE Accelerating and Improving the NIEM process

©2012 Oracle Corporation 26 IEPD Delivery Tasks / Tools Matrix CAMCAMV Open-XDX (OPX) Exchange Schema Development X Canonical Model Development / Generation X Exchange Schema Generation X NIEM Compliance Checking / Reporting X WSDL / JAXB Binding Generation X Operation Message Test Case Generation X Production Services Message Generation X SQL Schema Mapping XX NIEM IEPD Artifact Generation X - Cross-reference spreadsheet X - UML model X - Business Rules Report X - XSD schema package X Conformance Test Suite X Run time XML instance validation X In order to be NIEM-conformant, the IEPD must adhere to: 1.NIEM Conformance Document 2.NIEM Naming and Design Rules (NDR) v1.3 3.NIEM Model Package Description (MPD) Specification v1.0

© 2012/13 Oracle Corporation – Restricted Distribution27 CAM Toolkit for NIEM Concepts behind the Content Assembly Mechanism approach Understanding the steps involved in top down / bottom up exchange design, testing and delivery Applicability to NIEM and IEPD / MPD delivery * NIEM – National Information Exchange Model * IEPD – Information Exchange Package Documentation *MPD – Model Package Description

© 2012/13 Oracle Corporation – Restricted Distribution28 CAM Templates – WYSIWYG Exchange StructureRules Documentation CAM Template

© 2012/13 Oracle Corporation – Restricted Distribution29 INGEST CAM and W3C Schema Wizards EXPORT XSLT tools process entire XSD collection XSLT tools process entire XSD collection XSD Collection Structure Rules Documentation *Code list xsd Namespace * Optional XML Schema xsd subset *Constraint xsd *Extension xsd *Restriction xsd Industry Domain schemas Code list xsd LEGEND: Exchange xsd Exchange Schema CAM Template

CAM Editor – Main Screens

© 2012/13 Oracle Corporation – Restricted Distribution31 CAM Toolkit Features Dictionary “Drag and Drop” of components to build exchanges Visual editor and structure editing Cross-platform deploy - Windows, Ubuntu/LINUX, Mac; 32/64bit Mind map models – integration with Freemind tool UML/XMI export – import to UML tools / ArgoUML Code list table lookups – rule based and selectable This allows people to fully integrate their exchange development from dictionaries through to working deployed exchange schema. 31

© 2012/13 Oracle Corporation – Restricted Distribution32 XML Exchange Package Artifacts Template Evaluation Report – Naming and Design Rules assessment – Interoperability factors review – Spell checking Comparison to Dictionary – Reuse report, cross-reference spreadsheet, subset “wantlist” Generate Schema Bundle with layout styles (e.g. NIEM) Realistic test samples generation Testing with runtime validation engine (CAMV) Business information exchange documentation Models – XMI and mind maps

© 2012/13 Oracle Corporation – Restricted Distribution33 CAM Template Evaluation Report The purpose of the CAM template evaluation report is to provide a suite of checks against a template that are difficult, tedious or otherwise to determine by manual inspection of a template structure and its associated rules and annotation details The report covers the following areas: – alerts users to potential issues with their template and provides suggestions for improving the functional details – addresses logical and conceptual issues that simple syntax checking alone does not flag – can be used as part of a formal review process to ensure a baseline for quality control

© 2012/13 Oracle Corporation – Restricted Distribution34 CAM Toolkit Capabilities

© 2012/13 Oracle Corporation – Restricted Distribution35 Online Tutorials –

KEY PILLAR OF DICTIONARIES Introduction and Concepts

© 2012/13 Oracle Corporation – Restricted Distribution37 The Semantic Framework and Vision XML/NIEM Components in Neutral Dictionary Representation UML Technologies XML Technologies W3C Schema Tools Semantic Representations Rules Technologies HTML 5 rendering tools Security solutions Middleware solutions

© 2012/13 Oracle Corporation – Restricted Distribution38 Dictionaries: Key Pillar of Information Sharing Canonical XML Components Dictionary 1 1 Component Associations and Couplings 2 2 Exchange Templates and Rules 3 3 W3C Schema and Model Representations 4 4 Delivery Control, Messaging, Security 5 5 Presentation Collaboration Services Artefact relationships Implementation Artifacts and Examples 6 6 Navigation and Query Tools Domains Reuse Library Components Automatic Rendering Tools Deployment Environments and Middleware XML User Inserts Relationship Lookups Template Catalog Testing Workbench, Rules Engine, Data Samples, Integration ETL Schema XML Models Component s Canonical Dictionary Collections

©2012 Oracle Corporation 39 Dictionaries Semantics Approach Abstract dictionary representations with strong semantics, rules and code lists support Schema, models, documentation are generated from dictionary definitions automatically Contextual mechanisms allow linear number of true core components – not exponential growth Easier for practitioners to leverage with more predictable and repeatable results Enable domain use horizontally across government ETL mapping automation support Continuous collaboration environment

©2012 Oracle Corporation 40 Developing domain dictionaries (EIEM) Allows domains to manage their components libraries Provides consistency for project development teams Sets of NIEM consistent XML exchange components Aligned to enterprise data stores Optimized for reuse and interoperability Save time and effort across the enterprise Perennial question for developers – when should I use NIEM components, and when our own local ones? Are there components already available for that purpose? Provide formal mechanisms and procedures to share components and collaborate across SDLC process Provide external parties consistent data views *EIEM – Enterprise Information Exchange Model

©2012 Oracle Corporation 41 EIEM/BIEC Business Information Exchange Components (BIEC) NIEM-conforming XML schema data component definition that meets a particular recurring business requirement for an enterprise Enterprise Information Exchange Model (EIEM) NIEM-conforming schemas that define data components to be reused in IEPDs developed by an enterprise; collection of enterprise BIECs organized into a subset and one or more extension schemas

© 2012/13 Oracle Corporation – Restricted Distribution42 Managed Dictionaries – Authoritative Sources Slide 42 Centralized Distributed schema SQL SFIS Values Library Service, NIEM, Metadata Registry ALLOWED VALUES schema Managed Dictionaries Publish

SUMMARY Next Steps

© 2012/13 Oracle Corporation – Restricted Distribution44 Why CAM and NIEM? “Can you give us a brief overview of how you expect IT professionals and line of business managers to use CAM*?” – Van Ristau, CTO DLT Inc. *CAM editor toolkit – and OASIS Content Assembly Mechanism standard –

© 2012/13 Oracle Corporation – Restricted Distribution45 Response Line of business managers can use CAM to drive better alignment to NIEM and faster more consistent and predictable delivery of NIEM information exchanges CAM provides tools that allow business analysts to work collaboratively with software developers to build exchanges Teams use CAM to perform the 8 “D’s” tasks when implementing NIEM exchanges CAM automates the delivery of key XML artifacts for NIEM exchange services XML validation can be deployed using the CAMV engine Open-XDX provides rapid flexible open data web services

© 2012/13 Oracle Corporation – Restricted Distribution46 Public Sector Open Data Services Delivery Public Sector Open Data Services Delivery Open Platform Solutions Visual Editor SQL Browser Dictionaries Drag and Drop Query/Response Generate Data Validate Data Send Data JAX-WS soapUI CAM Editor Content Models OER Integration Web Forms UI Open Data Test Bed Services Dictionaries DB Web Open XDX CAMV Open-XDX / CAM Editor Dictionary Security SQL XML SQL Repository CAM Editor Templates Policy Rules XACML Templates XACML

© 2012/13 Oracle Corporation – Restricted Distribution47 Technology Components and Features Providing open data sharing – Open-XDX component accelerates deployment of interfaces Aligning meaning and definitions – Collaboration using XML component dictionaries and NIEM Formal information verification and content rules – CAMV rules engine provides runtime handling – Industry domain test bed facilities – Rules documentation and sharing Business process integration – Oracle BPM and Forms combined with CAMV and Open-XDX In progress – Security profiles and policy templates – Upcoming release of CAM Editor and XACML integration

© 2012/13 Oracle Corporation – Restricted Distribution48 CAMV interactive testing Cloud Services Deployment Open-XDX web services XML Technologies W3C Schema Tools Semantic Representations Rules Technologies Security solutions Middleware solutions CAMV test bed for industries Registry for Dictionaries Security Profile Templates Web site and collaboration tools Policy Rules Policy Rules

CAMeditor.ORG Project Statistics SNAPSHOT OF PROJECT ACTIVITIES 135,000+ CAMeditor.org page visits to site 165+ countries have downloaded tools; 30% of visitors are from U.S.; 800+ downloads weekly student views of online video training resources 8 languages now available

©2012 Oracle Corporation 50