IJIS Industry Summer Briefing – July 2011 NIEM Test Model Data Deploy Requirements Build Exchange Generate Dictionary Using NIEM EIEM Dictionaries for Better Information Alignment Exchange Development Peter Doolan, GVP & Chief Technologist, Oracle Public Sector David Webber, Information Architect, Oracle Public Sector IJIS Industry Summer Briefing – July 2011

Overview The NIEM NTAC has introduced the use of Enterprise Information Exchange Models (EIEM) and dictionaries How can your organization leverage these, what are the technical advantages, business benefits and challenges? Understand lessons learned from existing experience creating EIEM dictionaries and models and what are the resources and assets needed to do this for an enterprise. Creating IEPD exchanges using EIEM assets can enable rapid leveraging of NIEM for organizations; however, managing the process and stakeholders for the EIEM then provides additional needs.

Introduction What is the big picture of EIEM for federal enterprises? How does government acquire and manage these assets? How can you leverage your existing infrastructure? What roles do semantics and rules play? EIEM collaboration and alignment Lessons learned and examples What tools are available? Initial support provided in GTRI dictionary server CAM toolkit dictionary support

Transitioning from NIEM past/present to NIEM future EIEM – the big picture

NIEM Past and Present Challenges Vertical domain vocabulary for DHS and DOJ One dozen domain dictionaries Technology limited - built using XSD schema Core components highly contextual to DHS/DOJ Surprisingly – no actual logical models of information! Gap with semantic technologies integration Limited data content rules and code lists Mapping automation support missing Multi-year development life cycles

NIEM Future? Abstract dictionary representations with strong semantics, rules and code lists support Schema, models, documentation are generated from dictionary automatically Contextual mechanisms allow linear number of true core components – not exponential growth Easier for practitioners to leverage with more predictable and repeatable results Enable domain use horizontally across government ETL mapping automation support Continuous collaboration environment

Understanding EIEM today EIEM – Explored

Why do we need EIEM dictionaries? Allows domains to manage their components libraries Provides consistency for project development teams Sets of NIEM consistent XML exchange components Aligned to enterprise data stores Optimized for reuse and interoperability Save time and effort across the enterprise Perennial question for developers – when should I use NIEM components, and when our own local ones? Are there components already available for that purpose? Provide formal mechanisms and procedures to share components and collaborate across SDLC process Provide external parties consistent data views

How to create EIEM dictionaries Top down - directly from information models Enterprise modelling tools SQL database physical table definitions Bottom up Harvesting from existing data assets Harvesting from XML exchange structures and schema Using neutral abstract standards based representations

Conceptual Information Model Canonical Components Dictionary XML Follows Naming and Design Rule (NDR) principles and guidelines Each compound component Parent (ABIE) Item Parent (ABIE) Item Parent (ABIE) Item Parent (ABIE) Item . . . . . Each atomic component Attribute (ASBIE) Child (BBIE) Item Child (BBIE) Item Child (BBIE) Item Child (BBIE) Item Attribute (ASBIE) Shows overall components in the dictionary. Parents contain references to children. Children may exist atomically without parent. One child element may be referenced by multiple parents. Attributes can be associated with children or parent. Attribute (ASBIE) ebXML CCTS terms (ABIE, BBIE, ASBIE) Parent = Aggregate Business Information Entity Child = Basic Business Information Entity Attribute = Association Business Information Entity Attribute (ASBIE) Optional attributes of component * CCTS – Core Components Technical Specification

Example – Person Name Person Name (ABIE) Verified Details? (ASBIE) First Name (BBIE) Middle Name (BBIE) Last Name (BBIE) Previous Name? (ASBIE) Language Code (BBIE) Language Code may exist independently of Person Name Verified Details and Previous Name are flags that denote additional information about the entity they are associated with There are three component items aspects: structure relationships; content rules; definitions

Top Down - ICE EIEM dictionary generation Production Rules ICE Enterprise Data Model (EDM) ERwin Refactoring, Compress, Renaming Tool NIEM NDR Evaluations 1 2 3 OASIS CAM template Model Components XSD schema Ele Export Import Apply Def 5 4 DDL Code Lists Code Lists Excel Spreadsheet ERwin model Completed CAM template Analyst Review BENEFITS: Will reduce ICE development time and effort Pre-built NIEM consistent XML components XML consistent with ICE enterprise SQL data Removes scope for definition errors Maximizes reuse potential across ICE Open public standards based approach Open technology infrastructure ERwinClient 7 6 XML + XSD Canonical Dictionary of ICE exchange components Generate Standard Components Dictionary XML Write ebXML CCTS compatible (ABIE, BBIE, ASBIE)

Renaming Tool Expands abbreviations (has built-in lists with context) Corrects common typos and spelling errors (200+) Adjusts names to camel case with no separators (_) Retains acronyms Adds name suffix according to NIEM representation terms and type of content, e.g. date, number, code, etc Optionally refactor rules by their type for interoperability; make nillable optional; relax DB length restrictions Can optionally insert namespace prefix Can be configured by editing the lists in: word-utils.xsl Refactoring of rules by type removes inconsistent length constraints often found in databases. These are not consistent with XML approach of having unconstrained content. Two approaches possible. 1# - leave all constraints “as is” in the dictionary – control their use by using Expander tool options, or 2# fix with renaming tool.

Bottom Up – OASIS/NIST EML EML – Election Markup Language State and Local voting management systems data standard Option 2 – Derive from existing exchange XSD schema Import each XSD and merge into CAM dictionary 2 3 4 CAM template NDR Evaluation, Refactor, Renaming Tool Merge & Generate Dictionary XML Exchange XSD schema Import CAM template XML Exchange XSD schema Import OASIS CAM template Dictionary of canonical components Exchange XSD schema Import Analyst Review 1 ebXML CCTS compatible (ABIE, BBIE, ASBIE) EML Core XSD schema 5 OASIS CIQ schema Generate model diagrams UML/XMI Mind maps OASIS GIS schema

N-DEx Dictionary and Model Dictionary viewer with N-Dex components Freemind Interactive Model (Harvested from N-DEx schema – ndexia.xsd and ndexibp2.xsd)

Leveraging EIEM dictionaries Where can we use EIEM assets? What formats do we need to use? Are there applicable standards / guidelines?

Visual Designer with Dictionary Collection Domain dictionary Industry dictionary 1 Component Definitions Exchange Designer Component Definitions 2 Search Tools Insert Dictionary Parent Components Drag n’ Drop Tool 3 4 5 Completed Exchange Template

Alien Transfer Manifest – Mock Up dictionaries XML ICE EDM dictionary NIEM Immigration dictionary EID extensions dictionary ICE components referenced EID pre-defined collections based on NIEM + ICE + new Immigration component Namespaces of dictionary components Definitions stored as syntax neutral canonical XML NIEM components Collection This slide shows a conceptual prototype for an alien transfer exchange for repatriation purposes. The exchange structure is built using a selection of components from ICE, NIEM and domain defined assemblies. The dictionaries provide the structural and content rule details for those items. New items and collections in the EID dictionary eventually will be promoted to ICE EDM during the SLM process. Technical Note: Inline child elements for ice: components shown for illustrative purposes only; actually are type reuse references (paperclip icon on parent node).

Challenges and Opportunities Learning curve for application development teams Teams used to manually crafting XML Schema and Java code in developer XML IDE tools Paradigm shift Integration into SDLC and IEPD processes Integration with existing SOA stack environments Currently working with SAIC and State Department Automatic generation of WSDL and Java code assets Unit test XML instance generation XML test framework and validation services

Vision: Top Down Exchange Assembly Components Canonical XML Components Dictionary 1 XML User Inserts Navigation and Query Tools Relationship Lookups Component Associations and Couplings 2 Domains Reuse Library Exchange Templates and Rules 3 Artefact relationships Catalog Template Automatic Rendering Tools Presentation W3C Schema and Model Representations 4 Schema XML Collaboration Services Models Components Delivery Control, Messaging, Security Deployment Environments and Middleware 5 Testing Workbench, Rules Engine, Data Samples, Integration ETL Implementation Artifacts and Examples 6

Exchange Delivery Lifecycle Component Dictionary Drag and Drop Visual Editor Template Updates Design Monitor Author Test CAM Template XML Samples Deploy XSD Schema Production Log Results XMI / UML Models Compiled Templates Documentation

CAM Toolkit http://www.cameditor.org

CAM V2.0 Release Features Dictionary “Drag and Drop” of components to build exchanges Enhanced visual editor and performance - 3x faster, smaller memory footprint Cross-platform deploy - Windows, Ubuntu/LINUX, Mac; 32/64bit Mind map models – integration with Freemind tool UML/XMI export – import to UML tools / ArgoUML Code list table lookups – rule based and selectable This allows people to fully integrate their exchange development from dictionaries through to working deployed exchange schema. NIEM Team - Public Sector Sales

Available XML Dictionaries LEXS 3.1.4 dictionary OASIS EDXL dictionary OASIS EML dictionary NIEM 2.1 dictionaries CBRN dictionary Emergency dictionary Family dictionary Immigration dictionary Infrastructure dictionary Intelligence dictionary Justice dictionary Maritime dictionary Screening dictionary Trade dictionary Immigration blueprint NIEM core dictionary Packaged with CAM editor see dictionary folder of install + spreadsheet + sample control files XML XML XML Note: Those marked in bold are model style dictionaries with recursive components. Available from download site direct link: http://sourceforge.net/projects/camprocessor/files + includes spreadsheets and sample models Sampling of available dictionaries. Those marked in bold are model style dictionaries with recursive components. XML XML XML XML XML XML

EIEM Next Steps Align CAM dictionary tools with GTRI dictionary server Integrate to other modelling tools – not just ERwin Provide extended reporting tools for analysis across dictionary of components and between dictionary collections – report on Inconsistency Common components Extensions