1. 1 1 Accelerated NIEM IEPD Development Process, Procedures and Techniques

2. 2 2 NIEM and IEPD Overview NIEM – National Information Exchange Model approach (http://www.niem.gov)http://www.niem.gov Provides Naming and Design Rules (NDR) and process for schema development Provides set of core component schema (NIEM core) and schema for domains – family services, census, environment, trade, hazmat, nuclear, justice, immigration, et al (15 domains). IEPD – Information Exchange Package Documentation Describes formal exchange that conforms to NIEM Provides schema, example XML, supporting artifacts Re-uses NIEM components Defines domain specific components

3. 3 3 IEPD “Fish Hook” lifecycle 3 2 4 1 5

4. 4 4 Agenda Developing NIEM IEPDs Current accepted practice – pitfalls and challenges How to do this faster, simpler, more reliably? Accelerated process overview Blueprints and Dictionaries Constructing your exchange and blueprints Leveraging re-use – NIEM, LEXS, UCORE dictionaries Creating your own domain dictionary from XSD or UML Generating IEPD artifacts NDR evaluation, Exchange schema, NIEM mapping, “want list” + subset, XML instances, business rules documentation Summary

5. 5 5 Developing IEPDs Current Practice – Pitfalls and Challenges How to do this faster, simpler, more reliably? Accelerated Process Overview

6. 6 6 Current Practice – Conceptual

7. 7 7 Sample XML Instance Sample XML Instance Sample Style Sheet Sample Style Sheet Flows/ Use Case Model Flows/ Use Case Model Exchange Model Exchange Model Schema Building Schema Building Subschema Generator Subschema Generator Data Modeling Data Modeling Tool Modeling Tool Other IEPDs Other IEPDs Exchange Inception Exchange Inception JIEM & Ref Model JIEM & Ref Model Business Model Business Model Standards Mapping Standards Mapping Search Tool Search Tool Mapping Tool Mapping Tool Current Practice - Mechanics NIEM Mapping NIEM Mapping Wantlists Packaging & Posting Packaging & Posting IEP Dev Tool IEP Dev Tool All IEP Artifacts All IEP Artifacts Test via Validating Test via Validating Forms XML Editor XML Editor Support Docs Support Docs Metadata IEPD Posting IEPD Posting UML Domain Model UML Domain Model Business Rules Business Rules Document Schemas Document Schemas Extension Schemas Extension Schemas Constraint Schemas Constraint Schemas

8. 8 8 Current Practice – Team Matrix Skillset QualificationsExperienceResource NIEM practitioner / project lead W3C XSD schema expert Domain business analyst UML/ data modelling practitioner NIEM tooling user XML testing and development Documentation resources - Proficient in NIEM methodology and IEPD development requirements - - Proficient in complex XSD syntax writing. Familiar with developer support tooling and constraints - - Familiar with project requirements and business applications and also developing XML based exchanges - - Use of UML diagramming and models. Information modelling - - Knowledge of NIEM tools available from www.NIEM.gov website. - - Creating test environments, working with XML test cases, test data generation - - Writing documentation and spreadsheets Prior NIEM project work One to two years actively writing XSD schema One year or more in application area Prior UML based modelling NIEM tools training and XML Data analysis and XML content creation Technical writer 1 FTE

9. 9 9 IEPD - Pitfalls and Challenges Significant amount of manual labor needed to develop all the NIEM IEPD artifacts and XML related end products Multi-discipline team and supporting cast of NIEM / XML savvy developers needed Disconnect between the software delivery teams’ schedule and process and the NIEM IEPD development team and process; production system not matching what the IEPD says it does Alignment to existing domain Enterprise Data Model (EDM) Varying quality of hand checked results and no consistency of technical approach to schema development techniques and re- use of NIEM Process not repeatable and predictable Scalability - differing production XML details across teams, often incompatible across implementations

10. 10 IEPD Level of Effort Estimates Component TasksTimingsConstraints Collect exchange needs Perform XSD schema development with NIEM alignment Documentation of each element Document NIEM dictionary mapping (want list) Create test cases and examples Perform interoperability testing Create IEPD document Model information needs XSD syntax writing Excel spreadsheet Sets of XML instances Build test environments Word documentation Weeks 2 to 5 days Weeks Spiral analysis Complex with steep learning curve and limited practitioners. Manual preparation and review Manual hand editing of XML from XSD Test harnesses vary Manual preparation Currently 800+ hour process for 300+ node exchange

11. 11 Improving the Process Resolving the issues and challenges Ensuring consistent results that can be easily reviewed Leverage existing dictionary work and repositories of components that the enterprise already has Reduce the learning curve and need for specialized skills Business analysts not excluded from design, review and implementation by technical barriers Customizable and configurable so can adapt to changing requirements

12. 12 Faster, Simpler, Predictable Tooling automates much of the manual tasks; ensures predictable quality of results Reduce need for specialized technical knowledge of NIEM and XML Provide consistent approach that leverages NIEM techniques with built-in NIEM smarts and knowledge Tooling checks for common pitfalls, applies NDR checks Allow business analyst to complete much of the design work and crosscheck application details Leverage reuse of domain component dictionaries and blueprints Lockstep development to IEPD artifacts and their delivery Accelerate development tasks (test cases, testing, schema writing) Produce result that are neutral to developer tooling platforms Process repeatable and replicatable when requirements / versions change

13. 13 Using Dictionaries & Blueprints Dictionaries provide reference sets of components to be used in exchanges; three possible sources: NIEM domain dictionaries imported from existing NIEM schema Domain dictionary built from an Enterprise Data Model schema Reverse engineered out from existing exchange schema Blueprint Is the outline of the structure components to be used in an exchange schema Can import components from one or more domain dictionary collections Sketches out the desired information exchange with re-use of existing exchange component structures, plus any local additions / extensions Expander tool reads the blueprint, references the dictionary, and constructs the complete exchange schema

14. 14 Accelerated Process Overview Business Modelling Charter Work- Group Project Inception dictionaries Exchange Content Modelling Blueprints Dictionary Expand and Evaluate Results SSGT Wayfarer Excel Search tools Review / Tailor Exchange Structures Generate IEPD Artifacts and Test them Generate IEPD Artifacts and Test them Package IEPD Complete d Exchange Structures NDR Evalu - ation Dictionary Generation Completed IEPD artifacts Test Cases Validation Engine XML examples XSD schemas Subset Schema “Wantlist” Mapping Spreadsheet Rules Report Rendering Stylesheet Import XSD Naming / NDR / spell check Generate Dictionary Import EDM Test examples and rules Model Exchange Expand, Review, Refine

15. 15 Blueprints and Dictionaries Constructing your exchange and blueprints Leveraging re-use – dictionaries for NIEM, LEXS/UCORE Creating your own domain dictionary from XSD or UML

16. 16 Blueprint Approach Overview Def Ele NIEM dictionaries formatted as XML IEPD generator tools (CAM) Automated Manual LEGEND Components Definition (XML) Local domain dictionary formatted as XML Build 5 5 2 2 Exchange Blueprint Designer User Interface Expand Structure Exchange Structure Pick Components Structure Outline Blueprint Target applications EDM Ele Def DDL IEPD Package IEPD Package Exchange Components Exchange Components 1 1 3 3 6 6 7 7 Enterprise Data Model Import and refactor for use with NIEM Dictionary Repository 4 4

17. 17 Blueprint Development Tools Expander Tool Expander Tool NIEM dictionary Domain dictionary Component Definitions 4 4 SSGT Excel Searc h Tools Searc h Tools 2 2 Blueprint Designer Blueprint Designer 1 1 Insert Dictionary Parent Components Insert Dictionary Parent Components 3 3 Completed Exchange Template Completed Exchange Template 5 5

18. 18 Blueprint follows model Console Log detail Blueprint Expander Example Exchange components outline EXCHANGE BLUEPRINT COMPLETED EXCHANGE TEMPLATE 1 1 3 3 Structure Details Expanded Expander Tool Expander Tool 2 2 Dictionary Lookups

19. 19 Exchange Template Editor Exchange Structure items Rules for each item Domain Components Namespaces Can add new custom domain Items 1 1 2 2 3 3 4 4

20. 20 Generating IEPD artifacts NDR evaluation, NIEM mapping, “want list” + subset, Exchange Schema, XML instances, business rules documentation

21. 21 IEPD Generation Steps Toolkit for IEPD artifacts generation Suite of menu options and tools in desktop CAM toolkit editor (each run custom xslt scripts on the exchange CAM template xml) CAM Template 1.Run template Evaluation Report 2.Compare to NIEM dictionary - create mapping spreadsheet - create wantlist.xml - NIEM online SSGT uploading + subset schema generation Generate business rules report 3.Generate exchange XSD schema 4.Generate XML test instance(s) 2 2 1 1 3 3 4 4 5 5 EXCHANGE TEMPLATE Structure Rules Documentation

22. 22 NDR Evaluation Report Provides scoring, alerts, warnings and potential issues including: Naming and Definition Conventions Naming / Restriction consistency checks Interoperability Enablers/Inhibitors checks Rules integrity and duplicates Statistics on exchange size

23. 23 Evaluation Report (NDR) example Part of an example report for LEXS getDataItemRequest template displayed in toolkit HTML viewer

24. 24 Compare to NIEM dictionary References NIEM dictionary of names and properties currently in NIEM 2.1 or 2.0 releases Matches on physical names Reports mapping details Compatible with Microsoft Excel Report can be used to do spell checking (in Excel) Generates wantlist.xml file Wantlist can be uploaded to SSGT and then subset schema generated (http://niem.gtri.gatech.edu/niemtools/ssgt/SSGT- Options.iepd )http://niem.gtri.gatech.edu/niemtools/ssgt/SSGT- Options.iepd

25. 25 NIEM cross-reference spreadsheet Formatted view in Microsoft Excel of import of cross- reference report details (from generated XML file)

26. 26 NIEM online SSGT and wantlist.xml Options menu and Load Wantlist option with resulting load of subset details

27. 27 Documentation and Schema Documentation: Create HTML report of exchange schema details and associated content and business rules Report layout and content designed to be reviewed by business analysts Schema: Generate XSD schema for exchange Customizable IEPD folder layout management by namespace for extension, subset and exchange schema components Writes XSD schema in syntax that is clear, simple and compatible with deployment tooling environments

28. 28 Business Rules Documentation Part of the example rules for LEXS getDataItemRequest template displayed in toolkit HTML viewer

29. 29 Export Exchange to XSD Schema Completed Exchange Structure Set Exchange Options Complete set of NIEM exchange schemas generated Export Template to Exchange XSD:

30. 30 Exchange Schema Generated Each namespace file is import for those specific type definitions Reviewing XSD results in a schema editor tool Set of XSD files with filename and namespace suffix

31. 31 Examples Generation Support for software development testing process Designed to allow creation of concrete realistic examples not just random value based Hinting system allows insertion of actual test system values into XML examples Can create both valid and invalid examples to support unit testing of application software Exclude capability allows generator to create examples that contain only a portion of the entire exchange Control over random seed value used allows re- generation of identical test cases

32. 32 XML example generation wizard View of CAM toolkit with LEXS getDataItemRequest and dialogue for XML test example generator tool

33. 33 Running validation rules tests Built-in validation engine allows testing of XML instances against actual exchange rules. Critical to ensure that the exchange validates actual live production example scenarios correctly Allows deployed solution to match IEPD exchange details Errors can be reviewed interactively in exchange visual interface Post-processing of validation results allows unit regression tests to be created with reporting of errors, warnings and information level notes

34. 34 Run Exchange Template Pick XML test case to validate Run validation Review results in visual editor

35. 35 Formal IEPD Packaging Details IEPD ArtefactDescription Exchange Files Subset Schema (SSGT)Subset of the full NIEM schema—a compressed directory of schemas to distinguish from other schema sets. Wantlist User requirements -distinguishes user-required data components from components they depend on for conformance. Generated by and up-loaded to the Schema Subset Generation Tool (SSGT). This is an open specification and the SSGT is not required to create a wantlist, though it is easier. Exchange Schema Base document schema that defines the XML root element and is generally named after the IEPD itself. Also known as the document schema, reference schema, or root schema. Constraint SchemaConstraints for separate constraint validation path—a compressed directory of schemas to distinguish from other schema sets. Extension SchemaSpecification for extended components—separate local name-spaces of components not contained in NIEM. Sample XML InstanceExample instance(s) – may reference optional stylesheet. StylesheetExample stylesheet for display of instance(s). Documentation Master Documentation The Master Document is the main document for which all of the context and details around the exchange are explained. This document includes, the overview, as well as details surrounding the exchange, business drivers and requirements Exchange model Exchange model in standard open format (xmi, vsd, zargo) and standard open graphic (jpg, pdf, etc.) preferably a Unified Modeling Language (UML) model. Business rules Business rules in one of the following formats: (1) plain or structured English, (2) written into master documentation, (3) Schematron or other formal business rule language, or (4) generated by a development tool. Mapping to NIEM Mapping of domain components, tagged with constraints (i.e., cardinality, etc.) to NIEM components; preferably the Component Mapping Tool (CMT). Extended components Components created because they were not in NIEM—may be part of mapping spreadsheet and include structure and definitions of new components. Prefer CMT. Change logRecord of cumulative changes from previous IEPD versions. The initial IEPD simple records its creation date. Catalog Catalog XML fileA machine-readable list of artifacts in the IEPD in an open, portable and browser-displayable format (simple manual edit). Metadata XML fileAll metadata of owner and domain to be registered with the IEPD (simple manual edit).

36. 36 Summary Dictionary driven exchanges Blueprint enabled reuse Automated IEPD generation Alignment to NDR Principles and Rules Testing and validation support

37. 37 Review Top Down development Reference dictionary components Create exchange blueprint Run Expander tool Refine desired structure in visual editor Automated IEPD generation Schemas, XML, documentation, mapping to NIEM Test generated example XML with rules validation Dictionary driven reuse Enterprise Data Model and NIEM/LEXS components NDR Principles and Rules Best practices for interoperability and schema techniques

38. 38 Reference Materials References and Links

39. 39 Links and References DOWNLOADS - CAM Toolkit download https://sourceforge.net/projects/camprocessor SUPPORTING NIEM MATERIALS - NIEM Naming and Design Rules (NDR) 1.3 http://www.niem.gov/pdf/NIEM-NDR-1-3.pdf NIEM Practical Implementer’s Course (online) http://www.niem.gov/elearning.php NIEM Course Modules (17 free PDFs) http://www.niem.gov/course.php Techniques for Building and Extending NIEM XML Components http://www.niem.gov/Techniques_for_Building_and_Exten ding_NIEM.pdf http://www.niem.gov/Techniques_for_Building_and_Exten ding_NIEM.pdf