Limited Distribution Release Open Information Interoperability Tool Suite Dr. Len Seligman, Dr. Ken Smith, Catherine Macheret, Chris Wolf

© 2009 The MITRE Corporation. All rights reserved 2 Limited Distribution Release Outline  Background: Interoperability Challenges  OpenII overview  Demo  Status and Next Steps

© 2009 The MITRE Corporation. All rights reserved 3 Limited Distribution Release Challenges and Opportunities  Resource-Intensive Interoperability Tasks: –Enterprise challenge: Developing, maintaining, discovering and harmonizing data standards –Challenge for programs: Adopting, translating and migrating standards to implement data exchanges  Interoperability tools reduce effort –Improving stewardship of standards –Increasing adoption of standards –Easing data translation among standards –Increasing number of data exchanges

© 2009 The MITRE Corporation. All rights reserved 4 Limited Distribution Release Mapping Needs for Standards Based Exchanges Target Database Source Database Exchange Schema Map NIEM Other Standards: Geospatial, Health, … Map

© 2009 The MITRE Corporation. All rights reserved 5 Limited Distribution Release Outline  Background: Interoperability Challenges  OpenII overview  Demo  Status and Next Steps

© 2009 The MITRE Corporation. All rights reserved 6 Limited Distribution Release Problem  It takes too long to productively share data today  Many available tools are vendor-specific and costly –Interoperability tools don’t interoperate!  More functionality needed –e.g., Find my sharing opportunities? IBM Oracle Microsoft

© 2009 The MITRE Corporation. All rights reserved 7 Limited Distribution Release OpenII  Collaboration with industry and academic leaders to build an open source information interoperability toolkit  Extensible, plug-in architecture  Rich & growing tool suite  Open source advantages: –Free! –Anyone can customize it, incorporate it into contractor tools OpenIntegration.org

© 2009 The MITRE Corporation. All rights reserved 8 Limited Distribution Release Demo

© 2009 The MITRE Corporation. All rights reserved 9 Limited Distribution Release Step 1: Manage Schemas Step 3: Generate Mapping Between Schemas Step 2: Identify Associations Among Schemas The Schema Integration Process Step 4: Use Results of Schema Mappings Schemas DDL XML OWL

© 2009 The MITRE Corporation. All rights reserved 10 Limited Distribution Release Importing Schemas

© 2009 The MITRE Corporation. All rights reserved 11 Limited Distribution Release Searching for Schemas

© 2009 The MITRE Corporation. All rights reserved 12 Limited Distribution Release Finding Schema Associations

© 2009 The MITRE Corporation. All rights reserved 13 Limited Distribution Release Analyzing Schema Relatedness

© 2009 The MITRE Corporation. All rights reserved 14 Limited Distribution Release Matching Schemas

© 2009 The MITRE Corporation. All rights reserved 15 Limited Distribution Release Use Results of Schema Mappings CREATE TABLE "SK260" ( "skid" int GENERATED ALWAYS AS IDENTITY,"V0" String, "V1" String, "V2" int, "V3" int, "V4" String, "V5" String ); INSERT INTO "SK260" ( "V0", "V1", "V2", "V3", "V4", "V5" ) SELECT DISTINCT "T0"."title","T0"."authorFName","T0"."ISBN","T0"."productID","T0"."authorLName","T0"."publisher" FROM "Books" AS "T0", "Data" AS "T1" WHERE "T0"."" = "T1"."ID"; CREATE TABLE "SK243" ( "skid" int GENERATED ALWAYS AS IDENTITY,"V0" String, "V1" String, "V2" int, "V3" int, "V4" String, "V5" String ); INSERT INTO "SK243" ( "V0", "V1", "V2", "V3", "V4", "V5" ) SELECT DISTINCT "T0"."title","T0"."authorFName","T0"."ISBN","T0"."productID","T0"."authorLName","T0"."publisher" FROM "Books" AS "T0", "Data" AS "T1" WHERE "T0"."" = "T1"."ID"; INSERT INTO "Data" ("ID","") SELECT DISTINCT ( SELECT "skid" FROM "SK260" WHERE "V0" = "T0"."title" AND "V1" = "T0"."authorFName" AND "V2" = "T0"."ISBN" AND "V3" = "T0"."productID" AND "V4" = "T0"."authorLName" AND "V5" = "T0"."publisher"), ( SELECT "skid" FROM "SK260" WHERE "V0" = "T0"."title" AND "V1" = "T0"."authorFName" AND "V2" = "T0"."ISBN" AND "V3" = "T0"."productID" AND "V4" = "T0"."authorLName" AND "V5" = "T0"."publisher") FROM "Books" AS "T0", "Data" AS "T1" WHERE "T0"."" = "T1"."ID"; INSERT INTO "Books" ("author","description","format","ID","ISBN","length","productId","publisher","supplierID","title") SELECT DISTINCT "T0"."authorLName",1,1, ( SELECT "skid" FROM "SK243" WHERE "V0" = "T0"."title" AND "V1" = "T0"."authorFName" AND "V2" = "T0"."ISBN" AND "V3" = "T0"."productID" AND "V4" = "T0"."authorLName" AND "V5" = "T0"."publisher"),"T0"."ISBN",1,"T0"."productID","T0"."publisher",1,"T0"."title" FROM "Books" AS "T0", "Data" AS "T1" WHERE "T0"."" = "T1"."ID"; DROP TABLE "SK260"; DROP TABLE "SK243"; SQL Data Transform

© 2009 The MITRE Corporation. All rights reserved 16 Limited Distribution Release RMAP OpenII Architecture & Toolkit OpenII SchemaStore Metadata Repository (M3 Metamodel) Importers, Exporters, Schema Search, & Display, Admin functions Eclipse Framework WSDL, Repository Selection Affinity Overlap HarmonyUnity XMAP Common Ground Workbench

© 2009 The MITRE Corporation. All rights reserved 17 Limited Distribution Release Outline  Background: Interoperability Challenges  OpenII overview  Demo  Status and Next Steps

© 2009 The MITRE Corporation. All rights reserved 18 Limited Distribution Release Selected Customer Applications  Army: Distributed Common Ground Station (DCGS-A)  National Geospatial-intelligence Agency (NGA) and DHS: aligning infrastructure protection models  Air Force: aligning multiple large code lists; also data models with architecture products  Army/G8/Force Development/Warfighter Analysis Office

© 2009 The MITRE Corporation. All rights reserved 19 Limited Distribution Release Status  Most OpenII tools are mature prototypes –Proven useful on real problems –Innovative –Not shrink-wrapped for NIEM applications, however  Proposed next step: Customize OpenII to better support NIEM workflows –Tune matching algorithms for known NIEM schema patterns –Create bridge to metadata repositories to maximize reuse –Integrate with IEPD production tools –Apply to DHS pilots

© 2009 The MITRE Corporation. All rights reserved 20 Limited Distribution Release Proposed NIEM Use Cases  NIEM Domain and Core Vocabulary Stewardship –Creating new core vocabularies –Harmonizing newly introduced NIEM domains into existing set –Mapping enterprise conceptual models to core vocabularies and data assets  Data Exchange Implementation –Searching IEPD repositories for relevant IEP schemas to leverage –Mapping business data model to NIEM models –Mapping source and/or target schemas to NIEM exchange schema and auto-generating data translation code

© 2009 The MITRE Corporation. All rights reserved 21 Limited Distribution Release Summary  Open architecture, open source –Contributions from industry, academic leaders –Free to incorporate into COTS or GOTS –Customizable, extensible  Benefits –Improved stewardship and harmonization of standards –Easier adoption of standards –Quicker standards-based interoperability We are ready to begin pilots and to customize tools for DHS needs

© 2009 The MITRE Corporation. All rights reserved 22 Limited Distribution Release Backup

© 2009 The MITRE Corporation. All rights reserved 23 Limited Distribution Release Domain Overlaps in Maritime and International Trade

© 2009 The MITRE Corporation. All rights reserved 24 Limited Distribution Release Developer View of Gaps  No easy way to browse across program and application schemas  No automatic way to find common data across the enterprise  Metadata harmonization process is manual and time consuming  No automatic way to find gaps between the enterprise logical model and the various enterprise data stores  Building Data Exchanges is Time Consuming  Programs need tools to ease adoption of data standards

© 2009 The MITRE Corporation. All rights reserved 25 Limited Distribution Release Developer View of Gaps II  Progress: NIEM, Data Asset Catalogs and enterprise data models are positive steps  Continuing Needs: –Visibility at data element level, to identify schema reuse and harmonization opportunities –Data asset clustering, to speed definition of new NIEM domains and IEPDs –Matching, to identify correspondences between IEPDs and existing systems –Mapping and CodeGen, to speed generation of data exchange code

© 2009 The MITRE Corporation. All rights reserved 26 Limited Distribution Release OpenII Framework and Tools SchemaStore Schemas ExtensionsMappings Harmony Identify schema correspondences API Transfer data via SQL Generate new domain models / exchange schemas Transfer data via XQuery Cluster schemas around themes Affinity RMap XMap Unity XSD OWL DDL Import / Export Search Examine degree of schema relatedness Overlap Common Ground Toolkit