SOA Data Integration - The Unsolved, Unspoken Problem.

Slides:



Advertisements
Similar presentations
Web Service Architecture
Advertisements

Future of NIEM Tools Delivery Public Sector NIEM Team, July 2012 Futureof NIEM Tools.
Web Service Ahmed Gamal Ahmed Nile University Bioinformatics Group
Snejina Lazarova Senior QA Engineer, Team Lead CRMTeam Dimo Mitev Senior QA Engineer, Team Lead SystemIntegrationTeam Telerik QA Academy SOAP-based Web.
NIEM, CAM and the 7 “D’s” David Webber - Public Sector NIEM Team, November 2011 NIEM Test Model Data Deploy Requirements Build Exchange Generate Dictionary.
Semantics and Information Exchanges Overview – Public Sector NIEM Team, June 2011 CAM Test Model Data Deploy Requirements Build Exchange Generate Dictionary.
Enterprise Integration Architecture IPMA Professional Development Seminar June 29, 2006 Scott Came Director, Enterprise Architecture Program Washington.
1 Introduction to XML. XML eXtensible implies that users define tag content Markup implies it is a coded document Language implies it is a metalanguage.
SOA with Progress Philipp Walther Consultant. © 2007 Progress Software Corporation2 Agenda  SOA  Enterprise Service Bus (ESB)  The Progress SOA Portfolio.
A New Computing Paradigm. Overview of Web Services Over 66 percent of respondents to a 2001 InfoWorld magazine poll agreed that "Web services are likely.
B2B e-commerce standards for document exchange In350: week 13: Nov. 19,2001 Judith A. Molka-Danielsen.
Peoplesoft: Building and Consuming Web Services
XSD and jCAM tutorial - Leveraging Open Standards - XSD ingesting - Interoperability for SOA / WSDL - Exchange Management David RR Webber Chair OASIS CAM.
1 1 Roadmap to an IEPD What do developers need to do?
Open Data API delivery “Open-XDX” David Webber, Information Architect, Oracle Public Sector Open Data Exchange October, 2012.
IBM Corporate User Technologies | November 2004 | © 2004 IBM Corporation An Introduction to Darwin Information Typing Architecture: DITA Presented by Dave.
Just a collection of WS diagrams… food for thought Dave Hollander.
SOA – Development Organization Yogish Pai. 2 IT organization are structured to meet the business needs LOB-IT Aligned to a particular business unit for.
SOA, BPM, BPEL, jBPM.
THE NEXT STEP IN WEB SERVICES By Francisco Curbera,… Memtimin MAHMUT 2012.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 1 Quick Tutorial – Part 1 Using Oracle BPM with Open Data Web Services David Webber.
James Cabral, David Webber, Farrukh Najmi, July 2012.
Framework: ISA-95 WG We are here User cases Studies
Copyright OASIS, 2005 / 2007 CAM Technology Introduction David Webber Chair OASIS CAM TC Presentation February 9 th, 2007 Bethesda MD.
Copyright OASIS, 2005 / 2007 Scalable PESC transaction handling using OASIS CAM technology David Webber Chair OASIS CAM TC Presentation.
Using the Universal Business Language for Internet Paperless Trading by Tim McGrath APEC Symposium on ebXML Bangkok, Thailand, July
Copyright OASIS, 2005 / 2006 Standards and Open Source Tools for Implementing Loosely Coupled SOA XML, metadata techniques and micro-services tools SOA.
EbXML Technical Architecture From: ebXML Technical Architecture Specification v1.04,
Web Services Architecture1 - Deepti Agarwal. Web Services Architecture2 The Definition.. A Web service is a software system identified by a URI, whose.
NIEM Blue Team Presentation April 20, 2010 Phil Letowt, Mini Kanwal, Ken Sall, David Webber ICE OCIO / Task ASAS ICE Information Exchange Reuse with NIEM.
What is Service Oriented Architecture ? CS409 Application Services Even Semester 2007.
1 Quick Guide to CAM Dictionaries Building and using Canonical XML dictionaries for CAM (CAM – Content Assembly Mechanism Specification) Author: David.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 1 Quick Tutorial – Part 2 Open Data Web Services for Oracle BPM August, 2013 Forms.
International Telecommunication Union Geneva, 9(pm)-10 February 2009 ITU-T Security Standardization on Mobile Web Services Lee, Jae Seung Special Fellow,
David Webber, NIEM Team, Oracle Public Sector Rapid NIEM XML Exchange Design, Semantics and UML Models NIEM Test Model Data Deploy Requirements Build Exchange.
Copyright © 2004 by The Web Services Interoperability Organization (WS-I). All Rights Reserved 1 Interoperability: Ensuring the Success of Web Services.
OASIS Week of ebXML Standards Webinars June 4 – June 7, 2007.
Interfacing Registry Systems December 2000.
Web Services based e-Commerce System Sandy Liu Jodrey School of Computer Science Acadia University July, 2002.
Copyright OASIS, 2005 / 2007 Slaying the Complexity Monster David Webber Chair OASIS CAM TC Presentation January 24 th, 2007 Reston VA.
Development Process and Testing Tools for Content Standards OASIS Symposium: The Meaning of Interoperability May 9, 2006 Simon Frechette, NIST.
Using XML for Business and Life May 2006 David RR Webber Technology and Standards in Higher Education.
XML Web Services Architecture Siddharth Ruchandani CS 6362 – SW Architecture & Design Summer /11/05.
Middleware for FIs Apeego House 4B, Tardeo Rd. Mumbai Tel: Fax:
EbXML Technical Architecture From: ebXML Technical Architecture Specification v1.04,
Challenges of Information Collaboration Roles, Methods, Responsibilities and Context David RR Webber SOA Architect IntegrityOne Partners Ph: (703)
1 1 CAM Toolkit for NIEM IEPD Development Phil Letowt – DHS/ICE David Webber – ICE Data Architect.
Copyright OASIS, 2005 / 2006 Presenter Chair OASIS CAM TC Presentation January 14 th 2006 Washington DC Introducing CAM V1.5 Technology.
SOA Data Integration - The Unsolved, Unspoken Problem.
FDT Foil no 1 On Methodology from Domain to System Descriptions by Rolv Bræk NTNU Workshop on Philosophy and Applicablitiy of Formal Languages Geneve 15.
OASIS CAM – Technology Introduction Business Rules Exchange Workgroup Call October 30 th, 2007.
Leveraging SET, OWL, CAM and Dictionary based tools to enabled automated cross-dictionary domain translations David Webber OASIS SET TC / CAM TC (with.
U NITED N ATIONS C ENTRE F OR T RADE F ACILITATION A ND E LECTRONIC B USINESS Under the auspices of United Nations Economic Commission for Europe UN/CEFACT.
Kemal Baykal Rasim Ismayilov
SOAP-based Web Services Telerik Software Academy Software Quality Assurance.
® IBM Software Group © 2004 IBM Corporation Developing an SOA with RUP and UML 2.0 Giles Davies.
CASE (Computer-Aided Software Engineering) Tools Software that is used to support software process activities. Provides software process support by:- –
Dictionary based interchanges for iSURF -An Interoperability Service Utility for Collaborative Supply Chain Planning across Multiple Domains David Webber.
OASIS CAM Technology - Brief Introduction January, 2008 Creating EDXL Use Pattern Templates David Webber Chair OASIS CAM TC
Silverstein Group Presenter Moshe Silverstein A Content Assembly Mechanism Technology Overview Context & Integration A Content Assembly Mechanism Technology.
Copyright 2007, Information Builders. Slide 1 iWay Web Services and WebFOCUS Consumption Michael Florkowski Information Builders.
Software Architecture Patterns (3) Service Oriented & Web Oriented Architecture source: microsoft.
By Jeremy Burdette & Daniel Gottlieb. It is an architecture It is not a technology May not fit all businesses “Service” doesn’t mean Web Service It is.
Service Oriented Architecture (SOA) Prof. Wenwen Li School of Geographical Sciences and Urban Planning 5644 Coor Hall
CIM Modeling for E&U - (Short Version)
Sabri Kızanlık Ural Emekçi
SOA (Service Oriented Architecture)
XML Based Interoperability Components
The Re3gistry software and the INSPIRE Registry
Service Oriented Architecture (SOA)
Presentation transcript:

SOA Data Integration - The Unsolved, Unspoken Problem

David Webber SOA Architect Chair OASIS CAM TC November, 2007

Metatopia Conference Washington DC 3 Agenda  SOA information integration needs  Selection of illustrative field examples  Solution and technology approach  Demonstration  Summary / Next Steps

Metatopia Conference Washington DC 4 Agenda  SOA information integration needs  Selection of illustrative field examples  Solution and technology approach  Demonstration  Summary / Next Steps

Metatopia Conference Washington DC 5 CEO - Enterprise Strategic Vision Improve Program Integrity Improve Products & Services Reduce Costs Improve Human Capital Management Modernize & Integrate Systems - Reconcile current business demands, economics and future business strategies with present state of legacy systems, technology and human resource skills -Strengthen financial management and internal controls -Provide effective oversight -Ensure the accuracy of all data -Implement an integrated set of information systems - Enable cost reduction and re- engineered business processes - Establish metrics to measure performance and productivity trends across the enterprise over time -Ensure enterprise has the right people in the right jobs with the right skill sets -Create and foster an environment for collaborative problem solving and decision making -Listen to the customer to understand and prioritize needs and/or desired products and solutions -Inform business solutions by incorporating industry best practices

Metatopia Conference Washington DC 6 Some SOA claims and justifications Service Oriented Architecture means: Reusable components exposed through services shared with business partners and enterprise systems enabling low- cost, low-impact incremental and organic adaptation

Metatopia Conference Washington DC 7 SOA Characteristics 7 Modular Approach Questions? Supports Strategic Initiatives and Goals Swappable Agile Components Open Standards - SOA implements the functionality of the system using formal open standards for information exchange, transport and delivery. - SOA is an attempt to modularize large complex systems in such a way that they are composed of independent software components that offer services to one another through well- defined interfaces. - This supports the notion that any of the components could be ‘swapped’ for a better one, when it becomes available. - Reconcile current business demands, economics and future business strategies with present state of legacy systems, technology and human resource skills - How can we swap things if the information interfaces are not clearly defined and easily verifiable? - How can we leverage inconsistent legacy interfaces?

Metatopia Conference Washington DC 8 What are SOA Services? The ability to orchestrate service logic with business processes Not just static point information service sinks or sources Ability to associate behavior with information exchanges –Support for context and role –Support for business process actions and signals –Support for access profiles and partner agreements –Support for dynamic structure queries and responses Use of open public standards to ensure predictable access to services Independent of platform deployment and legacy systems constraints

Metatopia Conference Washington DC 9 What makes something Agile? Re-usable methods that can be applied to many areas –Based on open standards and approach; not proprietary Context and role driven and aware –This allows tailoring to specific profiles and use pattern templates dynamically Self-adaptive –When requirements change can be adjusted on-the-fly in real time Fault tolerant and not brittle –Ability to ignore non-critical interchange items and especially not to fail for trivial reasons or slight version nuances –Leveraging XML capabilities to make self-describing transactions possible rather than static fixed legacy exchanges Able to support new uses without extensive re-programming –Usage patterns set via external configuration allowing broad but controlled uses

Metatopia Conference Washington DC 10 How to Support Discovery? As an organization develops extended numbers of services the need to catalogue and share these collaborative increases. Services need to: –Describe their means of connection and interfacing –Provide a reference to the domain of use –Be associated with a catalogue and classification system The most powerful discovery mechanism is not simple text search but context and profile agreement parameters linked to templates Associate with a domain vocabulary to facilitate shared meaning and core components Align to interoperability of information exchanges and agile concepts Bottom line – answer the questions: –Can I use this? (context) –How can I set it up? (templates) –Will it work & connect with my existing services (behaviors)?

Metatopia Conference Washington DC 11 Defining Information Services Create Samples Rules Editor XML Analyst Templates Samples Develop Verify Structure Rules Context Vocabulary 3 2 Rules html XML Results html Publish 1 Detail Use Rules Test Rules on Samples 4 Share Results Enabling Agile Information Exchanges Structure + Vocabulary Rules + Context Templates Verify Template Outcomes

Metatopia Conference Washington DC 12 Partner Conformance Test and Certification Validate Templates Report Partner Creates XML Results Test Pass / Fail Templates XML html Structure Rules Context Publish

Metatopia Conference Washington DC 13 Facilitation Needs For business communities: –Ability to create sharable templates for communities of practice that need consistent XML transaction handling definitions that are open and public. For business data analysts: –Printable rule documentation support and features. –Example domain templates within communities of practice –Genericode codelists implementation. For programmers: –XML content manipulation support and rules ( xslt, XPath …) –Web services and SOA support

Metatopia Conference Washington DC 14 Select your technology set Many technologies are available, including application servers, distributed objects, and integration servers. Agile use of XML a prerequisite. The choice of technology will likely be a mix of components that together meet your SOA needs. –Avoid vendor lock-in –Prefer Open Standards and Open Source solutions –Ensure proper use of layers to separate and manage complexity –Select simple components that integrate easily

Metatopia Conference Washington DC 15 Basic WSDL Web Service Transport Security Internet Infrastructure Delivery Data Services Process Requirements UML diagrams Application Logic Packaging / Encoding XML Validation XSD Query / Response WSDL 1 1 SOAP / http / https

Metatopia Conference Washington DC 16 Extended Data ServicesTransport Security Internet Infrastructure Delivery Process Query / Response WSDL 1 1 SOAP / http / https Data Services 4 4 Vocabulary / Semantics Packaging / Encoding XML / edi Validation / Assembly Mapping / Transform XSD, CAM, Schematron Business Model Service Agreement Context / Roles Business Processes Description Msg Exchange Profile State & Context Addressing / Envelope Transactions Coordination Push / Pull

Metatopia Conference Washington DC 17 SOA OrchestrationTransport Security Internet Infrastructure Delivery Data Services Process Registry Business Model Service Agreement Context / Roles Business Processes Description Msg Exchange Profile Search / Classification Ownership Profiles Requests to Consume Service Definitions Management / Versioning Vocabulary / Semantics Discovery Vocabulary / Semantics Packaging / Encoding XML / edi Validation / Assembly Mapping / Transform XSD, CAM, Schematron State & Context Addressing / Envelope Transactions Coordination Push / Pull Other protocols Other services Security Credentials Quality of Service Reliable Messaging REST WSDL B2BB2B WSIWSI 1 1 SOAP / http / https

Metatopia Conference Washington DC 18 Agenda  SOA information integration needs  Selection of illustrative field examples  Solution and technology approach  Demonstration  Summary / Next Steps

Metatopia Conference Washington DC 19 Domain Examples DOJ NIEM/LEX Fannie Mae EDI-esque XML PESC Dictionary and schema UBL schema and CCTS -Universal Business Language (UBL) -OASIS standard -Uses UML models and CCTS approach -Derived from xCBL and simplEDI approach -Supply chain schema with joint initiative EU / Asia / US -Developed by GTRI for DOJ community -Vocabulary based -LEXS schema built using NIEM vocabulary -Law enforcement and court applications -Developed for Education Department -form-style XML -Older XSD techniques for simple flat schema model -No re-use of common structures -Student loans and transcripts -Developed by BAH and NG for form-based application submissions -Translate previous paper form into multiple schema sections -Massive schema with extended namespaces -Set of functional transactions and configuration templates for managing and processing elections -Common set of components and vocabulary -Widely differing use patterns and items by country localization OASIS EML Grants.gov form based schema -Transactions replacing EDI -Overloaded reuse of structure components -Financial reporting data -Financial transaction data -Extended use of code sets to label transaction content / purpose

Metatopia Conference Washington DC 20 Challenges for Users and Implementers DOJ NIEM/LEX Fannie Mae EDI-esque XML PESC Dictionary and schema UBL schema and CCTS -Complex schemas that meet extended use cases -Less obvious how to create sparse subset for typical everyday uses (in State / in Country) -Verbose syntax and structures from CCTS models -Complex schema with use model that does not necessarily map to domain information stores and search mechanisms -Platform transport formats determine transaction packaging - Limited interoperability of search methods -Implementers have created baseline samples for typical student use cases -Backend applications developed by solution providers -Interoperability verified between providers solutions -Extended form field edits and code lists -Large structure and verbose syntax -Excessive use of namespaces -Extensive cross- field and form relations -Related content edits (PDF documents) -Documenting widely differing use patterns and items by country localization -Understanding how to use flexible structure constructs for their own applications -Verification and certification of exact usage and computations OASIS EML Grants.gov form based schema -Simple structures with need to show transaction use patterns -Extensive use of codes -Calculations and numbers hard to validate using schema alone -Extensive cross-field relations

Metatopia Conference Washington DC 21 Solution Metrics Can I create a standard simple open format to describe my message structures and data content rules? Can my partners validate their transactions in test BEFORE they send them? How do people know what I will send them? I want something that’s simple and standards based – leverages existing XML components Can I generate HTML documentation that is readable by business analysts?

Metatopia Conference Washington DC 22 Why not use XSD? Today’s XML schemas have complex structures with no context awareness + no cross field association rules + no dynamic lookups The XSD provides a model of ALL possible structure instances – not the particular instance Excessive use of namespaces make for fragile XML transaction handling Generating valid sparse transaction layout is tough Documentation diagrams hard to read How to create simple re-usable templates?

Metatopia Conference Washington DC 23 XSD – PESC “GPA” model example EVERYTHING is optional! So what do I REALLY send to you?

Metatopia Conference Washington DC 24 Conceptual View of PESC transactions Transaction Templates Structure Rules Context 3 2 Use Rules html XML Reports html Publish 1 Common Details Rules 4 Share Details Address Course Contact Student Loan Sponsor Transcript Content Rules Lookup Values Context Versioning

Metatopia Conference Washington DC 25 What about these needs? Versioning Content consistency Use rules Codelists Associations (what uses which?) Guidelines Providing local contextual validation services

Metatopia Conference Washington DC 26 Versioning Challenges If the schema version changes – how to ensure it does not break our in place validations? How to rapidly adapt to rule changes in a production environment? How to develop user context driven deep version control and re-use of sub-components? Enhance and automate Test release cycle by improving transparency for bug fix process and expose change deltas to speed testing process? Support for regression testing?

Metatopia Conference Washington DC 27 Agenda  SOA information integration needs  Selection of illustrative field examples  Solution and technology approach  Demonstration  Summary / Next Steps

Metatopia Conference Washington DC 28 History and Status of CAM work OASIS technical committee Five years of combined development in UN/CEFACT and OASIS OASIS v1.1 public standard and specification jCAM open source implementation in Java Creating simple XML-scripted open standard mechanisms for XML transaction assembly and processing Re-use: leverages XPath and xslt, and extends schematron approach Developing templates for common industry formats

Metatopia Conference Washington DC 29 CAM Process Architecture CAMCAM XML Parser / DOM XML Parser / DOM Built-inFunctionsBuilt-inFunctions XPathhandlerXPathhandler SQLpersistenceSQLpersistence TermsRegistryTermsRegistry Rule Engine Post-Processing / Errors Post-Processing XML-awareXML-aware EXTENSIONS

Metatopia Conference Washington DC 30 jCAM Functional Components CAMCAM XML Parser / DOM XML Parser / DOM Built-inFunctionsBuilt-inFunctions XPathhandlerXPathhandler SQLpersistenceSQLpersistence TermsRegistryTermsRegistry Rule Engine EXTENSIONS jaxen xerces e.g. JRules, Others… Saxon - xslt planned using AJAX / ebXML Post-Processing / Errors Post-Processing XML Data Mapping XML-awareXML-aware cam XX

Metatopia Conference Washington DC 31 Deployment Options B2B Web services jCAM engine jCAM engine XML Templates XML Message System XML Java API Receive Process Validate Server Templates Request XML html Response Java API Process Structure Rules Context Standalone EDITOR Template XML html Report

Metatopia Conference Washington DC 32 How does CAM work? CAM uses WYSIWYG approach to XML Starting with your XML sample – creates structure template from that + default data content model Next – add your structure use rules – optional / repeatable, date fields, allowed values, lookups Then make context business rules – cross field use rules, exclude, include, variables Save template – run against samples Eclipse editor tool makes this all easy to do! Deploy to production using jCAM processor

Metatopia Conference Washington DC 33 Eclipse CAM Editor Available structures Structure Rule Viewer Rule Details Results Viewer Validation Process

Metatopia Conference Washington DC 34 Agenda  SOA information integration needs  Selection of illustrative field examples  Solution and technology approach  Demonstration  Summary / Next Steps

Metatopia Conference Washington DC 35 Using jCAM : Start with sample XML Use Eclipse template editor Load XML, generate CAM Enhance base template Test, refine and deploy Generate documentation Deliver business solution Build Simple Build Simple Template Build Simple Build Simple Template Extending Template Extending Template Verify Results Verify Results Verify Results Verify Results Simple XML instance XML instance Simple XML instance XML instance Document Rule Details Rule Details Document Rule Details Rule Details

Metatopia Conference Washington DC 36 Summary of CAM template features Ability to use to modularize complex content Use of XPath references enables agile rule handling Full contextual support including variables, rules and context sections Code lists can be externally built and then referenced Extensions section provides customization options Integration with xslt provides extended results handling and error reporting capabilities Template representation model and XML enables extended post-processing and documentation options

Metatopia Conference Washington DC 37 CAM Functions Summary excludeAttribute()startBlock()useElementByID()useAttribute() excludeElement()endBlock()useTreeByID()useChoice() excludeTree()checkCondition()setDefault()useElement() makeOptional()makeRecursive()restrictValuesByUID()useTree() makeMandatory()setUID()orderChildren()useAttributeByID() makeRepeatable()restrictValues()setRequired()useChoiceByID() setChoice() setId() setLength() setNumberRange() setLimit() setValue() setDateMask() setStringMask() setNumberMask() allowNulls() datatype() or setDataType() <as:constraint = “ action="restrictValues(//shipTo/state,'WA')"> <as:documentation type="documentation">Can only ship item to Washington State <as:constraint condition="$QuickBooks = true“ action="excludeElement(//Item/comment)" /> Example of Rules Use

Metatopia Conference Washington DC 38 Interactive Documentation (iDoc) CAM Template XSLT iDoc wiki HTML

Metatopia Conference Washington DC 39 Agenda  SOA information integration needs  Selection of illustrative field examples  Solution and technology approach  Demonstration  Summary / Next Steps

Metatopia Conference Washington DC 40 Value Proposition Making XML transaction handling simpler and predictable Extends and clarifies your existing XSD schema structures Quick and easy rule building from sample XML transaction Enabling more robust fault tolerant processing + versioning Providing open sharable templates and documentation Re-use easier through support for includable components Ability to integrate to business processes and context Open source, open public standard toolkit – editor + engine

Metatopia Conference Washington DC 41 What’s Next / Call to Action Develop template sets for your business domain Integrate into your messaging exchanges –OrionSMG supports jCAM –Web service test-bed Publish to registry to facilitate adoption –Create document templates to generate registry content / guidelines Use Wiki / SourceForge to facilitate sharing

Metatopia Conference Washington DC 42 Questions?

Metatopia Conference Washington DC 43 camprocessor.sourceforge.net (JRules) docs.oasis-open.org/cam Resources:

Metatopia Conference Washington DC 44 A special mention for our contributors to the CAM and jCAM work: UK- Martin Roberts and team from BTplc US- Sidhartha Nagolu from AC-Tech / NIH Credits: