1. Enterprise Service Bus Lowering the cost of integration Date: 9/1/2009 Dan McCreary President Dan McCreary & Associates dan@danmccreary.com (952) 931-9198 M D Metadata Solutions

2. M D Copyright 2008 Dan McCreary & Associates2 Agenda/Objective What is an ESB? What is enterprise messaging? Calculating the ROI on enterprise service bus

3. M D Copyright 2008 Dan McCreary & Associates3 Why Build an ESB? Three primary business benefits –Precision –Consistency –Agility Precision Consistency Agility

4. M D Copyright 2008 Dan McCreary & Associates4 Precision Requirements reference Business Capabilities and are defined in business approved terms and data elements. Business terms and data elements are stored in the metadata registry Metrics and business rules reference data elements that are in the registry Shared processes and services use data elements that are in the registry Precision

5. M D Copyright 2008 Dan McCreary & Associates5 Consistency There is a common compensation processing model and each system/product maps into that common model. The processing model references Thrivent Business Capabilities, business terms, data elements, rules and metrics that are in the registry. Everyone (architects, analysts, developers, testers) use the same definitions for business terms, data elements, rules and metrics – all stored in the registry. Consistency

6. M D Copyright 2008 Dan McCreary & Associates6 Agility A single change to elements in the registry allows all models to be updated A single change to models can more seamlessly be reflected in updates to applications and services Changing and testing the models, then transforming the models into executable code allows faster response to change, and higher quality of delivered changes. Agility

7. M D Copyright 2008 Dan McCreary & Associates7 Quality Metrics for an ESB ATAM Output

8. M D Copyright 2008 Dan McCreary & Associates8 The Challenge N applications that need to communicate with each other 1 st application needs N-1 connections 2 nd application needs N-2 connections 100 th system needs 99 new adapters created O(N*N) 100 systems = O(10,000 interfaces) A1 A2 A3 A4 A5 A6 A7 AN

9. M D Copyright 2008 Dan McCreary & Associates9 The Solution Build a central EAI information bus that has “adapters” to each of the applications O(N) adapters A1 A2 A3 A4 A5 A6 A7 AN EAI Bus

10. M D Copyright 2008 Dan McCreary & Associates10 The Challenge How to define an Adapter for each system so that other applications can “subscribe” to its data and process services Make the transport reliable Application EAI Bus Adapter Messaging Protocol

11. M D Copyright 2008 Dan McCreary & Associates11 Incoming! XML Has this web thing gone away yet? web

12. M D Copyright 2008 Dan McCreary & Associates12 Publish/Subscribe Securely Subscriber 1 Publisher 1 Enterprise Service Bus Publisher 2Publisher 3 Subscriber 2Subscriber 3 Metadata Registry

13. M D Copyright 2008 Dan McCreary & Associates13 3 EAI Architectures Presentation Database Method Presentation Database Presentation Method Data

14. M D Copyright 2008 Dan McCreary & Associates14 The Heart of the Enterprise The Metadata Registry A metadata registry is a central location in an organization where metadata definitions are stored and maintained in a controlled method. http://en.wikipedia.org/wiki/Metadata_registry

15. M D Copyright 2008 Dan McCreary & Associates15 Quality How do you measure it? –Quality Metrics How can you put in “benchmarks” to make sure it is improving?

16. M D Copyright 2008 Dan McCreary & Associates16 Reuse Rule #1 If you don’t know about it …you can’t reuse it If we have any asset, how does a person know it exists? What are the corporate repositories of reusable assts? How easy is it to use and be notified if an asset changes?

17. M D Copyright 2008 Dan McCreary & Associates17 Metadata Publishing Workflow Funnel Develop a simple workflow system for publishing data elements Include harvesting areas of simple glossary-of-terms found in documentation, web sites and by using metadata “scrapers” to inventory all columns in relational database systems Get stakeholder teams to “accept” a data elements, review them and take on the data stewardship role for these data elements Commit to change-control only after data elements are marked “approved for publication” by over 50% of the stewardship team Under Review Approved for Publication Glossary Of Terms Initial Draft Metadata Harvesters

18. M D Copyright 2008 Dan McCreary & Associates18 The Garden Metaphor Beautiful gardens don’t just grow by themselves They need to be designed, cultivated and maintained Getting out the weeds is a key part of all beautiful gardens (removing duplicate elements)

19. M D Copyright 2008 Dan McCreary & Associates19 Data Stewardship An organization needs to believe that data and the management of enterprise data is a strategic goal of the enterprise At it core the ongoing data stewardship process in not a function of IT, it is an enterprise value instilled in each business unit We need to find data steward champions in each business unit IT can facilitate the maintenance of data but the business unit needs to be empowered I Love Precise Data Definitions

20. M D Copyright 2008 Dan McCreary & Associates20 Federated Search Search many sources using a single tool

21. M D Copyright 2008 Dan McCreary & Associates21 Search Results Internal Data Standards External Data Standards

22. M D Copyright 2008 Dan McCreary & Associates22 Rules Process Data Rules Can they be separated?

23. M D Copyright 2008 Dan McCreary & Associates23 Rules and Data Rules should always be expressed in terms that are consistent with corporate metadata standards –Tae Sondreal, Business Rules Expert But how do we show this in our POC?

24. M D Copyright 2008 Dan McCreary & Associates24 Assume a Future SOA ESB Each system has adapters that send and receive data in standardized XML formats governed by an enterprise metadata registry Adapter Enterprise Service Bus

25. M D Copyright 2008 Dan McCreary & Associates25 What are SOA-Centric Rules? Rule are used by any business process that executes conditional logic on a fragment of XML data on the Enterprise Service Bus XML File Rule Exception Happy Path Business Event

26. M D Copyright 2008 Dan McCreary & Associates26 Data Type Rules A rule that only looks at a leaf on a and XML tree Example: –Is -1.O3 a valid number? –Is “2008-28-29” a valid date? Rules do not depend on any resources outside of that node Executes quickly using streaming memory “Leaf-level rules” Not applied to branches Use w3c XML Schema datatype standards Most code tables under 1,000 values Patterns validation using regular expressions Enterprise SimpleTypes are stored in the metadata registry with the data element definition scope See w3c XML Schema datatypes

27. M D Copyright 2008 Dan McCreary & Associates27 Structural Rules Rules that express the structure of a document What branches are valid? What branches are required and how many repeating values (cardinality)? Can be expressed by XML Schema constraints Output is a valid/invalid Boolean value for the entire document Executed in a single pass even for gigabyte files No need to keep the entire document in working memory XML Schema.xsd

28. M D Copyright 2008 Dan McCreary & Associates28 Transformation Rules Rules that change an XML document either by adding or changing structure Rules split up documents or combine documents Expressed as XQuery or XSLT Stored in transformation libraries that promote reuse Classified by the Enterprise Integration Patterns (Content Enrichment, Content Filter etc.) ABC C = A + B

29. M D Copyright 2008 Dan McCreary & Associates29 Conditional Rules Rules that can be take every node in the tree into account Expressed as a Boolean (true/false) outcome Used to redirect a document in a workflow Expressed as XPath expressions using Schematron rules file and compiled into an XSLT file XForms application checks for valid data element definitions ABC If C does not equal A + B then report an error. Rule registry Metadata registry

30. M D Copyright 2008 Dan McCreary & Associates30 Service Rules Rules that require sending a subset of the XML data to an external web service Depends on an enterprise resource on a remote server Required consideration of unreliable networks and services Web Service database

31. M D Copyright 2008 Dan McCreary & Associates31 Classification of Rules in the SOA Realm Simple Business Rules –Data type rules (XML Schema data types) –Structural rules (XML Schemas, ER relationships) –Done quickly using streaming technology (SAX) Medium Business Rules –Transformation and Conditional rules –Stated as XSLT or if-then-else expressions –Assume and in-memory model (DOM) Service Rules –Any check that requires data not easily stored in a simple XML Schema or a rules files –Address validation –CIF lookup –Legacy services require web service adapters

32. M D Copyright 2008 Dan McCreary & Associates32 Rules in the Registry We have datatype, structural and services in the registry but we do not yet have a registry of transformation and conditional rules Metadata Registry Property Data Elements Wantlists XML Schema Version Registry SimpleTypes Subset Generation Service Registry

33. M D Copyright 2008 Dan McCreary & Associates33 Rule Registry The Rule Registry would contain rule sets Each rule set would be controlled by a group Rule sets contain many Rules Each Rule contains many And (Or) expressions that are either of type assertion or report Expressions my only reference registered data elements or literal values Rule Registry Rule Set Rule And (Or) Expression contains many reference many Data Element Metadata Registry registered in the Group Controlled by assertionsreport literal

34. M D Copyright 2008 Dan McCreary & Associates34 Scope of Proof-of-Concept We can do an architectural proof-of-concept today as long as we don’t have to deal with network reliability issues right away Network reliability issues need to be considered for production systems Can be simulated with “wrapper” web services

35. M D Copyright 2008 Dan McCreary & Associates35 Rule as a Service Rule Engines can be implemented as services that hangs off of the ESB They have XML/REST interfaces The take XML data in and send XML data out They can live within the eXist XQuery engine or DB2 v9 or MS-SQL 2005 or Oracle 11g XML Schemas define their inputs and outputs Adapter Rule Engine Adapter Enterprise Service Bus xml

36. M D Copyright 2008 Dan McCreary & Associates36 Distributed Computing Issues What if a remote web service is down? Request will time out after timeout interval (by default – 15 seconds) Ideally we would have a “watcher” that would watch over these calls and keep trying until the remote service returns How can this be added to the XQuery environment?

37. M D Copyright 2008 Dan McCreary & Associates37 Reliable Web Services Issues When you call any service on a remote computer it gets transparently put in a local message queue The XQuery waits till the result returns People are notified when messages are not returned with an acceptable timeframe Example WS-Reliable Messaging

38. M D Copyright 2008 Dan McCreary & Associates38 What is Orchestration? The ability for any web service to call other web services Web Service Call Web Service X Call Web Service Y Call Web Service X Request Response Also known as a “composite” Web service

39. M D Copyright 2008 Dan McCreary & Associates39 XQuery XQuery is inherently a choreography language xquery version "1.0"; let $y := doc(‘http://www.example1.com/service-y.xq’)/my-xpath-into-y let $z := doc(‘http://www.example2.com/service-z.xq’)/my-xpath-into-z return {$y} {$z} x.xq

40. M D Copyright 2008 Dan McCreary & Associates40 Strategy Continue to build an infrastructure that is consistent with the metadata registry –Selection list are context sensitive Use simple XML-centric standards (like XPath) –Rules expressed in terms of metadata Leverage “as-you-type” user interfaces (see suggest demo)

41. M D Copyright 2008 Dan McCreary & Associates41 Thank You! Please contact me for more information: Native XML Databases Metadata Management Metadata Registries Service Oriented Architectures Business Intelligence and Data Warehouse Semantic Web Dan McCreary, President Dan McCreary & Associates Metadata Strategy Development dan@danmccreary.com (952) 931-9198