Enterprise Application Integration An architecture for integration

Introduction EAI was an approach to integration of applications first developed in the late 1990s. Pre-dated widespread use of J2EE, XML and Web Services. EAI currently has become subsumed into Service Oriented Architecture software stacks However, the need for the underlying capabilities have not changed. 2

3 Internet Diverse systems and channels Example of a banking environment

4 Purchased Application Package(s) – ERP etc. New Web- based application(s) Legacy Application System(s) Program IT reality – Islands of Computing Enterprise IT reflects the structure and history of each enterprise not the business processes required today Departments have their own IT systems Legacy systems are left in place and new systems built separately.

5 Program Load Program Extract Program Extract Program Load Program Extract Program Load Program Screen Scrape Database Replicator Down load File Down load File Trans action File Message Queue Down load File Trans action File Message Queue Application System B Application System A Application System C Tactical approach to integrating the Business Each requirement is addressed with a point to point solution Typically consisting of a data transfer mechanism and a format converter. Increasing levels of supplier/client integration points further complicates the situation.

6 Strengths Focused on getting the job done Each individual solution can be created quickly Technical Weaknesses Numerous Point-to-Point Interfaces Inconsistent Business Processes Inconsistent Qualities of Service Business Weaknesses Cost of maintenance Requirement to retain knowledge of the solution in-house. Potential security or loss of service/process failure due to inconsistencies. Strengths and weaknesses of tactical approach

7 Enterprise integration requirement Unrestricted sharing of data and business processes throughout the networked applications or data sources in an organization Extend to customers, suppliers and regulators The Linking of Data, Business Processes, Applications to automate business processes While ensuring that there are consistent qualities of service (security, reliability etc) Reduce the on-going cost of maintenance and reduce the cost of rolling out new systems.

8 Solution: Enterprise Application Integration (EAI) Devised ways to efficiently reuse legacy and existing apps and add new ones EAI products are built on combination of technologies JEE XML (eXtensible Markup Language) for enterprise-wide content and data distribution using common DBs and Data Standards Message queuing Business Process Orchestration engine Sometimes Web Services

9 Layers of an EAI Stack EAI Data Transformation Business Activity Monitoring Message Storage & Routing Business Process Orchestration Adapter Provides real-time and historical data on performance of processes and assists in making decisions. Manages and tracks business transactions that might span multiple systems and last minutes to days. Ensures the data is the correct format for delivery to The next system. Provides “open” connectivity into data sources while allowing filtering and transformations of data. Ensures the reliability of data delivery across the Enterprise or between systems.

10 Adapters EAI Data Transformation Message Storage & Routing Business Process Orchestration Also known as “Connectors”, “Translators”, “Wrappers” or “Bridges” Provide seamless connectivity to the underlying application or data Convert information and events into data that can be utilized by the infrastructure Two way translation between the proprietary format and the common format Many adapters available off the shelf From EAI Software vendors such as Tibco, or application software vendors such as SAP. JEE architecture provides JCA Adapter Business Activity Monitoring

11 Reliable messaging Products that connect applications running on different systems by sending and receiving application data as messages JEE architecture provides JMS interface Message storage Central repository for temporary storage of transactions until they can be delivered Also called “Message Warehousing” Message queuing and routing Asynchronous communications style / Publish Subscribe Set of tools that route messages between sources and consumers based on pre- defined business rules Message storage and routing EAI Data Transformation Business Process Orchestration Adapter Message Storage & Routing Business Activity Monitoring

12 Data Transformation Transformation to and from formats used by Different Systems Handling simple or complex data structures Typically representing data using XML standards such as XQuery and XSLT GUI Configuration tools to help define the transformations Data Transformation EAI Data Transformation Business Process Orchestration Adapter Message Storage & Routing Business Activity Monitoring

13 Data Transformation EAI typically converts the source data into an intermediate format and then convert it into the target format Intermediate format is called the canonical format. SourceTarget F. name: Ronan L. name: Bradley Affiliation: DIT, Dublin, Ireland Name: Ronan Bradley Company: DIT City: Dublin Country: Ireland Intermediate or canonical format

14 Issues in data transformation Challenge 1: Coping with software from different vendors A mostly solved problem (Oracle vs. DB2 vs. SQL Server) Challenge 2: Coping with different formats A solved problem: Relational vs. XML vs. ISAM Challenge 3: Coping with different data models Solvable for the smaller scale cases one person understands all the schemas As the complexity and scale of the data models increases becomes harder and harder Theoretically possible but lots of practical problems Challenge 4: Coping with data from new sources such as new systems, suppliers, customers Lots of unsolved problems

15 EAI: Building a canonical data model Create a common data model including a set of terms for the concepts in the domain of the data sources being integrated e.g., Employee, Customer, Patient, weight, height, bodyTemperature, … EAI solutions do this within their own domain (canonical format) XML Ontologies attempt to do it for any domain required in a self- contained way but are very hard to build Mappings relate data items in data sources to terms in data model Background knowledge about terms essential for transformations e.g., Employee subClassOf Person, 2 people with the same last name, first name and street address are likely to be the same,

16 Mapping the formats Using the canonical data model, each message format can be mapped into and out of that format. Requires many potentially complex mapping definitions.

17 Process Modeling Often GUI based Describing the flow of information in the context of business processes Using the input/output of processes as the Integration points between diverse business processes Process Brokering Execution of discrete steps within a business process Ability to recover from failed steps Workflow engine Process Management Monitor business processes Correlate metrics to specific business process steps Business Process Orchestration EAI Data Transformation Business Process Orchestration Adapter Message Storage & Routing Business Activity Monitoring

18 Business Activity Monitoring EAI Data Transformation Business Process Orchestration Adapter Message Storage & Routing Business Activity Monitoring is using your data assets to make better decisions BAM is used to gather information about what is occurring in the EAI deployment Status of current processes Identification of problem areas E.g. Spotting applications that are not responding Monitoring of unusual activity E.g. Unusually large orders The collected information is displayed for operators or managers to diagnose and determine the solution Business Activity Monitoring

19 Potential EAI benefits Lower development costs Integration becomes simpler because new requirements can be addressed within deployed framework Lower opportunity costs Integration is done more quickly corresponding cost savings reachieved sooner Lower maintenance effort A single architecture to maintain – with consistent mechanisms for each integration and quality of service

20 Potential EAI Challenges Hub and spoke architecture concentrates all of the processing into a single server/cluster. Often became hard to maintain and evolve efficiently Hard to extend to integrate 3 rd parties on other technology platforms The canonical data model introduces an intermediary step Added complexity and additional processing effort EAI products typified by Heavy customisation required to implement the solution Lock-In: Often built using proprietary technology and required specialist skills Lack of flexibility: Hard to extend or to integrate with other EAI products! Requires organisation to be EAI ready

What is EAI readiness? Much of the challenge associated with adopting EAI is not related to the technology. EAI readiness is a state of business maturity in which its systems, processes, employees, structure, and culture are prepared for the successful introduction of both EAI technology and the business disciplines that drive its introduction. All five of these factors must be properly situated and aligned before an EAI implementation can succeed. Processes TechnologyEmployees Culture Structure ©2001, IT Catalysts, Inc.

Process readiness Business strategic goals for project (i.e. process efficiency, customer satisfaction, supply-chain optimization) clearly defined and agreed upon by company executive leadership. Company makes use of mature methodologies that design integrated business change. Desired business processes designed, documented, and valuated. Gap between current and desired business processes well understood.

Employee readiness Staff accustomed to using information technology in performance of day-to-day work. Staff accustomed to adaptation of new roles, responsibilities, processes, and knowledge … to business change. Staff have broad understanding of the business and their contribution to it. Executives understand the role of business sponsorship, and the nature of investing time, staff and budget into the business infrastructure in order to reap future returns.

Technology readiness No systems to be integrated using EAI tools have critical business logic not in the application itself This can happen if the logic required is in a batch programs or reports which are not part of the actual application and hence cannot be integrated with. No systems to be integrated using EAI tools will be retired/ decommissioning soon. There is no point integrating with a system which will soon be removed Most of the business functionality required in process designs exists in one or more legacy systems. Applications portfolio has been evaluated for semantic compatibility. Check it is possible to transform from 1 data model to the other Implementation of EAI not defined as the purpose of the effort. The project should be about the business objective, not about the technology

Structural readiness Keep it simple by ensuring that the EAI system involves as few departments/organisations as possible. Each organisation involved adds to the complexity of the project as more people have to be engaged with and there are potentially greater variety of technologies and architectures. Stakeholder analysis performed to determine overall potential for business resistance to the planned change. EAI projects change the way that each department/organisation works. Therefore, EAI projects need the cooperation of each department/organisation. Business sponsor established and ready to provide leadership to the business change.

Cultural readiness Corporate culture embraces the importance of well-defined business processes. Staff accustomed to adaptation of new roles, responsibilities, processes, and knowledge … to business change. Individuals and departments willing to change the way the applications and data is defined.