Enterprise Application Integration Paulo Marques Informatics Engineering Department University of Coimbra 2008/ “EAI in 2 hours!”

2 Motivating Example Warehouse (Amazon)

3 What’s the problem in using... Sockets? RPCs? String prentice = “bulk_store.prenticehall.com"; int port = 1234; try { Socket connection = new Socket(prentice, port); DataOutputStream os = new DataOutputStream(connection.getOutputStream()); os.writeUTF("Unix Network Programming"); os.writeInt(10); connection.close(); } catch (IOException e) { System.out.println("Oops! Could complete the operation. What now?"); }

4 Some problems... Resource localization is not transparent The format used for data transmission is not universal What happens if it’s not possible to open a connection? What happens if the server (or client) crashes during a transaction?

5 Some problems... (2) Any means used for Enterprise Application Integration must address, at least, the following issues: How is Data Represented – Each application has its own data model and information schema. There may be shared concepts between applications but, typically, they are represented in different ways. It’s essential to have ways of mapping and transforming data. Localization of Services – Each application was developed as a “software silo”. Thus, it can unexpectedly change localization. It’s essential to have ways of localizing and remotely accessing applications that are executing autonomously. Time and Synchronicity – The RPC model is synchronous. This means that both parties have to be present when a call is made. Such constraint is not viable when communicating among autonomous independent systems. Typically, it’s necessary to use an asynchronous and robust communication model between systems. Fault Tolerance – Networks are inherently fallible. Simplistic approaches like re-trying operations do not work. It’s necessary to have robust mechanisms for tolerating faults and, in many cases, distributed transactions (ACID or not).

6 Data Representation Ideality, all applications to be integrated would share the same data model, which would give them a unique, well-defined, non ambiguous form of storing and transferring information. In practice, data management and its associated meta-information (schemas) differs radically among systems. Even the storage approach is different (e.g. relational BDs, text files, binary files, Object-Oriented DBs, etc.) From the point of view of EAI, the formats used to share information across system boundaries are much more important than what’s inside. Over the years, different initiatives and middleware for remote invocation tried to standardize different formats to represent information (e.g. RPC XDR, ASN.1, CORBA CDR, Java Serialization Format, etc.) The big companies developed their own format for data representation in business scenarios (purchase orders, inventory, etc.): EDI – Electronic Data Interchange (ANSI X12 standards) XML and its corresponding schemas are becoming the de facto standard for data interchange.

7 Time and Synchronicity The RPC model is typically inadequate for Enterprise Application Integration The calling entity is blocked during the invocation. Invocations can take hours or days to complete. It forces both entities to be active at the same time. Asynchronous solutions are necessary  MOM = Message Oriented Middleware (e.g. IBM MQueue Series)

8 Message Oriented Middleware The communication is done through MESSAGE QUEUES Asynchronous Interaction Who sends the information is not blocked SEND-AND-FORGET The receiving end-point does not have to be active STORE-AND-FORWARD Can be made transactional (but, typically, not ACID...)

9 RPC vs. MOM Bank A Bank B Connect ACK deposit(“John”, 100) NO_ERROR Bank A Bank B DEPOSIT_QUEUE John 100 XML

10 (Large-Scale) Enterprise Application Integration Middleware for EAI Microsoft Biztalk Server BEA Weblogic IBM WebSphere TIBCO, WebMethods, Vitria, etc.

11 A “small” example... Widget-Gadget Corp *Take Orders: Customers can place orders via Web, phone or fax *Process Orders: Processing an order involves multiple steps, including verifying inventory, shipping the goods and invoicing the customer *Check Status: Customers can check the order status Change Address: Customers can use a Web front-end to change their billing and shipping address New Catalog: The suppliers update their catalog periodically. WGRUS needs to update its pricing and availability based in the new catalogs. Announcements: Customers can subscribe to selective announcements from WGRUS. Testing and Monitoring: The operations staff needs to be able to monitor all individual components and the message flow between them

12 Purchase Order...

13 How are the orders processed? Check if the item is in the inventory Check the client’s credit If OK: Ship the goods Send the invoice

14 A Detailed View on processing the order of a new item

15 And what happens if an order has more than one item?

16 High Level Primitives Splitter+Router+Aggregator = Composed Message Processor

17 But, how are the orders correlated?

18 How to know the current status of an order? And on the Point-to-Point Channels?

19 In reality, the system can be simplified... SOA = Service Oriented Architecture * All services available on the ESB are published on a common service directory * All the infrastructure is thought for being “service oriented” Enterprise Service Bus (ESB)

20

21 Legacy Systems... How can legacy systems know where to send messages?

22 Some important questions regarding this example All the system is in inherently asynchronous, interconnecting different applications and processes Legacy systems are interconnected using adaptors The adaptors’ code is manually written by programmers The messages circulating in the system have to be modified at runtime Translated, Enriched, Filtered, Duplicated, etc. Enterprise Service Bus All applications are seen as entities (services) which are able to communicate using a common bus which can be controlled and monitored Service Oriented Architecture All the services in the bus are structured and registered on a common directory service. The services are orquestrated for achieving a common goal, having a corresponding business process behind There’s middleware available for implementing all this Message exchange Process orchestration

23 So, what have we covered in today’s class? Any half-decent approach used for Enterprise Application Integration must address, at least, the following problems: How is Data Represented – Each application has its own data model and information schema. There may be shared concepts between applications but, typically, they are represented in different ways. It’s essential to have ways of mapping and transforming data. Localization of Services – Each application was developed as a “software silo”. Thus, it can unexpectedly change localization. It’s essential to have ways of localizing and remotely accessing applications that are executing autonomously. Time and Synchronicity – The RPC model is synchronous. This means that both parties have to be present when a call is made. Such constraint is not viable when communicating amount autonomous independent systems. Typically, it’s necessary to use an asynchronous and robust communication model between applications. Fault Tolerance – Networks are inherently fallible. Simplistic approaches like re-trying operations do not work. It’s necessary to have robust mechanisms for tolerating faults and, in many cases, distributed transactions (ACID or not).

24 Where to go next? Enterprise Integration Patterns by Gregor Hohpe & Bobby Woolf Introduction & Preface Chapter 1 (all)

25 IMPORTANT NOTICE YOU ARE FREE TO USE THIS MATERIAL FOR YOUR PERSONAL LERNING OR REFERENCE, DISTRIBUTE IT AMONG COLLEGUES OR EVEN USE IT FOR TEACHING CLASSES. YOU MAY EVEN MODIFY IT, INCLUDING MORE INFORMATION OR CORRECTING STANDING ERRORS. THIS RIGHT IS GIVEN TO YOU AS LONG AS YOU KEEP THIS NOTICE AND GIVE PROPER CREDIT TO THE AUTHOR. YOU CANNOT REMOVE THE REFERENCES TO THE AUTHOR OR TO THE INFORMATICS ENGINEERING DEPARTMENT OF THE UNIVERSITY OF COIMBRA. (c) 2007 – Paulo Marques,