Choosing an application integration strategy Anko Duizer Class-A

Overview Why integration? What is integration? Challenges Common scenario’s Which technologies does Microsoft offer? Messaging Design considerations A different viewpoint: The Ring(s) The Matrix

Why? Enterprises are typically comprised of hundreds applications –Custom built, acquired by a third party, legacy systems Reasons –Creating a single, big application to run a complete business is impossible –Flexibility to select “the best” –Use a collaborative partner model

Those who can not remember the past are condemned to repeat it!

The current situation F B E A D C

What? Connecting –Computer systems –Companies –People Application integration is the secure and orchestrated sharing of processes and/ or data between applications

Integration scenarios Information portals Data replication Shared business functions Service-oriented architectures Distributed business processes Business-to-business integration

Basic challenges Networks are unreliable Networks are slow Any two applications are different Change is inevitable Organizational problems

Integration approaches Simple association Central hub The big database approach Message bus Service integration Hybrid

Simple association UI BO Data Application UI BO Data Application Endpoint

Central hub UI BO Data Application UI BO Data Application Endpoint ERP System

The big database approach UI BO Data Application UI BO Application

Message bus UI BO Data Application UI BO Data Application Endpoint Message bus

Service integration (1) UI BO Data Application UI BO Data Application Endpoint Service

Service integration (2) Application Endpoint Service Endpoint Application Endpoint Application Endpoint Application

Microsoft technology BizTalk Server 2004 Web Services MSMQ DTS Indigo (not available yet) SQL Service Broker (not available yet)

BizTalk Server 2004 action orchestration designer biztalk editor biztalk mapper messaging management developmentoperation management administration tracking messaging services orchestration services config tracking port pipeline designer applications monitoring Business rules

Web Services Host Component Client Application Host Page Browser XML over HTTP HTTP request.NET COM EJB Perl … listener proxy Internet

MSMQ Queue msg Sender Receiver

DTS OLE DB ODBC Fixed field ASCII delimited Source OLE DB ODBC Fixed field ASCII delimited Repl. publication Destination DTS Data Pump In Out ActiveX Script Copy Trim String … Custom Transforms

Indigo Port Service Message Channel Service Channel

SQL service broker SQL Server objectDescription Message TypeDefines valid messages for exchange between services ContractSpecifies type of messages and their direction in a conversation, initiator or target QueueStores messages before sending and after receipt as result set Service ProgramThe part of a service broker application that reads messages from a queue and processes them. ServiceAddressable endpoint for service communication msg Service Contract Service program Queue Message type

Messaging Loosely coupled Communicating asynchronously Reliable Makes the messaging responsible for transferring data from one application to another

Sync or async? time Process A Process B blocked time Process A Process B

Basic messaging concepts Application Channel Endpoint Message Translation Routing

Messaging patterns Y X Y X 1) Fire and Forget No Response required 2) Request / Reply System makes request and gets response 3) Needs ability to cache response from multiple requests to respond to one message Y X EAI B X 4A/B) Multiple Applications needed to satisfy request 4A: XREF Look-up 4B: XREF Update AC XREF EAI A X 5) Publish Subscribe A, B and C subscribe to message B C EAI 6A) Needs data from Z to be able to update Y Y X EAI Z

The message is the message!

Main design considerations Timing/ Latency Identification –Industry keys, Internal keys, Chained keys & Matching algorithms Sharing entities, activities, processes and utilities Designing endpoints Message transformation

Main design considerations Routing Security System management ACID Transactions Service-oriented architectures

Shared data ownership E-Commerce Inventory E-Commerce

Choosing

To EAI or to ETL? Data (ETL) Data Transformation Services Messages (EAI) BizTalk Server 2004 Unit of modeling “Package” – an XML definition of “workflow” and data flow Schema that capture structure/ Contracts that capture behavior Unit of interaction “Synchronous” DTS pipeline for data movement Asynchronous message passing Instancing DTS Service controls instancesWeak/opaque URI-based references Extensibility Scripts, SQL, custom components Intermediaries and message routing Platform DatabasesMultiple

Evaluation criteria CriterionDescription LatencyHow quickly is the data to be transferred? TransformationComplexity of the transformation ScopeApplication, organization or partner? VolumeQuantity of exchange by a transfer IntrusionDegree of change to existing applications in order to effect transfer EffortEffort required to build and maintain the solution OrganizationWhich solution fits best with the organization?

The Ring(s) 1. Application 2. Enterprise 3. Collaborative partners 4. Everybody else

The Matrix BizTalkSSBIndigoMSMQDTS LatencyNear real time Real timeNear real time Batch TransformationHigh N/A High ScopeEnterprise & partners App. App. & Enterprise VolumeMedium LowMediumHigh IntrusionMediumLowHigh Low EffortHighMediumHighMedium Organizationtbd

The Matrix - Reloaded IndigoMSMQSSB Environment Any WS-* compliant WindowsSQL Server 2005 (both) Application Any distributed application AsynchronousDatabase application Message store In-memory or database NT File system SQL Server 2005 Type of message Persistent & Non-peristent Reliable, Express, Transaction Transactional only Protocol Various TCP only

The Matrix - Revolutions Ring0123 CommunicationSynchronousAsynchronous FocusPerformanceSecurity Integration Security Integration Firewall IndigoYesMaybeNo BizTalk ServerNoYes WS-*MaybeYes MSMQNoYesNo SSBYesNo

Conclusions Integration is hard! Integration requires good thinking! Different technologies available, use them smartly! Start thinking asynchronously and message based!

References Books –Enterprise integration Patterns, Hohpe & Wools –Enterprise Integration Solutions, Spackman & Speaker Web – – –

Questions ?

The Rings(s) 1.Synchronous,.NET or J2EE?, Performance, Indigo, SSB, Web Services 2.Asynchronous, MSMQ or MQSeries?, DTS, Security, WS-* 3.Asynchronous, BizTalk Server, Security, WS-* 4.Firewalls, for the rest you don’t care!