Internet and Intranet Protocols and Applications Lecture 14 Introduction to MQSeries And Asynchronous Messaging May 1, 2002 Joseph Conron Computer Science Department New York University jconron@cs.nyu.edu

What is MQSeries? A middleware product that implements a messaging and queuing framework.   Middleware - an intermediate software component that bridges dissimilar computing environments. Unix, MVS, OS/400 Tandem, VMS, NT, etc.  SNA, NetBios, TCP/IP Cobol, C, JAVA

Messaging and Queueing Messaging - programs communicate by sending data in messages rather than by calling each other directly. Queuing - messages are put on queues in storage, eliminating the need for programs to be logically connected. A messaging and queuing framework is inherently ASYNCHRONOUS!

Asynchronous vs. Synchronous Communications Synchronous: App sends request, then blocks until request is processed. Requires service available at EXACTLY same time as client needs service. Asynchronous: App sends request and checks at some future time if complete. Service need not be available when client sends request

Synchronous: good/bad Easy to program Outcome is known immediately Error recovery easier (usually) Better real-time response (usually) Service must be up and ready requestor blocks, held resources are “tied up” Usually requires connection-oriented protocol

Asynchronous: good/bad Requests need not be targeted to specific server. Service need not be available when request is made No blocking, so resources could be freed Could use connectionless protocol Response times are unpredictable error handling usually more complex Usually requires connection-oriented protocol Harder to design apps?

Messaging vs. Procedure Calls As programmers, many of us think procedurally, so using procedures is “natural” extension of how we think. Messages are an abstract concept: harder for us to conceptualize relationship between actions and messages! But Wait! Something good happens when we use abstractions!

<Soap Box> </Soap Box> We are free to concentrate on the design of the application itself. We are no longer concerned with details of the environment. Our application is suddenly portable and to some degree, extensible. </Soap Box>

A Brief History of MQSeries 1992 Systems Strategies (SSI) develops ezBridge, a messaging and queuing product for VMS, Tandem, and Unix IBM announces Networking Blueprint defining three standard APIs for program to program communication: CPI-C, RPC, MQI 1992-3 State Street Bank (Boston) evaluates IBM messaging product (code name “Victory”) for IBM CICS/ESA and SSI’s ezBridge on VMS and Tandem. “State Street Bank would like to announce the wedding of IBM and Systems Strategies!” 1993 IBM buys intellectual property rights for ezBridge from SSI

The Bride’s Wedding Preparation (What SSI had to do) Implement IBM Channel Protocol over TCP/IP and LU6.2 (more about channels later). Implement the MQI interface functions (MQCONN, MQOPEN, MQPUT, MQGET, MQCOMMIT, MQCLOSE, MQDISC). Implement MQI Error Semantics (failure conditions should look the same on all systems)

MQSeries/ezBridge Integration Objectives Applications written in C using ezBridge on VMS and Tandem had to exchange messages with applications written in (COBOL?) Using MQI under CICS/ESA. System intercommunication using channel protocol over TCP/IP and LU6.2 (that is, A VMS system had to “look” to IBM machine just like another IBM system!).

MQSeries Platform Rollout Initially, IBM’s version of MQSeries ran only on mainframe (CICS/ESA, IMS/ESA, and eventually VSE). SSI version (called MQSeries Version 1) initially released on: VMS, Tandem, AS/400, and Unix (SCO, UnixWare). 1994/1995 -IBM releases the first three “distributed” platforms: AIX, OS/2, and AS/400. The AIX version becomes the “reference port”. Over time, IBM replaced SSI versions with “ports” of its reference system. Today - MQSeries runs on over 35 platforms

How Messaging & Queuing Works Programs communicate by putting messages on queues. Here, program A puts a message on Queue1, which is read by program B. Note: A and B need not be on the same machine!

How Messaging & Queuing Works (2) Communication can be one-way or two-way. Here, A sends to B on Queue1, and B responds to A on Queue2

Messaging and Queuing Characteristics Three key facts about Messaging and Queuing differentiate it from other communication styles: 1) Communicating programs can run at different times. 2) There are no constraints on application structure. 3) Programs are insulated from environmental differences. Let’s examine each of these characteristics in more detail.

Applications Can Run at Different Times Either program can be unavailable Key Concept: message queue exists independently from programs that use them!

No Constraints on Application Structure There can be a one to many relationship between applications Or a many to one relationship between applications

Applications Shielded from Environmental Differences Don’t care what OS is used. Don’t care what language they’re written in Don’t care what the underlying communication protocol is used.

Applications Shielded from Environmental Differences Programmatic API Queue Manager Queue Manager Communications using Message Channels Message Queue

MQSeries Objects: Queue Manager Controls access to queues: administration (create, delete, etc) usage (Put, Get) serves as transaction (syncpoint) coordinator for all queue operations. Accessed through the Message Queue Interface (MQI) Queue Managers have names (identities) that are UNIQUE in a network (like host names).

MQSeries Objects: Queues MQSeries defines four types of queues. A queue instance is fully qualified by its queue manager and queue name. Local Queue - an actual queue for which storage is allocated. Remote Queue - a definition of a queue on a different queue manager (acts somewhat like a pointer) Alias Queue - another name for a local or remote queue. Typically used to switch queue destinations without modifying program code. Model Queue - a template whose properties are copied when creating a new dynamic local queue (“ create queue xxx “like” queue yyy).

MQSeries Queues: Properties Maximum Message Size Maximum Queue Depth High/Low Factors Enable/Disable Put or Get Persistent/Not Persistent

MQSeries Queues: Events and Triggering Local queues can generate events (messages) under certain conditions (like queue full). These “event” messages can be used to “trigger” the execution of a program. These events are called trigger messages. The queue on which they are put is called an Initiation Queue.

MQSeries Objects: Processes Process defines an application to an MQSeries queue manager. A process definition object is used for defining applications to be started by a trigger monitor. A trigger monitor is a program that listens on an initiation queue and executes commands named in Process definitions. Triggering is useful when you don’t want to deploy long-running programs.

MQSeries Objects: Message Channels provide a communication path between two queue managers on the same, or different, platforms. A message channel can transmit messages in one direction only. If two-way communication is required between two queue managers, two message channels are required. Question: why make message channels one-way?

MQSeries Objects: Message Channels(2) Types of message channels: Sender - initiates connection to Receiver Server - Accepts request to start from requester, then becomes Sender Receiver - Passive; waits for initiation sequence form Sender Requester - Active at start, then becomes Receiver Cluster-sender (used amongst Cluster Queue Managers) Cluster-receiver (ditto)

MQSeries: Message Channel Concepts The Sender side of the session is the “transaction coordinator”. Message channels implement a protocol that includes a commitment protocol. Channels recover from failure by agreement: they must agree on the last committed unit of work [would this be harder if channels were bi-directional??] Message Chanels are implemented by programs called Message Channel Agents (MCA)

How messages move across channels (1) Application puts message on transmission queue (4) Message is available on local queue for applications (2) Sender MCA gets message and sends to partner MCA (3) Receiver MCA puts message on target queue

MQSeries: Assured Delivery If queues are persistent, and message is persistent, then MCAs will eventually deliver the message to the target queue, and target application will get it! MCAs have recoverable state - they’re STATEFUL - and a connection-oriented protocol. Message is not removed from xmit queue until partner MCA confirms placement on target queue

MQSeries Objects: Messages Messages consist of: Header (MQMD) Used by Queue Manager and application to handling properties of the message User Data The application-to-application data (“payload”) transparent to MQSeries

MQSeries Messages: Properties Destination Queue Reply Queue name Time to live (expiry) Format Correlation ID Persistence “Report” options

MQSeries Messages: Properties (2) Messages can be individually designated persistent or non-persistent (persistent messages are logged to enable recovery) Correlation ID - select which message to get from queue Segmented Messages - allows ending of VERY LARGE messages (> 100 MB)

MQSeries Messages: message types Request - used by one program to ask another program for something (usually data). A request message needs a reply. Reply - used in response to a request message. A one-way message, as you would expect, doesn’t need a reply, though it can carry data. A report message is used when something unexpected occurs, or to give extra information like: message delivered to target queue message taken by application message not deliverable message exceeded time-to-live

MQSeries Messages Units of Work and Transactions Messages are added and removed from queues in Units of Work The smallest Unit of Work is one message. Units of work are atomic. When an app reads a message from a queue, a message “appears” to have been removed, but in fact, it is still in storage until the app “commits” the unit of work.

MQSeries: Transaction Support Unit of recovery - a piece of work that changes data from one point of consistency to another. Syncpoint - A point of consistency (also called a or commit point). It is a moment at which all the recoverable data that an application program accesses is consistent.

MQSeries: Transaction Support (2) Applications are responsible for delimiting the beginning and end of a transaction. How can messaging be coordinated with a data base update? MQSeries is XA compliant and can operate with other XA compliant systems as either a transaction manager (coordinator) or resource manager (particpant). Some examples: Sybase, DB2, Oracle.

MQSeries: Logging and Recovery All operations that affect the “state” of the Queue Manager and its objects are logged to a log file. What is “state”? Object definitions (queue manager, queues, processes, channels, etc) Queue content (messages) What about message channel state? Message channel states are logged separately by each channel

MQSeries: Logging and Recovery (2) Two forms of Logging Circular – log records are written sequentially across several files, then “wrap” back to the first file. Linear - log records are written sequentially across files. New files are allocated as current files fill. No automatic reuse of file space! Problem: Length (in time) of longest running transaction vs amount of writes to log determines size of log needed.

MQSeries: Logging and Recovery (3) Observations: Circular logging is easy to manage, but is fatal if log is damaged (hard to backup circular logs!). Linear logging is hard to maintain but provides for archiving of previous logs (still a problem if “current” log is damaged).

MQI: The MQSeries Programming Interface MQCONN Connect to queue manager MQDISC Disconnect from queue manager MQOPEN Open queue MQCLOSE Close queue MQPUT Put message MQGET Get message

MQI: The MQSeries Programming Interface (2) MQPUT1 Put one message (implied open/put/close) MQBEGIN Begin unit of work MQCMIT Commit MQBACK Back out MQINQ Inquire about queue attributes MQSET Set queue attributes

MQSeries: Language Support C C++ Cobol JAVA (native and JMS) PL/I System 390 Assembler TAL (Tandem) Visual Basic For More Information: http://www-4.ibm.com/software/ts/mqseries/