1. Internet and Intranet Protocols and Applications
Lecture 14
Introduction to MQSeries
And Asynchronous Messaging
May 1, 2002
Joseph Conron
Computer Science Department
New York University

2. What is MQSeries?
A middleware product that implements a messaging and queuing framework.
Middleware - an intermediate software component that bridges dissimilar computing environments.
Unix, MVS, OS/400 Tandem, VMS, NT, etc.
 SNA, NetBios, TCP/IP
Cobol, C, JAVA

3. Messaging and Queueing
Messaging - programs communicate by sending data in messages rather than by calling each other directly.
Queuing - messages are put on queues in storage, eliminating the need for programs to be logically connected.
A messaging and queuing framework is inherently ASYNCHRONOUS!

4. Asynchronous vs. Synchronous Communications
Synchronous: App sends request, then blocks until request is processed.
Requires service available at EXACTLY same time as client needs service.
Asynchronous: App sends request and checks at some future time if complete.
Service need not be available when client sends request

5. Synchronous: good/bad
Easy to program
Outcome is known immediately
Error recovery easier (usually)
Better real-time response (usually)
Service must be up and ready
requestor blocks, held resources are “tied up”
Usually requires connection-oriented protocol

6. Asynchronous: good/bad
Requests need not be targeted to specific server.
Service need not be available when request is made
No blocking, so resources could be freed
Could use connectionless protocol
Response times are unpredictable
error handling usually more complex
Usually requires connection-oriented protocol
Harder to design apps?

7. Messaging vs. Procedure Calls
As programmers, many of us think procedurally, so using procedures is “natural” extension of how we think.
Messages are an abstract concept: harder for us to conceptualize relationship between actions and messages!
But Wait!
Something good happens when we use abstractions!

8. <Soap Box> </Soap Box>
We are free to concentrate on the design of the application itself.
We are no longer concerned with details of the environment.
Our application is suddenly portable and to some degree, extensible.
</Soap Box>

9. A Brief History of MQSeries
1992
Systems Strategies (SSI) develops ezBridge, a messaging and queuing product for VMS, Tandem, and Unix
IBM announces Networking Blueprint defining three standard APIs for program to program communication: CPI-C, RPC, MQI
1992-3
State Street Bank (Boston) evaluates IBM messaging product (code name “Victory”) for IBM CICS/ESA and SSI’s ezBridge on VMS and Tandem.
“State Street Bank would like to announce the wedding of
IBM and Systems Strategies!”
1993
IBM buys intellectual property rights for ezBridge from SSI

10. The Bride’s Wedding Preparation (What SSI had to do)
Implement IBM Channel Protocol over TCP/IP and LU6.2 (more about channels later).
Implement the MQI interface functions (MQCONN, MQOPEN, MQPUT, MQGET, MQCOMMIT, MQCLOSE, MQDISC).
Implement MQI Error Semantics (failure conditions should look the same on all systems)

11. MQSeries/ezBridge Integration Objectives
Applications written in C using ezBridge on VMS and Tandem had to exchange messages with applications written in (COBOL?) Using MQI under CICS/ESA.
System intercommunication using channel protocol over TCP/IP and LU6.2 (that is, A VMS system had to “look” to IBM machine just like another IBM system!).

12. MQSeries Platform Rollout
Initially, IBM’s version of MQSeries ran only on mainframe
(CICS/ESA, IMS/ESA, and eventually VSE).
SSI version (called MQSeries Version 1) initially released on:
VMS, Tandem, AS/400, and Unix (SCO, UnixWare).
1994/ IBM releases the first three “distributed” platforms:
AIX, OS/2, and AS/400.
The AIX version becomes the “reference port”.
Over time, IBM replaced SSI versions with “ports” of its reference system.
Today - MQSeries runs on over 35 platforms

13. How Messaging & Queuing Works
Programs communicate by putting messages on queues. Here, program A puts a message on Queue1, which is read by program B.
Note: A and B need not be on the same machine!

14. How Messaging & Queuing Works (2)
Communication can be one-way or two-way. Here, A sends to B on Queue1, and B responds to A on Queue2

15. Messaging and Queuing Characteristics
Three key facts about Messaging and Queuing differentiate it from other communication styles:
1) Communicating programs can run at different times.
2) There are no constraints on application structure.
3) Programs are insulated from environmental differences.
Let’s examine each of these characteristics in more detail.

16. Applications Can Run at Different Times
Either program can be unavailable
Key Concept: message queue exists independently from programs that use them!

17. No Constraints on Application Structure
There can be a one to many relationship between applications
Or a many to one relationship between applications

18. Applications Shielded from Environmental Differences
Don’t care what OS is used.
Don’t care what language they’re written in
Don’t care what the underlying communication protocol is used.

19. Applications Shielded from Environmental Differences
Programmatic API
Queue Manager
Queue Manager
Communications using Message Channels
Message Queue

20. MQSeries Objects: Queue Manager
Controls access to queues:
administration (create, delete, etc)
usage (Put, Get)
serves as transaction (syncpoint) coordinator for all queue operations.
Accessed through the Message Queue Interface (MQI)
Queue Managers have names (identities) that are UNIQUE in a network (like host names).

21. MQSeries Objects: Queues
MQSeries defines four types of queues. A queue instance is fully qualified by its queue manager and queue name.
Local Queue - an actual queue for which storage is allocated.
Remote Queue - a definition of a queue on a different queue manager (acts somewhat like a pointer)
Alias Queue - another name for a local or remote queue. Typically used to switch queue destinations without modifying program code.
Model Queue - a template whose properties are copied when creating a new dynamic local queue (“ create queue xxx “like” queue yyy).

22. MQSeries Queues: Properties
Maximum Message Size
Maximum Queue Depth
High/Low Factors
Enable/Disable Put or Get
Persistent/Not Persistent

23. MQSeries Queues: Events and Triggering
Local queues can generate events (messages) under certain conditions (like queue full).
These “event” messages can be used to “trigger” the execution of a program.
These events are called trigger messages. The queue on which they are put is called an Initiation Queue.

24. MQSeries Objects: Processes
Process defines an application to an MQSeries queue manager. A process definition object is used for defining applications to be started by a trigger monitor.
A trigger monitor is a program that listens on an initiation queue and executes commands named in Process definitions.
Triggering is useful when you don’t want to deploy long-running programs.

25. MQSeries Objects: Message Channels
provide a communication path between two queue managers on the same, or different, platforms.
A message channel can transmit messages in one direction only. If two-way communication is required between two queue managers, two message channels are required.
Question: why make message channels one-way?

26. MQSeries Objects: Message Channels(2)
Types of message channels:
Sender - initiates connection to Receiver
Server - Accepts request to start from requester, then becomes Sender
Receiver - Passive; waits for initiation sequence form Sender
Requester - Active at start, then becomes Receiver
Cluster-sender (used amongst Cluster Queue Managers)
Cluster-receiver (ditto)

27. MQSeries: Message Channel Concepts
The Sender side of the session is the “transaction coordinator”.
Message channels implement a protocol that includes a commitment protocol.
Channels recover from failure by agreement: they must agree on the last committed unit of work [would this be harder if channels were bi-directional??]
Message Chanels are implemented by programs called Message Channel Agents (MCA)

28. How messages move across channels
(1) Application puts message on transmission queue
(4) Message is available on local queue for applications
(2) Sender MCA gets message and sends to partner MCA
(3) Receiver MCA puts message on target queue

29. MQSeries: Assured Delivery
If queues are persistent, and message is persistent, then MCAs will eventually deliver the message to the target queue, and target application will get it!
MCAs have recoverable state - they’re STATEFUL - and a connection-oriented protocol.
Message is not removed from xmit queue until partner MCA confirms placement on target queue

30. MQSeries Objects: Messages
Messages consist of:
Header (MQMD)
Used by Queue Manager and application to handling properties of the message
User Data
The application-to-application data (“payload”)
transparent to MQSeries

31. MQSeries Messages: Properties
Destination Queue
Reply Queue name
Time to live (expiry)
Format
Correlation ID
Persistence
“Report” options

32. MQSeries Messages: Properties (2)
Messages can be individually designated persistent or non-persistent (persistent messages are logged to enable recovery)
Correlation ID - select which message to get from queue
Segmented Messages - allows ending of VERY LARGE messages (> 100 MB)

33. MQSeries Messages: message types
Request - used by one program to ask another program for something (usually data). A request message needs a reply.
Reply - used in response to a request message.
A one-way message, as you would expect, doesn’t need a reply, though it can carry data.
A report message is used when something unexpected occurs, or to give extra information like:
message delivered to target queue
message taken by application
message not deliverable
message exceeded time-to-live

34. MQSeries Messages Units of Work and Transactions
Messages are added and removed from queues in Units of Work
The smallest Unit of Work is one message.
Units of work are atomic.
When an app reads a message from a queue, a message “appears” to have been removed, but in fact, it is still in storage until the app “commits” the unit of work.

35. MQSeries: Transaction Support
Unit of recovery - a piece of work that changes data from one point of consistency to another.
Syncpoint - A point of consistency (also called a or commit point). It is a moment at which all the recoverable data that an application program accesses is consistent.

36. MQSeries: Transaction Support (2)
Applications are responsible for delimiting the beginning and end of a transaction. How can messaging be coordinated with a data base update?
MQSeries is XA compliant and can operate with other XA compliant systems as either a transaction manager (coordinator) or resource manager (particpant).
Some examples: Sybase, DB2, Oracle.

37. MQSeries: Logging and Recovery
All operations that affect the “state” of the Queue Manager and its objects are logged to a log file.
What is “state”?
Object definitions (queue manager, queues, processes, channels, etc)
Queue content (messages)
What about message channel state?
Message channel states are logged separately by each channel

38. MQSeries: Logging and Recovery (2)
Two forms of Logging
Circular – log records are written sequentially across several files, then “wrap” back to the first file.
Linear - log records are written sequentially across files. New files are allocated as current files fill. No automatic reuse of file space!
Problem: Length (in time) of longest running transaction vs amount of writes to log determines size of log needed.

39. MQSeries: Logging and Recovery (3)
Observations:
Circular logging is easy to manage, but is fatal if log is damaged (hard to backup circular logs!).
Linear logging is hard to maintain but provides for archiving of previous logs (still a problem if “current” log is damaged).

40. MQI: The MQSeries Programming Interface
MQCONN Connect to queue manager
MQDISC Disconnect from queue manager
MQOPEN Open queue
MQCLOSE Close queue
MQPUT Put message
MQGET Get message

41. MQI: The MQSeries Programming Interface (2)
MQPUT1 Put one message (implied open/put/close)
MQBEGIN Begin unit of work
MQCMIT Commit
MQBACK Back out
MQINQ Inquire about queue attributes
MQSET Set queue attributes

42. MQSeries: Language SupportC
C++
Cobol
JAVA (native and JMS)
PL/I
System 390 Assembler
TAL (Tandem)
Visual Basic
For More Information: