1. Click to add text © 2012 IBM Corporation 1 Messaging Toolkit InfoSphere Streams Version 3.0 Anjali Agarwal Toolkit

2. © 2012 IBM Corporation 2 Important Disclaimer THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “ AS IS ”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM ’ S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: CREATING ANY WARRANTY OR REPRESENTATION FROM IBM (OR ITS AFFILIATES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS); OR ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF IBM SOFTWARE. The information on the new product is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information on the new product is for informational purposes only and may not be incorporated into any contract. The information on the new product is not a commitment, promise, or legal obligation to deliver any material, code or functionality. The development, release, and timing of any features or functionality described for our products remains at our sole discretion. THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.

3. © 2012 IBM Corporation 3 Agenda  Introduction to Messaging Toolkit  Use cases -Why use Messaging Toolkit  Operators  Terminologies and Concepts  Software Dependency  Enhancements/Features common to XMSSource and XMSSink  Enhancements/Features for XMSSink  Enhancements/Features for XMSSource

4. Click to add text © 2012 IBM Corporation 4 Introduction to Messaging Toolkit

5. © 2012 IBM Corporation 5 What is Messaging Toolkit  A set of edge adapters which help to read/write to/from a queue or topic in WebSphere Message Queue –XMSSource –XMSSink  Uses the JMS api's exposed by the IBM XMS client in C++ language

6. © 2012 IBM Corporation 6 Brief Architecture Diagram WebSphere MQ Queue/Topic Streams XMS Source XMS Sink Read Message Write Message Create connection XMS Create Producer/ Consumer MQ Client Uses

7. Click to add text © 2012 IBM Corporation 7 Use Cases

8. © 2012 IBM Corporation 8 Real time campaign management  Let's take the case of a Large Bank which has lots of credit card customers. Imagine if someone performs a credit card purchase in a classy mall. We know where the transaction has been committed and the bank can offer various cross sell and up sell products. The transaction committed is captured by Streams and then Streams processes this and posts the message on to the Messaging queue layer. Unica further connects to the messaging queue and instantiates the campaign. This example illustrates how organisations are utilizing SOA technologies to integrate their enterprise systems and how messaging toolkit adapters fit into the eco- system seamlessly to respond in real time and increase efficiency and timeliness of the processes.

9. © 2012 IBM Corporation 9 Tracking stolen vehicles  DMV issues licences to lots of vehicles. Electronic Toll Collection(ETC) allows for electronic collection of tolls by identifying cars and electronically debiting the account of registered car owners. Information from ETC can be used for smarter crime detection and prevention. –Tracking stolen cars: Assuming all toll information is continuously sent and filtered against the stolen car list by Streams, when the stolen car passes a toll, it can be flagged and sent to BPM systems. –Tracking stolen devices: When a device theft is reported to enforcers, an entry is made to systems containing details about stolen devices. When a car containing the stolen device passes a toll, it can be flagged and sent to BPM systems. –Further we can use GeoSpatial (GIS) toolkit to find out the tentative location of the vehicle and based on the speed and the route which that vehicle took we can help in tracking it. Further this information can also be plotted in a map, and displayed to interested parties.

10. Click to add text © 2012 IBM Corporation 10 Operators

11. © 2012 IBM Corporation 11 XMSSource  This operator reads data from a WebSphere MQ queue or a topic and pushes the data into Streams. The main functionality includes: –Zero input ports, One or Two output ports –Validates if the Streams application can connect to WebSphere MQ during runtime –Validates that the queue/topic name specified in the initial context element exists during runtime –Validates that all the Stream tuple elements which do not have an explicit values assigned to them are present in the native schema and the datatype of these elements/attributes match during compile time –Validates if the message class specified in the native schema is a valid MQ message class during compile time –For Initial/ Transient connection failures, the operator tries to reconnect based on the reconnection policy specified by the user –If an error occurs while reading the input from queue/topic, writes the error string into the error output port (if one is specified) –On successful read, submits the tuple to the output stream corresponding to the message on MQ

12. © 2012 IBM Corporation 12 XMSSink  Publishes data from Streams to a WebSphere MQ queue or a topic. The main functionality of this operator includes: –One input port, One output port(optional) –Validates if the Streams application can connect to WebSphere MQ during runtime –Validates that the queue/topic name specified in the initial context element exists during runtime –Validates that all the attributes in the native schema are present in the input stream tuple and the datatype of these elements/attributes match during compile time –Validates if the message class specified in the native schema is a valid MQ message class during compile time –For initial/transient connection failures, the operator tries to reconnect based on the reconnection policy specified by the user –If an error occurs while writing to a queue/topic, writes the error string and the erroneous input tuple into the error output port (if one is specified) –On successful write submits the tuple in the input stream to the corresponding message on MQ

13. Click to add text © 2012 IBM Corporation 13 Terminologies and concepts

14. © 2012 IBM Corporation 14 Connection Specification Document  xml file which specifies the details of connection to WebSphere MQ

15. © 2012 IBM Corporation 15 Message class and native schema  Message class specifies how a Stream tuple is to be serialized into a MQ message. There are 7 types of message classes: –map : A message containing a collection of name/type/value triplets –stream : A message containing ordered list of type/value pairs –empty : An empty message –xml : A message containing a generic XML message –bytes :A Message containing a stream of binary data –wbe : A message contains an XML document in the WebSphere Business Events event packet format –wbe22 : A message contains an XML document in the old WebSphere Business Events event packet format 2.2  Native schema: Specifies the data attributes (their name, type(external type) and length(optional)) for a message which is read from /written to a queue/topic in WebSphere MQ

16. © 2012 IBM Corporation 16 Mapping between message class, native schema Msg class XMSSink XMSSource map Constructs XMS Mapmessage containing elements whose names and types are taken from native_schema and values taken from the operator input stream. Expects a XMS MapMessage, containing elements whose names and types appear in the native_schema element. stream Constructs XMS StreamMessage containing values taken from the input stream,serialized into the message in the order in which they are specified in the native_schema element, using the data types from that element. Expects a XMS StreamMessage containing the values of the elements listed in the native_schema, in the order in which they are specified in the native_schema and with the data types specified in that schema. bytes Constructs XMS BytesMessage containing values taken from the input stream, serialized into the message in the order in which they are specified in the native_schema element, using the data types from that element. Expects a XMS BytesMessage, containing the values of the elements listed in the native_schema, in the order in which they are specified in the native_schema and with the data types specified in that schema. empty Constructs an empty XMS message.Expects an empty XMS message. xml Constructs a XMS message containing a generic XML message Not Supported wbe Constructs a XMS TextMessage containing an XML document in the wbe format. Not Supported wbe22 Constructs a XMS TextMessage containing an XML document in the wbe 2.2 format Not Supported

17. Click to add text © 2012 IBM Corporation 17 Software Dependancies

18. © 2012 IBM Corporation 18 Software Dependancies  Operating System: All Operating System supported by Streams except Power  IBM Message Service client for C/C++ (also known as XMS client) version 2.0.1 is required –The XMS client must be installed on the same machine as Streams is installed and an environment variable (XMS_HOME) needs to be set to point to the install location of XMS Client  The WebSphere MQ Server and WebSphere MQ Client need to be at: –WebSphere MQ 7.0 + All Fix Pack 7.0.0.1 or later –WebSphere MQ 6.0 + Refresh Pack 6.0.1.1 or later –WebSphere MQ 7.1 is not supported by the current version of XMS : 2.0.1 –The WebSphere MQ client libraries must be on the same machine as Streams and an environment variable (MQ_HOME) needs to be set to point to the MQ Client libraries.

19. Click to add text © 2012 IBM Corporation 19 Features common to both Operators

20. © 2012 IBM Corporation 20 Blob support for multiple message classes  The SPADE version supported sending payloads in the message  Blob support introduced for message class map, stream and bytes in both operators  Blob support introduced for message class xml in XMSSink as an enhancement composite XMSSinkMain { graph stream BlobData = FileSource(){ param file : "Image.png"; format : block; blockSize : 1276431234u; } () as Sink1 = XMSSink(BlobData){ param connection : "Conn1"; access : "OutTrading1"; }

21. © 2012 IBM Corporation 21 Support for Reconnection Policies  The default reconnectionPolicy in case of initial/transient connection failure was to try the connection infinitely  Support was introduced to allow the user to specify the reconnectionPolicy parameter with valid values of NoRetry, InfiniteRetry and BoundedRetry  Additional parameters reconnectionBound and period were introduced to give more flexibility to the user  Default is BoundedRetry with reconnectionBound of 5, period of 60 sec composite XMSSourceMain { graph (stream InputData) = XMSSource(){ param connection : "Conn1"; access : "InTrading1"; reconnectionPolicy : BoundedRetry; reconnectionBound : 10u; period : 1.0; } composite XMSSourceMain { graph (stream InputData) = XMSSource(){ param connection : "Conn1"; access : "InTrading1"; reconnectionPolicy : NoRetry; } composite XMSSourceMain { graph (stream InputData) = XMSSource(){ param connection : "Conn1"; access : "InTrading1"; reconnectionPolicy : InfiniteRetry; period : 1.0; }

22. © 2012 IBM Corporation 22 Support for an optional error output port  Support was introduced in both the operators to write error string on to the error output port in case of read/write failures  Optional error output port is non-mutating and its punctuation mode is free composite XMSSourceMain { graph (stream InputData; stream Error) = XMSSource(){ param connection : "Conn1"; access : "InTrading1"; } () as Sink1 = FileSink(InputData){ param file : "XMSSourceOutput.txt"; flush : 1u; } () as Sink2 = FileSink(Error){ param file : "XMSSourceError.txt"; flush : 1u; }

23. © 2012 IBM Corporation 23 Other Enhancements  Supporting external types in native schema –Previously SPL types were specified in the native schema –Changes to accommodate specifying external types in native schema for ease of use  Introduction of environment variables to point to MQ and XMS install locations –The XMS install location was hard coded to /opt/IBM/XMS and the MQ install location hard coded to /opt/mqm –Environment variables XMS_HOME and MQ_HOME introduced for allowing flexibility  Support for uint8, uint16 and uint32 data types. –Support initially existent only for signed integers –Support for uintb data types introduced

24. Click to add text © 2012 IBM Corporation 24 Enhancements/Features for XMSSink

25. © 2012 IBM Corporation 25 Negative length support on message class bytes  Message class bytes allows specification of -2,-4, and -8 as valid values in the native schema for string and blob  Negative length indicates that a 2,4 or 8 byte length needs to be pre-pended with the data representing the length of the data composite XMSSinkMain { graph stream InputData = Beacon(){ param iterations:1u; output InputData :k="ThisIsALongString"; } () as Sink1 = XMSSink(InputData){ param connection : "Conn1"; access : "OutTrading1"; } composite XMSSinkMain { graph stream InputData = Beacon(){ param iterations:1u; output InputData :k="ThisIsALongString"; } () as Sink1 = XMSSink(InputData){ param connection : "Conn1"; access : "OutTrading1"; } <destination identifier="Dest" delivery_mode="persistent" message_class="bytes" /> <destination identifier="Dest" delivery_mode="persistent" message_class="bytes" />

26. © 2012 IBM Corporation 26 Truncation and padding support for string  When a positive length is specified in the native schema for string in message classes map, stream and xml, it is treated as an upper bound for that string  For message class bytes, the string is truncated/padded in accordance with the length in the native schema composite XMSSinkMain { graph stream InputData = Beacon(){ param iterations:1u; output InputData :k="ThisIsALongString"; } () as Sink1 = XMSSink(InputData){ param connection : "Conn1"; access : "OutTrading1"; } composite XMSSinkMain { graph stream InputData = Beacon(){ param iterations:1u; output InputData :k="ThisIsALongString"; } () as Sink1 = XMSSink(InputData){ param connection : "Conn1"; access : "OutTrading1"; }

27. © 2012 IBM Corporation 27 Performance Metrics Support  XMSSink has the following metrices –nTruncatedInserts: The number of tuples that had truncated attributes while converting to a message –nFailedInserts: The number of failed inserts to the WebSphere MQ –nReconnectionAttempts: The number of reconnection attempts made before a successful connection

28. Click to add text © 2012 IBM Corporation 28 Enhancements/Features for XMSSource

29. © 2012 IBM Corporation 29 Support for initDelay parameter  Support for initDelay parameter introduced in XMSSource composite XMSSourceMain { graph (stream InputData) = XMSSource(){ param connection : "Conn1"; access : "InTrading1"; initDelay : 10.0; } () as Sink1 = FileSink(InputData){ param file : "XMSSourceOutput.txt"; flush : 1u; } composite XMSSourceMain { graph (stream InputData) = XMSSource(){ param connection : "Conn1"; access : "InTrading1"; initDelay : 10.0; } () as Sink1 = FileSink(InputData){ param file : "XMSSourceOutput.txt"; flush : 1u; }

30. © 2012 IBM Corporation 30 Performance Metrics Support  No performance metrics was existent in the SPADE version  XMSource supports the following performance metrics –nMessagesRead: The number of messages read successfully – nMessagesDropped: The number of messages dropped –nReconnectionAttempts: The number of reconnection attempts made before a successful connection

31. © 2012 IBM Corporation 31

32. © 2012 IBM Corporation 32