Worldwide Messaging Support for High Performance Real-time Collaboration Pete Burnap, Hasan Bulut, Shrideep Pallickara, Geoffrey Fox, David Walker, Ali Kaplan, Beytullah Yildiz, and Mehmet A. Nacar Community Grids Lab, Indiana University School of Computer Science, Cardiff University, UK

What is NaradaBrokering? Distributed messaging infrastructure Provides support for P2P, Centralised and Distributed interaction Efficiently routes messages between originator and registered consumers OBJECTIVE - To provide a recognised open-source standard for a messaging infrastructure to build SOAs and Grids upon It is hoped that these results will influence others to use NaradaBrokering as a messaging protocol for their own applications

Motivation Build distributed systems from interoperable services linked by messages (SOAP) – architect capabilities as services Grids are just large scale sets of such services Need to support real time streams and NOT just files (collections of messages) consistent with WS standards (P2P and central) Manage messaging for Optimize communication for bad links, firewalls etc Collaboration (multi-cast streams) Fault tolerance with re-transmitted messages and Replicated Services Replay – access any message at any time Virtual addressing with pub-sub metaphor Performance from protocol (UDP v Parallel TCP..) and representation Heterogeneous Clients – filter to and from PDAs Open Source (4 downloads/day) is a scalable distributed pub-sub system supporting multiple standards (JMS, WS) and subscription methodshttp:// Implements Service Internet and Notification areas of WS-* Infrastructure Candidate for Axis2-MOM (Message Oriented Middleware) infrastructure

Architecture

NaradaBrokering Features Multiple protocol transport support In publish-subscribe Paradigm with different Protocols on each link Transport protocols supported include TCP, Parallel TCP streams, UDP, Multicast, SSL, HTTP and HTTPS. Communications through authenticating proxies/firewalls & NATs. Network QoS based Routing Allows Highest performance transport Subscription FormatsSubscription can be Strings, Integers, XPath queries, Regular Expressions, SQL and tag=value pairs. Reliable deliveryRobust and exactly-once delivery in presence of failures Ordered deliveryProducer Order and Total Order over a message type. Time Ordered delivery using Grid-wide NTP based absolute time Recovery and ReplayRecovery from failures and disconnects. Replay of events/messages at any time. Buffering services. SecurityMessage-level WS-Security compatible security Message Payload optionsCompression and Decompression of payloads Fragmentation and Coalescing of payloads Messaging Related Compliance Java Message Service (JMS) 1.0.2b compliant Support for routing P2P JXTA interactions. Grid Feature SupportNaradaBrokering enhanced Grid-FTP. Bridge to Globus GT3. Web Services supportedImplementations of WS-ReliableMessaging, WS-Reliability and WS-Eventing.

NaradaBrokering Services

Buffering Service This service is incorporated into the system to facilitate the buffering of events prior to releasing them. Buffering service time orders events and releases event based on three metrics Number of events in the buffer Size of the buffer Time spent by event in a buffer.

Time Differential Service This service is essential to reduce jitters in large distributed environments. Networks introduce unpredictable delays that increase jitter. Service takes events released by buffering service, and ensures that it preserves time spacing between events. TDS can provide time spacing resolution of up to 1 millisecond between events.

Reliable Delivery Service Guaranteed delivery in multiple producer/ consumer settings. Guarantees hold true in the presence of Node/Link Failures Links can lose messages and garble message order. Storage failures: Stores need to recover after failure. Prolonged entity disconnects Exactly-Once and Ordered delivery of events Uses both positive & negative acknowledgements Supports Replay and Fast Recovery from failures Independent of underlying archival system. Was used to enhance fault tolerance in Grid-FTP.

Dealing with large payload sizes To cope with large payloads, the substrate incorporate 2 sets of services. Compression/Decompression service: The substrate incorporate support for zlib based compression and decompression of payloads. Fragmentation/Coalescing Service: These service can break up a large payload into smaller fragments. The coalescing service can take these smaller fragments and coalesce them into the original large payload. This was used to deal with transfer of large payloads (up to 1 GB) in the NB enhanced Grid-FTP application.

Broker Discovery Service Locates the nearest available broker that a client can connect to Incorporates specialized nodes – broker discovery nodes – to maintain broker info. Depending on load or security issues, brokers may decide to respond/ignore discovery requests. If available the scheme can exploit IP multicast for discovery. Nearest broker determined by ping times, loss rates and available bandwidth.

Topic Discovery Service Allows publishers and subscribers to advertise topics. Creator of topic possesses credentials to indicate ownership of the topic. Discovery of topics takes into account credentials of client trying to discover topic. Topic owner may restrict discovery to a limited authorized set of clients. Discovery requests can be made using simple strings or regular expression queries.

Security Service [ Based on Message Level Security ] Messages organized into topics Each topic has a separate key; Topics can be organized into sessions

Web Services Support Currently we have incorporated support for the following Web Service specifications WS-Eventing (WSE): This is a publish/subscribe based notification framework from Microsoft and IBM. WS-ReliableMessaging (WSRM): This is a protocol for ensuring the guaranteed delivery of SOAP messages between 2 Web Service endpoints. This specification is from IBM and Microsoft. WS-Reliability (WSR): This is a competing specification from Oracle and Sun in the area of reliable messaging between Web Services. These handlers are available for use in Axis1.2 or exploiting NB SOAP Intermediary support without a container Axis1.2 version can be used inside container or as a Proxy

Web Services Support – cont.. We are also working on implementing support for the WS-Notification (WSN) suite of specification that is part of the Web Services Resource Framework (WSRF). WS-Notification explicitly adds brokers to Eventing Note that almost all these specifications leverage the WS-Addressing (WSA) specification. We have incorporated support for all the rules associated with WSA.

Parallel TCP Streaming Data transfer across high-performance networks suffers within current TCP implementation due to tendency to hold back maximum bandwidth usage Can be solved either by using TCP multiple streams in parallel TCP streams consists of three basic steps: splitting of data into sub packets at sender side sending these sub packets over the network by using multiple Java socket streams in parallel merging of received sub packets at the receiver side.

TCP Test Observations This illustrates the capability of NaradaBrokering to maintain a high speed message delivery performance despite the scaling of message content payload. Mean transmission delay times start to rise as the payload increase becomes increasingly steep between 1000 and bytes

TCP Streaming Results LAN Test Continental WAN Test Bandwidth (mbps)

TCP Streaming Results cont… Inter-Continental WAN Test

Future of NaradaBrokering Support for replicated storages within the system. In a system with N replicas the scheme can sustain the loss of N- 1 replicas. Clarification and expansion of NB Broker to act as a WS container Integration with Axis 2.0 as Message Oriented Middleware infrastructure Support for High Performance transport and representation for Web Services Needs Context catalog under development Performance based routing The broker network will dynamically respond to changes in the network based on metrics gathered at individual broker nodes. Replicated publishers for fault tolerance Pure client P2P implementation (originally we linked to JXTA) Security Enhancements for fine-grain topic authorization, multi-cast keys, Broker attacks

Further Information Web - Geoffrey Fox - Shrideep Pallickara -