Performance of a possible Grid Message Infrastructure Edinburgh e-Science Centre December 9 2002 PTLIU Laboratory for Community Grids Geoffrey Fox, Shrideep Pallickara Computer Science, Informatics, Physics Indiana University, Bloomington IN 47404 http://grids.ucs.indiana.edu/ptliupages gcf@indiana.edu 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

Some Basic Observations/Goals Grids manage and share asynchronous resources in a rather centralized fashion Peer-to-peer networks are “just like” Grids with different implementations of services like registration and look-up Web Services interact with messages and Everything will be a Web Service Microsoft Office will be a WS? Computers are fast and getting faster. One can afford many strategies that used to be unrealistic Software/Application-level message processing/routing sensitive to performance information Uniform messaging framework for Middle-tier, “High performance layer”, Multimedia (audio-video) etc. 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" Deduction? Grids or P2P Networks consist of a sea of message-based Services Services inject and extract messages whose transport and manipulation is support by a logically distinct “MessageGrid” of brokers/routers supplying “Grid Message Layer” The message brokers support adaptive routing, filtering, workflow They have publish/subscribe queued connection semantics They separate logical and actual transport They form a federated XML database and support asynchronous collaboration They process real-time messages in about a millisecond and support synchronous collaboration NaradaBrokering is a prototype of this idea 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

Peer to Peer Grid Peers Peers Database Database Peers Service Facing Web Service Interfaces Event/ Message Brokers Peer to Peer Grid Peers User Facing Web Service Interfaces Peer to Peer Grid of Resources supported by internal “MessageGrid” of message or event brokers Events are “just” Time-stamped messages 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

Event (Message) Service Shared Input Port (Replicated WS) Collaboration Collaboration as a WS Set up Session with XGSP Web Service F I U O R WS Display WS Viewer Master Web Service F I U O R WS Display WS Viewer Event (Message) Service Other Participants Web Service F I U O R WS Display WS Viewer 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" NaradaBrokering Based on a network of cooperating broker nodes Cluster based architecture allows system to scale to arbitrary size Originally designed to provide uniform software multicast to support real-time collaboration linked to publish-subscribe for asynchronous systems. Now has four major core functions Message transport (based on performance measurement) in heterogeneous multi-link fashion General publish-subscribe including JMS & JXTA and support for RTP-based audio/video conferencing Filtering for heterogeneous clients Federation of multiple instances of Grid services 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

Role of Event/Message Brokers We will use events and messages interchangeably An event is a time stamped message Our systems are built from clients, servers and “event brokers” These are logical functions – a given computer can have one or more of these functions In P2P networks, computers typically multifunction; in Grids one tends to have separate function computers Event Brokers “just” provide message/event services; servers provide traditional distributed object services as Web services There are functionalities that only depend on event itself and perhaps the data format; they do not depend on details of application and can be shared among several applications NaradaBrokering is designed to provide these functionalities MPI provided such functionalities for all parallel computing 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

Engineering Issues Addressed by Event / Messaging Service Application level Quality of Service – e.g. give audio highest priority Tunnel through firewalls & proxies Filter messages to slow (collaborative/real-time) clients Choose Hardware or Software multicast Scaling of software multicast Efficient calculation of destinations and routes. Integrate synchronous and asynchronous collaboration with same messaging, control, archiving for all functions Supports local broker accesses Transparently replace single server JMS systems with a distributed solution. Provides reliable inter-peer group messaging for JXTA 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

Features of Event Service MPI nowadays aims at a microsecond latency The Event Web Service aims at a millisecond (computer) latency Typical distributed system travel times are many milliseconds (to seconds for Geosynchronous satellites) Different performance/functionality trade-off Messages are not sent directly from P to S but rather from P to Broker B and from Broker B (via other brokers) to subscriber S. Synchronous systems: B acts as a real-time router/filterer Messages can be archived and software multicast Asynchronous systems: B acts as a database & workflow engine Subscription is in each case, roughly equivalent to a database query Company X sets up a firewall The event service sets up brokers either side of firewall to optimize transport through the firewall. 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" Narada Broker Network (P2P) Community For message/events service Broker Broker (P2P) Community Resource Hypercube topology for brokers? Tree for distance education with teacher at root Broker Broker Broker Data base (P2P) Community Software multicast Broker (P2P) Community 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

Performance in Message-based Architecture I Satellite UDP Firewall HTTP Dial-up Filter A B1 Hand-Held Protocol Fast Link Software Multicast B2 Useful analogies with transportation grids and parallel computing In traveling from cities A to B (say 3 separate passengers), one chooses between and changes transport mechanism at waystations to optimize cost, time, comfort, scenic beauty … Waystations are now NB brokers where one chooses transport protocol Able to choose between car, type of car, plane, train etc Able to dynamically create waystations to cope with problems and acts as hubs for multicast messages Knows about traffic jams and can assign the “HOV lane” B3 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

Performance in Message-based Architecture II Application level QoS – can optimize among managed streams (audio versus video) using performance subsystem This is just a variant of the NP complete load balancing problem of parallel computing where all reasonable heuristics worked Load-balance in Space-time (strings) not just Space (particles) “Performance” needs to measured carefully as includes QoS I delayed shared application update to ensure audio quality and filtered image to lower resolution So “application” has changed to satisfy performance constraints 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

NaradaBrokering Communication Applications interface to NaradaBrokering through UserChannels which NB constructs as a set of links between NB Broker waystations which may need to be dynamically instantiated UserChannels have publish/subscribe semantics with XML topics Links implement a single conventional “data” protocol. Interface to add new transport protocols within the Framework Administrative channel negotiates the best available communication protocol for each link Different links can have different underlying transport implementations Implementations in the current release include support for TCP,UDP, Multicast, SSL and RTP. HTTP, HTTPS support will be available in Feb 2003 release. Supports communication through proxies such as iPlanet, Netscape and Apache. Supports communication through firewalls such as Microsoft ISA, Checkpoint. 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

Types of Performance Measurement “Core Performance”: Equivalent of node Clock Cycle and node to node MPI Ping/Bandwidth measurements Cosmic single number which does not satisfy anybody but nevertheless dominates discussion: Equivalent to bisection bandwidth or LINPACK Detailed performance numbers for particular Grid applications: e.g. Time to assimilate global weather data and simulate hurricane We will present “core performance” numbers for NaradaBrokering running in different modes Will not discuss “Web Service” performance – rather performance of Message substrate 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" Note on Optimization Note in parallel computing, couldn’t do much dynamic optimization as aiming at microsecond latency Natural to use hardware routing In Grid, time scales are different 100 millisecond quite normal network latency 30 millisecond typical packet time sensitivity (this is one audio or video frame) but even here can buffer 10-100 frames on client (conferencing to streaming) 1 millisecond is time for a Java server to “think” Jitter in latency (transit time) due to routing, processing (in NB) or packet loss recovery is important property Grid needs and can tolerate significant dynamic optimization 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" Sender/receiver/broker - (Pentium-3, 1 GHz, 256 MB RAM). 100 Mbps LAN. JDK-1.3, Red Hat Linux 7.3 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" Sender/receiver/broker - (Pentium-3, 1 GHz, 256 MB RAM). 100 Mbps LAN. JDK-1.3, Red Hat Linux 7.3 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" H.263 video file Average bit-rate of 600Kbps. Frame-rate of 30 frames/sec Jitter J = J + (|D(i-1, i)| - J)/16 Jitter J is defined by the RTP RFC as the mean deviation (smoothed absolute value) of the difference D in packet spacing at the receiver compared to the sender for a pair of packets Jitter represents the jitter value up until that point in the sample. 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" SSL Overhead I The test program simulated a SSL client transferring a 5 MB chunk of data to an SSL server. Test Environment 800 MHz PIII JDK 1.3, JSSE 1.0.3 Numbers are approximate over 5 iterations. SSL Results SSL Socket Creation: ~250 ms SSL Handshaking: ~500 ms Transfer of 5 MB: ~2050 ms 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

SSL Overhead (ISA HTTPS Proxy) Test Environment was as before except going through Microsoft ISA’s HTTPS proxy in between the client and the server. The proxy machine’s configuration was:    Pentium 900 MHz, 512 MB RAM. Microsoft ISA proxy Enterprise Edition allow HTTPS protocol through Proxy => Server network bandwidth 10 Mbps. SSL Socket Creation: ~250 ms SSL Handshaking (after 1st one):~400 ms Transfer of 50 MB: ~50 seconds (1 MB / second) Proxy CPU Usage: ~7% on the average Client CPU Usage: ~20% on the average 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" JMS Performance: Transit Delay and Std Deviation (NaradaBrokering & SonicMQ) Transit Delay Standard Deviation NaradaBrokering is Java Message Service (JMS) compliant Applications ported include Anabas – JMS based distance education system. OKC – Online Knowledge Center project at IU 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

NaradaBrokering and JXTA Narada-JXTA provides JXTA guaranteed long distance delivery Request/Response Present if targeted at Particular peer Narada JXTA Event 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

Narada Performance Web Service Performance measurements are used by Links in Reconfiguring Connectivity between nodes Deciding underlying transport protocol Determining possible filtering Each node determines performance of links of which it is endpoint Individual node web services are aggregated as another Web Service Probably should replace by a more sophisticated measurement package Factors measured include Transit delays, bandwidth, Jitter, Receiving rates. Performance measurements are Spaced out at increasing intervals for healthy channels. Factors selectively measured for unhealthy channels. No repeated measurements of bandwidth for example. Injected into Narada network as XML events Administrative Interface 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

Narada Link Firewall Architecture Control Channel (TCP) Specifics tunnel destination, parameters. [ SOAP port 80 ] Config Specifics default tunnel destination, parameters. Non-Firewall Proxy CTL SSL Tunnel Server Proxy UDP TCP SSL Tunnel Client Proxy SSL Lib API TCP UDP Firewall Proxy Fake SSL Impl. JSSE Impl. WinINET Impl. Proxy Detection API Required for MS Authentication support. Text Config WinINET Detection Required for Proxy location detection Narada Link Firewall Architecture 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" Start UDP Works Stream Media Types Connection Complete Doesn’t Work Start TCP Works Reliable Data Stream Doesn’t Work Windows ? WinINET Try SSL first then HTTP Works NaradaBrokering Link Transport Firewall Heuristic Doesn’t Work Try SSL Over HTTPS Proxy Does HTTPS Proxy Exist Works Yes Doesn’t Work Try HTTP Over HTTP Proxy Does HTTP Proxy Exist Works Yes Doesn’t Work “Fake” SSL Over Direct Try SSL Over Direct Try HTTP Direct Works Doesn’t Work Doesn’t Work Works 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

uri="http://www.naradabrokering.org" email="gcf@indiana.edu" Futures Redo performance with insight of this workshop – we desperately need a distributed testbed to support scaling tests Test Scaling with construction of dynamic NB broker network (what is best topology? Hypercube like?) Understand clients per broker for various applications including large multimedia sessions Complete transport/performance infrastructure Develop nifty “load balancing heuristics” Implement Web Service message-based Security Understand better how to implement distributed XML database Develop NB to support federation of Grid Services Generalize Filtering mechanism Download from http://www.naradabrokering.org 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

Architecture of Message Layer Need to optimize not only routing of particular messages but classic publish/subscribe problem of integrating different requests with related topics (subscribe to sports/basketball/lakers and sports) Related to Akamai, AOL … caching and Server optimization problem Hypercube of NB Brokers (logical not physical) N≈100 for Distance Education Scale to billions of grid nodes? 1-> N Grid Nodes 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

NaradaBrokering as an XML database NB is inevitably a distributed XML database supporting real-time update and access (JMS uses SQL like syntax) Performance data Event Logging Published “properties” of publish/subscribe messages Publish as XML topic; subscribe using XQuery? We use NB as a simple XML database for News groups and other “XML nuggets” see http://www.xmlnuggets.org XML Instance==Message; what is difference between a message and a database record except performance but Moore’s law is rewriting rules here Subscribe a real database (Oracle) to all topics for reliable storage 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

Federation of Services If you have a service – Notification (as with Grid or JXTA advertisements), Search, Scheduling, Audio-Video conferencing …. With a standard which client and server components Then can build a “server” that services all clients Alternatively can hierarchically consider collection of existing Server-client domains IBM or Globus OGSA islands Sun Grid Engine Schedulers Polycom/Access Grid A/V sessions Enterprise Search Engines Federation links islands together JXTA Search has XML specified federation approach – will try to include and generalize as a NaradaBrokering federation framework DoD High Level Architecture HLA does this for simulation NB Hub 9/17/2018 uri="http://www.naradabrokering.org" email="gcf@indiana.edu"