Distributed (Operating) Systems -Message-oriented Communication- -Stream-oriented Communication- Computer Engineering Department Distributed Systems Course Assoc. Prof. Dr. Ahmet Sayar Kocaeli University - Fall 2015

Communication Issues Message-oriented communication Transient vs Persistent Synchronous vs asynchronous Stream-oriented communication

Message-oriented Transient Communication Many distributed systems built on top of simple message-oriented model Example: Berkeley sockets – transport level sockes

Berkeley Socket Primitives

Message-Passing Interface (MPI) Sockets are designed for network communication (e.g., TCP/IP) Support simple send/receive primitives Abstraction not suitable for other protocols in clusters of workstations or massively parallel systems Need an interface with more advanced primitives Buffering and synchronization Large number of incompatible proprietary libraries and protocols Need for a standard interface Message-passing interface (MPI) Hardware independent Designed for parallel applications (uses transient communication)

MPI Key idea: communication between groups of processes Each endpoint is a (groupID, processID) pair MPI are intent to be used in clusters It is basically used for scientific applications and parallel systems It overcomes some of the TCP communications MPI basically run on UDP Socket based communication is also transient but it is based on synch communication but here MPI also support asynchronous communication

MPI Primitives

Message-oriented Persistent Communication Message queuing systems or Message oriented Middleware (MOM) Support asynchronous persistent communication Intermediate storage for message while sender/receiver are inactive Example application: email Communicate by inserting messages in queues Sender is only guaranteed that message will be eventually inserted in recipient’s queue No guarantees on when or if the message will be read “Loosely coupled communication”

Message-Queuing Model (1)

Basic interface to a Queue Put: append a message to a specified queue Get: block until the specified queue is nonempty and remove the first message Poll: check a specified queue for messages and removes the first . Never block Notify: install a handler to be called when a message is put into the specified queue.

Message-Queuing Model

General Architecture of a Message-Queuing System (2) Queue manager and relays Relays use an overlay network Relays know about the network topology and how to route

Message Brokers Message broker: application level gateway in MQS Convert incoming messages so that they can be understood by destination (format conversion) Also used for pub-sub systems

IBMʼs WebSphere MQ Queue managers manage queues Connected through message channels Message channel agent (MCA) Checks queue, wraps into TCP packet, send to receiving MCA

Stream Oriented Communication Communication as discussed so far has concentrated on exchanging more-or-less independent and complete units of information. It does not matter at what particular point in time communication takes place. Timing has no effect on correctness. Continuous media: the temporal relationship between different data items are fundamental to correctly interpreting what the data actually means. Audio and video streams Discrete media: temporal information is not important, ex. text and still images

Stream Oriented Communication Timing is crucial in certain forms of communication Examples: audio and video (“continuous media”) 30 frames/s video => receive and display a frame every 30ms If there is increasing delay there might be some frames dropped but human eye can see it TCP does not guarantee short time latency TCP enables retransmitting but frame should be displayed at right time and right place Retransmit does not make sense here UDP is actually better protocol for streaming

Data Streams A sequence of data units Discrete vs. continuous data streams TCP/IP connections – discrete data streams – file transfers Playing an audio – continuous data streams Isochronous communication : Data transfers have a maximum bound on end-end delay and jitter Push mode: no explicit requests for individual data units beyond the first “play” request Streaming live data vs. streaming stored data

To Push or Pull ? Client-pull architecture Server-push architecture Clients pull data from servers (by sending requests) Example: HTTP Pro: stateless servers, failures are easy to handle Con: limited scalability Server-push architecture Servers push data to client Example: video streaming, stock tickers Pro: more scalable, Con: stateful servers, less resilient to failure When/how-often to push or pull?

Examples

Streaming Live stream based comm, real time issues come up, ex- video conferencing, skype Stored stream based comm, latency does not matter, ex- youtube Live streaming has even more stringent constraints Unicast: 1 sends for 1 receiver Multicast: 1 send for n receiver

Streams and Quality of Service Time-dependent and other requirements are specified as quality of service (QoS) Requirements/desired guarantees from the underlying systems Application specifies workload and requests a certain service quality Contract between the application and the system Properties for Quality of Service: The certain loss rate The required bit rate at which data should be transported. The maximum delay until a session has been set up The maximum end-to-end delay . The maximum delay variance, or jitter. The maximum round-trip delay. The quality of video

Specifying QoS: Token Bucket The principle of a token bucket algorithm Parameters (rate r, burst b) Rate is the average rate, burst is the maximum number of packets that can arrive simultaneously

Specifying QoS: Leaky Bucket

Enforcing QoS (1) Observation: There are various network-level tools, such as differentiated services by which certain packets can be prioritized. Also: use buffers to reduce jitter

Enforcing QoS (2) Problem: How to reduce the effects of packet loss (when multiple samples are in a single packet)? Solution: simply spread the samples:

Stream synchronization Multiple streams: Audio and video; layered video Need to sync prior to playback Timestamp each stream and sync up data units prior to playback Sender or receiver? App does low-level sync 30 fps: image every 33ms, lip-sync with audio Use middleware and specify playback rates

Synchronization Mechanism Problem: Given a complex stream, how do you keep the different substreams in synch? Example: Think of playing out two channels, that together form stereo sound. Difference should be less than 20–30 μsec! Alternative: multiplex all substreams into a single stream, and demultiplex at the receiver. Synchronization is handled at multiplexing/demultiplexing point (MPEG).

Multicasting Multicast: sending data to multiple receivers Network- and transport-layer protocols for multicast bogged down at the issue of setting up the communication paths to all receivers. Group communication IP multicast versus application-level multicast Construct an overlay multicast tree rooted at the sender Send packet down each link in the tree Issues: tree construction, dynamic joins and leaves

Application-Level Multicasting IP multicasting works in IP layer. Why not just use that? Often difficult to configure, requires too much administrative support, agreement, etc. Application-level seeks to address those issues. Peer-to-peer communication using structured overlays can use application-layer protocols to support multicast The overlay network is used to disseminate information to members

Application-Level Multicasting Essence: Organize nodes of a distributed system into an overlay network and use that network to disseminate data. Two possible structures: Tree: unique path between every pair of nodes Mesh: multiple neighbors ensure multiple paths (more robust)