1. Distributed (Operating) Systems -Message-oriented Communication- -Stream-oriented Communication-
Computer Engineering Department
Distributed Systems Course
Assoc. Prof. Dr. Ahmet Sayar
Kocaeli University - Fall 2015

2. Communication Issues Message-oriented communication
Transient vs Persistent
Synchronous vs asynchronous
Stream-oriented communication

3. Message-oriented Transient Communication
Many distributed systems built on top of simple message-oriented model
Example: Berkeley sockets – transport level sockes

4. Berkeley Socket Primitives

5. Message-Passing Interface (MPI)
Sockets are designed for network communication (e.g., TCP/IP)
Support simple send/receive primitives
Abstraction not suitable for other protocols in clusters of workstations or massively parallel systems
Need an interface with more advanced primitives
Buffering and synchronization
Large number of incompatible proprietary libraries and protocols
Need for a standard interface
Message-passing interface (MPI)
Hardware independent
Designed for parallel applications (uses transient communication)

6. MPI Key idea: communication between groups of processes
Each endpoint is a (groupID, processID) pair
MPI are intent to be used in clusters
It is basically used for scientific applications and parallel systems
It overcomes some of the TCP communications
MPI basically run on UDP
Socket based communication is also transient but it is based on synch communication
but here MPI also support asynchronous communication

7. MPI Primitives

8. Message-oriented Persistent Communication
Message queuing systems or Message oriented Middleware (MOM)
Support asynchronous persistent communication
Intermediate storage for message while sender/receiver are inactive
Example application:
Communicate by inserting messages in queues
Sender is only guaranteed that message will be eventually inserted in recipient’s queue
No guarantees on when or if the message will be read
“Loosely coupled communication”

9. Message-Queuing Model (1)

10. Basic interface to a Queue
Put: append a message to a specified queue
Get: block until the specified queue is nonempty and remove the first message
Poll: check a specified queue for messages and removes the first . Never block
Notify: install a handler to be called when a message is put into the specified queue.

11. Message-Queuing Model

12. General Architecture of a Message-Queuing System (2)
Queue manager and relays
Relays use an overlay network
Relays know about the network topology and how to route

13. Message Brokers Message broker: application level gateway in MQS
Convert incoming messages so that they can be understood by destination (format conversion)
Also used for pub-sub systems

14. IBMʼs WebSphere MQ Queue managers manage queues
Connected through message channels
Message channel agent (MCA)
Checks queue, wraps into TCP packet, send to receiving MCA

15. Stream Oriented Communication
Communication as discussed so far
has concentrated on exchanging more-or-less independent and complete units of information.
It does not matter at what particular point in time communication takes place.
Timing has no effect on correctness.
Continuous media: the temporal relationship between different data items are fundamental to correctly interpreting what the data actually means.
Audio and video streams
Discrete media: temporal information is not important, ex. text and still images

16. Stream Oriented Communication
Timing is crucial in certain forms of communication
Examples: audio and video (“continuous media”)
30 frames/s video => receive and display a frame every 30ms
If there is increasing delay there might be some frames dropped but human eye can see it
TCP does not guarantee short time latency
TCP enables retransmitting but frame should be displayed at right time and right place
Retransmit does not make sense here
UDP is actually better protocol for streaming

17. Data Streams A sequence of data units
Discrete vs. continuous data streams
TCP/IP connections – discrete data streams – file transfers
Playing an audio – continuous data streams
Isochronous communication : Data transfers have a maximum bound on end-end delay and jitter
Push mode: no explicit requests for individual data units beyond the first “play” request
Streaming live data vs. streaming stored data

18. To Push or Pull ? Client-pull architecture Server-push architecture
Clients pull data from servers (by sending requests)
Example: HTTP
Pro: stateless servers, failures are easy to handle
Con: limited scalability
Server-push architecture
Servers push data to client
Example: video streaming, stock tickers
Pro: more scalable, Con: stateful servers, less resilient to failure
When/how-often to push or pull?

19. Examples

20. Streaming
Live stream based comm, real time issues come up, ex- video conferencing, skype
Stored stream based comm, latency does not matter, ex- youtube
Live streaming has even more stringent constraints
Unicast: 1 sends for 1 receiver
Multicast: 1 send for n receiver

21. Streams and Quality of Service
Time-dependent and other requirements are specified as quality of service (QoS)
Requirements/desired guarantees from the underlying systems
Application specifies workload and requests a certain service quality
Contract between the application and the system
Properties for Quality of Service:
The certain loss rate
The required bit rate at which data should be transported.
The maximum delay until a session has been set up
The maximum end-to-end delay .
The maximum delay variance, or jitter.
The maximum round-trip delay.
The quality of video

22. Specifying QoS: Token Bucket
The principle of a token bucket algorithm
Parameters (rate r, burst b)
Rate is the average rate, burst is the maximum number of packets that can arrive simultaneously

23. Specifying QoS: Leaky Bucket

24. Enforcing QoS (1)
Observation: There are various network-level tools, such as differentiated services by which certain packets can be prioritized.
Also: use buffers to reduce jitter

25. Enforcing QoS (2)
Problem: How to reduce the effects of packet loss (when multiple samples are in a single packet)?
Solution: simply spread the samples:

26. Stream synchronization
Multiple streams:
Audio and video; layered video
Need to sync prior to playback
Timestamp each stream and sync up data units prior to playback
Sender or receiver?
App does low-level sync
30 fps: image every 33ms, lip-sync with audio
Use middleware and specify playback rates

27. Synchronization Mechanism
Problem: Given a complex stream, how do you keep the different substreams in synch?
Example: Think of playing out two channels, that together form stereo sound. Difference should be less than 20–30 μsec!
Alternative: multiplex all substreams into a single stream, and demultiplex at the receiver. Synchronization is handled at multiplexing/demultiplexing point (MPEG).

28. Multicasting Multicast: sending data to multiple receivers
Network- and transport-layer protocols for multicast bogged down at the issue of setting up the communication paths to all receivers.
Group communication
IP multicast versus application-level multicast
Construct an overlay multicast tree rooted at the sender
Send packet down each link in the tree
Issues: tree construction, dynamic joins and leaves

29. Application-Level Multicasting
IP multicasting works in IP layer.
Why not just use that?
Often difficult to configure, requires too much administrative support, agreement, etc.
Application-level seeks to address those issues.
Peer-to-peer communication using structured overlays can use application-layer protocols to support multicast
The overlay network is used to disseminate information to members

30. Application-Level Multicasting
Essence: Organize nodes of a distributed system into an overlay network and use that network to disseminate data.
Two possible structures:
Tree: unique path between every pair of nodes
Mesh: multiple neighbors ensure multiple paths (more robust)