3. Communication in distributed system

3.1 Introduction Interprocess communication is at the heart of distributed systems Distributed Application Communication middleware Low-level message passing underlying network IP address port,... Name,...

Middleware Protocols An adapted reference model for networked communication. 2-5 P69

Distributed Middleware Communication middleware: RPC, RMI, MPI –name DCE, CORBA –message-oriented –various communication forms Authentication, authorization protocols Distributed commit protocols

Client-Server TCP Normal operation of TCP.Transactional TCP. 2-4 P65

Two issues for communication mechanism: How indicate destination? How many methods (forms)? S T Q P

Indicate sending and receiving processes 1) ex, 2) Unique name ex, name( ) S T Q P 1. Addressing

…... Domain name IP address Type A cs.nju.edu.cn A …… …... DNS FTPd web server ftp://nju.edu.cn

3) Mailbox (queue, indirect) multi-in, multi-out 2. Methods 1) Synchronous/asynchronous sender receiver mailbox

P1 P2 send receive waiting receive waiting send Synchronous: sender will be blocked if receiver is not ready send( ), recv( ); RPC

P1 P2 send receive Internet? Asynchronous UDP, Asynchronous RPC

2) Transient/persistent Persistent communication: A message is stored by communication system as long as it takes to deliver it. Neither the sender nor the receiver need to be up. Ex, Synchronous wait, no broadcast safe, simple Asynchronous buffer full,not safe parallelism, broadcast Defects merits Comparison P100-1

Persistence in Communication P100

Transient communication: A message is stored by communication system as long as the sending and receiving applications are executing Ex, send, recv in Socket Question: difference (relation) between 1) and 2) transient synchronous persistent asynchronous As to: 1) sender/receiver; 2) message send, recv; RPC UDP; asyn RPC

Two successive messages have a temporal relationship. Ex, movie, audio stream 3) Stream-oriented communication

P1P2 Proc G(u,v) Call P2.G(x,y) node 1 node 2 ……… RPC P Worm.msBlast

1. Conventional procedure call Node 1 process Procedure Q(u,box,v) { … { … Q(i,buf,j) … … } return addr. i buf j sp stack result

2. How does RPC work? node 1 node 2 process Procedure Q(u,box,v) { … { … Q(i,buf,j) … … } } ? stack return sp return addr 2 i buf j sp result return addr.1 buf j i proc Q int i ? buf int j result

Original procedure call: Now push return addr.1 push i Q(i,buf,j) push buf push j goto Q goto c_stub take result

C_stub pack parameters & name send them to S_stub receive result unpack result return S_stub receive message unpack message push return addr. goto Q( ) take & pack result send it to C_stub Q( ) { } transform

Steps of a Remote Procedure Call Client procedure calls client stub in normal way Client stub builds message, calls local OS Client's OS sends message to remote OS Remote OS gives message to server stub Server stub unpacks parameters, calls server Server does work, returns result to the stub Server stub packs it in message, calls local OS Server's OS sends message to client's OS Client's OS gives message to client stub Stub unpacks result, returns to client P72

Passing Value Parameters Steps involved in doing remote computation through RPC 2-8 P74

Issues to be handled Machine may be not identical Execute in different spaces Machine may crash 3. Parameter passing Implementation should consider: Different types of computers Pointer or call by reference

1) Deal with different types of computers 5Jill (a) Original message on the Pentium (b) The message after receipt on the SPARC (c) The message after being inverted. The little numbers in boxes indicate the address of each byte P75

How represent parameters? –In intermediate form –Indicating in type i type i type j Message should include parameters and their type 2) Deal with call by reference proc Q int i char buf int j

Node1Node2 Main( ) { int B[100] proc write(name,buf,len) … { int len, buf[ ]; write(f1,B,100)… … } proc read(name, buf, len) { int len, buf[ ]; read(f2,B,100 ) … } B 5200

read write

proc write(in char name, in int buf[], in int len); proc read(in char name, out int buf[], in int len); –Input parameter  deliver the array –Output parameter  needn’t 4. Where stubs come from? Write server specification (interfaces) including:specification Server name, version, list of proc s Stub compiler generates c_stub & s_stubgenerates

Specification of file server Specification of file_server, version 3.1: long read(in char name[MAX_PATH], out char buf[BUF_SIZE], in long bytes, in long position); long write(in char name[MAX_PATH], in char buf[BUF_SIZE], in long bytes, in long position); int create(in char name[MAX_PATH], in int mode); int delete(in char name[MAX_PATH]); end; return IDL

5. Steps for using RPC write specification compile write server code and link with s_stub write client code and link with c_stub specification C_stub S_stub Stub compiler

The steps in writing a client and a server in DCE RPC. P82

{…… }

6. Dynamic binding --- How client find server 1) Register: name, version, handle(IP,port), unique id 2) Look up: name, version (first time) 3) Reply: handle, unique id 4) Request: unique id, parameters 5) Deregister: name, version, unique id binder clientserver

Advantage: flexible –Server move won’t affect client –Binder can even load if multiple servers support the same interface Disadvantage –Overhead at running time –Binder is bottleneck

Addressing: name Method: synchronous, transient RPC NFS XDR Socket/TLI TCP|UDP IP ARP Driver ICMP FTP Telnet Addressing Method

7. Extended RPC --- asynchronous RPC (a) The interconnection between client and server in a traditional RPC (b) The interaction using asynchronous RPC ABAB 100 P79

A client and server interacting through two asynchronous RPCs 2-13 A client wants to prefetch network addresses of a set of hosts P80

3.3 RMI (or Remote Object Invocation) Common organization of a remote object with client-side proxy. P86

1. Distributed object RMI Compares to RPC: Same: –the steps of call is similar –separating interface from implementation Difference –procedure is a segment of program. –object is a variable of a data type, method is in object.

Object reference compare to pointer. System provides 2. Object references Object references are similar to pointer, but not the same. object reference remote point local object distributed obj

… employee promote() {…} dismiss() {… } hire() {… } Corp server Sale server salesman bonus() {… } advertise cost() {… } personnel 1. IP address ? Object servers P152 personnel.hire( )

Information object reference contains network address of the machine server id or endpoint (i.e.port number) object id P89

May use location server location server clientserver Server can move

Server id instead of server’s endpoint. A daemon listens to a well-known endpoint and knows every server. 3. Implementation of object references server1 server2 keep a table daemon

4. Several remote object invocation 1) CORBA Open, heterogeneous environment 2) DCOM Microsoft world Object C ++ Java C func P( ) How introduce object? chapter 9

3) Java RMI Java environment Distributed objects have been integrated into the language Summary Addressing: name Method: synchronous, transient

3.4 Message-oriented communication 1. Message-oriented transient communication 1) RPC and RMI advantages: hide communication; message-oriented disadvantage: synchronous; both need to be up 2) Recall socket P105 addressing: IP address, port forms: synchronous/asynchronous, transient shortage: low-level, byte-oriented/TCP-IP P99

Process0 Process1 Process5 Process4 Process3 Process2 Process1 Process0 Process2 3) MPI P107 Jupiter World designed for parallel applications; offer efficient communication primitives; for high-speed nets. communicator

Addressing (groupID, processID) Forms support most transient communication forms ?A B; Need for various communication forms Synchronous/asynchronous Transient/persistent 100

Various combinations:P103 Why list so many forms? Where to use? For efficiency. (a) (c) A B (e) send recv (f) RPC 100 (persistent, asynchronous) (transient, asynchronous) (transient, synchronous) (transient, synchronous/asyn)

2. Message-oriented persistent communication 1) Message-queuing model Addressing: name of destination queue Method: persistent, asynchronous 2) Basic idea Application inserts messages in specific queue the messages are forwarded over a series of communication servers eventually the messages are delivered to destination P109

Message-Queuing System The general organization of a message-queuing system with routers P112

Four combinations for loosely-coupled communications using queues P110

3) Basic interface to a queue Install a handler to be called when a message is put into the specified queue. Notify Check a specified queue for messages, and remove the first. Never block. Poll Block until the specified queue is nonempty, and remove the first message Get Append a message to a specified queue Put Meaning Primitive callback P110

4) Implementation All machines in message-queuing system should have a queue manager which manages: queues: source queue, destination queue (name) transmission of messages from S to D mapping queues to network locations

Two kinds of implementations a) Create its own forwarding servers (routers), like multicast gateway + Mbone b) with the aid of Internet routers, like a) Architecture of IBM MQSeries For IBM mainframes, access large scale database Addressing: (destQM, destQ),

Example: IBM MQSeries General organization of IBM's MQSeries message-queuing system P116

Message Transfer The general organization of an MQSeries queuing network using routing tables and aliases. (QA,SQ1)

Why do message-queue systems send messages to queue, instead of directly to receivers? Some applications need this kind of communication form. For example, tasks are sent to a set of processes who perform these tasks. Two ways: a dispatcher + workers workers P143

3.5 Stream-oriented communication 1. Requirements Real life –audio stream (simple stream) –movie (complex stream) Features –time-dependent information. –data stream Stream media? P119

Addressing: or destination name Method: continuous data stream

Requirement for transmission: –timing for continuous data stream –synchronization for complex stream Ex, –Samples in audio stream are played at intervals of 1/44100 sec. –Video and audio sub-streams need synchronize.

Data Stream Setting up a stream between two processes across a network. P122 MP3

An example of multicasting a stream to several receivers. Differentiated services

4. Stream Synchronization Meaning Maintain temporal relation between streams which may be continuous or discrete data stream. Essence Synchronize at the level of data units. What is a data unit? a single audio stream a series of samples a video stream a series of frames

Data unit Ex1, synchronization for stereo audio stream takes place every 1/44100 sec a data unit = a sample Ex2, what about synchronization between an audio stream and a video stream? video frame30Hz audio sample44100Hz sample/frame44100/30=1470 an audio data unit = 1470 samples 1/30 sec

Synchronization mechanism A process executes read (write) operations on several simple streams, ensuring timing and synchronization. Ex, alternating between reading an image and reading a block of audio samples every 33ms sender receiver audio stream video stream

Does sender send two separate streams or a complex stream? A complex stream. Where synchronization should take place, sending or receiving side? Sender: multiplex different streams into a single stream containing all data units, including those for synchronization(timestamp) Receiver: demultiplexes the stream, using the timestamps of each packet

Synchronization Mechanisms (2) The principle of synchronization as supported by high- level interfaces Offer interface to specify the rates

Summary 1. Key issues for designing communication mechanism Addressing: how indicate sender and receiver, LAN, WAN Methods : synchronous/asynchronous persistent/transient stream-oriented communication Middleware Examples?

How does it work? Issues to be handle: type, reference (object?) IDL: is used to write server’ specification C_stub Steps for using RPC S_stub Binder 3. RMI Difference between RPC and RMI Object reference and its features 2. RPC

–address, serverID, objectID CORBA etc 4. Message-oriented communication Why need it? Transient communication: MPI Persistent communication: Message-queuing system

Requirement for transmission Features Synchronization 5. Stream-oriented communication