1. Distributed Systems and Message Passing

2. 2 Context (Inter Process Communication, IPC) So far, we have studied concurrency in the context of Multiprogramming, where pseudo-parallel concurrent processes share the same CPU by context switching. But concurrent operation is equally important in real parallel systems, with > 1 CPU. So far also, we have concentrated on cases where process interaction has been about synchronisation or about safely sharing resources (mutual exclusion). Mechanisms to control these interactions (sem4’s, conditional critical regions, monitors) require that all processes share the same global memory space. There are two general architectures for real parallel systems: (A) Multiprocessing systems, where CPU’s are closely coupled and share the same memory space, e.g. via a bus, as below: CPU 1 CPU 2 CPU n memory …

3. 3 Context (2) –(B) MultiComputer systems, i.e. self contained computers forming a Distributed system, as below: –Distributed systems could in the same box, or be very far apart (eg using the Internet for communication. But there is no shared memory space, so communication using sem4’s etc is not possible - instead, messages have to sent via the interconnection system –In all these cases, it is also very common for processes to share actual data with each other i.e. to interact in a co-operative way, actively exchanging information. CPU 1 mem 1 CPU 2 mem 2 CPU n mem n... Inter-connection system

4. 4 IPC - Shared Memory vs Messages Shared Memory Message Passing Data structure X Process A Process B ---- read X ---- write X ---- X in shared memory ---- receive X ---- send X ---- Data structure X message Process A Process B

5. 5 Messages Are the only means of inter-process interaction in Distributed systems –can be used for synchronisation, to provide mutual exclusion, and other forms of inter-process control –can carry data (eg email) or code (Java Applets) –can be fixed or variable length –usually will have some kind of header, possibly describing type of message length of message id’s of sender, intended recipient Reliability issues and problems –Communications problems corruption - use of clever error check codes, auto correction, re-send loss of connection –Need for acknowledgement from Receiver (ACK), so that sender can re-transmit if no ACK. But the ACK message itself could be faulty!

6. 6 Mutual Exclusion in Distributed Systems It is sometimes said that ME is automatic in Distributed Systems. Because the separate processes don’t share each other’s memory spaces any data in one memory space is automatically excluded to other processes. This argument is misleading, since it only applies to resources in the form of data stored in memory. Many concurrent systems (including many distributed systems) will be embedded systems, with lots of interfaces to real sensors, machines etc. So Distributed Systems can easily share critical resources which require mutually exclusive access, as in the example below. --- A: open valve x --- B: close valve x --- Valve x (radioactive water) Process 1 has an interface to valve x P2 Process 2 has an interface to valve x P1 Message passing between P1 and P2 has to provide mutual exclusion between critical sections A and B

7. 7 Message passing fundamentals Basic idea: Processes send and receive messages via Channels. –Send(message) –Receive(message) Some logical questions –who to?, who from? - this is the question of addressing. –does the sender wait or carry on? –what happens to Receive() if there is no Send()? –how many processes? –what kind of logical links 1 n (broadcast); n 1; n n –buffering in the channel? –capacity of the link (pigeon or optical fibre)? –message size? –uni or bi-directional channels?

8. 8 Direct Communication (Direct naming) Here the main idea is that there is no intermediate ‘agent’ between the processes - so there must exist a direct channel between them. –Symmetrical case - processes P and Q are explicitly identified Send (P, msg) Receive (Q, msg) only involves 2 processes, which need to ‘know’ each other. Not very flexible, but very secure! –Asymmetrical case - only sender names receiver Send (P, msg) Receive (id, msg) ie receive from any process sending on the channel and find out the sender’s id when the message arrives (in the message header, or in the message itself) Typical case is the client-server model

9. 9 Sender-Receiver Synchronisation (1) Synchronous - like a Fax machine –sender blocks until message received (implies that the channel provides an ACK) –receiver blocks until message arrives –so both processes make a Rendezvous - (same place at the same time) –at most 1 outstanding message –good acknowledgement - sender knows what is happening –no good for servers or for broadcast –not easy in real time systems

10. 10 Sender-Receiver Synchronisation (2) Asynchronous - like a post box - ‘send and forget’ –this implies that there is a buffer in the channel –send is non-blocking, i.e. the sender gets on with the next thing –receive can be blocking or non-blocking (i.e. can stay awake and poll for the message) –ACK problem - mechanism will have to separate. Also the receiver doesn’t know the current state of the sender - which might even have terminated. –messages could pile up/overflow channel the buffering capacity - and the sender may not know about this.

11. 11 Indirect Communication (Naming), Mailboxes The key idea here is that there is a separate agent (the mailbox), which acts as an independent intermediary (and a buffer which can store messages) between processes. Processes can communicate if they can access the same mailbox. mailboxes have to be created (by an operating system call) and uniquely identified. more than two processes can be involved - many can send to a mailbox, many can receive. In fact this is quite common - if only two processes are involved, they could communicate directly, using a private channel between the two of them. identification issues (who is the message from, who is it for) have to be resolved within the messages, either in their headers or in their content. The mailbox is neutral - it doesn’t know who the messages are from, or for. This is very different from a channel,which is set up between known end points. This implies that there have to be agreed protocols between processes sending to and receiving messages from mailboxes. mailboxes can store uncollected messages (buffering), but can also get full. Receivers at the mailbox may have to wait for messages to appear, so mailboxes have to keep associated queues of waiting processes - these are normally FIFO, but could be priority based (e.g. in real time systems). Processes can communicate via more than one mailbox, if there are logically separate things to communicate about. A mailbox with many senders, but only one receiver, is a special case called a Port

12. 12 The copying issue, i.e. what actually gets sent? Either the whole message, i.e. send by value all the actual data bytes of the message. –Clearly this is possible in any system which allows messages to be passed from one process to another, either directly or indirectly, and whether it is loosely or closely coupled. It is simple, but slow if the message is very big. Or a pointer to the actual data of the message can be sent, i.e. send by reference. –This implies a shared address space, so is restricted to closely couple systems - so it is not an option in Distributed systems. –If there is a common address space, this will be faster than send by value,as well as more flexible and more amenable to Real Time operation.

13. 13 Control and Synchronisation using mailboxes Both Mutual Exclusion and the Producer/Consumer problem can be implemented in distributed systems using mailboxes. There is a separate handout showing these examples.

14. 14 Control and Synchronisation using messages As in our earlier discussions about sem4’s, powerful inter-process tools don’t guarantee safe and live designs. In particular, liveness (e.g. deadlock) problems can just as easily arise in distributed systems as in closely coupled systems. Consider the following example: 2 processes P1 and P2 want to exchange integer values by sending them as messages through two uni-directional channels A and B. A possible annotation is shown below: Since the sends are synchronous, P1 sends x1 and waits for an ACK from channel B, at the same time as P2 sends x2 and waits for an ACK from channel A. Deadlock! So the design has to include an asynchronous send in at least one of the processes P1 int x1=1; y1 syncsend (channel B, x1) ---- do some processing receive(channel A, y1) P2 int x2=2; y2 syncsend (channel A, x2) ---- do some processing receive(channel B, y2)

15. 15 Remote Procedure Call (RPC) Message passing is about single sender, single receiver, one way messages through channels. This is not wonderful for Client-Server applications – these would need at least two channels (one for each direction of communication) and many more for a server with many clients. RPC is a high level mechanism designed very much with Client – Server applications in mind. RPC allows parallel execution on distributed systems by allowing a process to call a procedure to execute on a different CPU (ie remote somewhere on the distributed system). So there has to be a distributed operating system environment which allows a call from one processor to another. Essentially the local process can use a procedure elsewhere as conveniently if it were local. All the necessary networking and marshalling of data (passing parameters and returning values) is done transparently by the environment. –RPC is a system construct, not a language construct. –Java provides RMI, which is an extended version of RPC, providing the language links to implement calling methods on other parts of a network –A RPC call is executed on behalf of the calling process by a surrogate process created on the other processor (by the the operating system) for this purpose. This is the idea of the agent again, which we have seen many times.

16. 16 Remote Procedure Call 2 RPC works by defining modules to contain the exported operations. A typical RPC module has the form: module mname headings for exported operations, eg op thisop(formal params) [returns result] body variable declarations; initialisation code; actual procedures to implement exported operations, eg proc thisop(formal identifiers) returns result identifier local supporting procedures (background processes) end mname A process in one module can then call a procedure in another module by executing: call mname.thisop(parameters)

17. 17 Remote Procedure Call 3 How does the call actually work? Since the two modules (module A where the call is made and module B where the servicing procedure is) will be in different address spaces, a new surrogate (agent) process has to be created in B to actually make the call. The surrogate process is created by the environment and so is invisible. It exchanging messages with the caller in module A to deal with parameters and returns. The surrogate executes the procedure that implements thisop (as we called it on the previous slide), while the calling process in module A waits for the return message. Once the transaction is finished the surrogate process terminates. Logically, it is as if the caller itself was executing the call, so the synchronisation between the two is implicit. Multiple remote calls for the same thisop would require local mutual exclusion mechanisms (e.g. sem4's) within the remote module.

18. 18 Remote Procedure Call: An example For a circular queue example, a module might look like: module Queue op enter (typeI), get (result TypeI); body typeT buf[n]; typeI item; int front = 1, back = 1, count = 0; proc enter(item) { if (count < n) { buf[back] := item; back := (back+1) mod n; } else sort out overflow } // of enter proc get(item) { if (count>0) { item := buf[front] mod n; count:= count-1; } else sort out underflow } // of get end of Queue module