1. Lecture 3 Page 1 CS 239, Spring 2001 Interprocess Communications in Distributed Operating Systems CS 239 Distributed Operating Systems April 9, 2001

2. Lecture 3 Page 2 CS 239, Spring 2001 Introduction Message passing communications –Message passing paradigms –Pipes and sockets –Multicast and broadcast Request/reply communications –RPC –Secure RPC

3. Lecture 3 Page 3 CS 239, Spring 2001 Message Passing Communications By definition, distributed systems require IPC Most distributed systems connected only by network So most communications must be by message –Other IPC abstractions built on messages

4. Lecture 3 Page 4 CS 239, Spring 2001 Levels of Abstraction of Communication Interprocess Communication Network OS Communications Network Transactions Request/Reply (RPC) Message Passing Transport Connection Packet Switching

5. Lecture 3 Page 5 CS 239, Spring 2001 Basics of Messages Messages are collections of data objects –Header –Payload At message level, sender and receiver regarded as peers –Neither is in complete control

6. Lecture 3 Page 6 CS 239, Spring 2001 Steps in Sending a Message Sending process formats message Invokes local OS to send it Local OS (usually) copies it into kernel space Local OS transfers message to local network interface Network interface passes message to network hardware

7. Lecture 3 Page 7 CS 239, Spring 2001 Steps in Receiving a Message Message delivered from network to receiver’s network interface Interface generates interrupt for OS OS copies message to buffer OS either interrupts receiver or waits for receive system call Message (usually) copied from OS buffer to receiver process

8. Lecture 3 Page 8 CS 239, Spring 2001 Moving the Message OS Sender Receiver 2. Copy message into OS buffer 3. Put the message in the network interface 1. Create the message at the sender send() 5. Transfer message to network interface 6. Transfer to OS buffer receive() 7. Transfer the message to receiver process 4. Transfer message over network

9. Lecture 3 Page 9 CS 239, Spring 2001 Basic Message Primitives send(destination, message) –Transfer contents of message to location specified by destination receive(source,message) –Accept a message from source and put it in location specified by message

10. Lecture 3 Page 10 CS 239, Spring 2001 What is a Source/Destination? Complete specification of identity of a remote process Typically, a process name –Often including a machine name Often with some specification of an internal address within the process

11. Lecture 3 Page 11 CS 239, Spring 2001 Common Source/Destination Naming Options Process name Link Mailbox Port

12. Lecture 3 Page 12 CS 239, Spring 2001 Process Name Varies depending on system specifics Possible to have global, transparent process namespace More commonly, specify hosting node Is this a good idea if processes move? If only component of name, all messages to a process share one queue

13. Lecture 3 Page 13 CS 239, Spring 2001 Symmetric Process Naming A B send(B,message1) receive(A,&message) send(A,message) receive(B,&message)

14. Lecture 3 Page 14 CS 239, Spring 2001 Assymmetry in Naming Sender assumed to know receiver’s address Sometimes, receivers don’t know sender’s address ahead of time –E.g., server handling multiple clients Sometimes receive() fills source into variable, rather than specifying value

15. Lecture 3 Page 15 CS 239, Spring 2001 Asymmetric Process Naming A B C send(C,message) receive(&sender,&message) send(C,message) receive(&sender,&message)

16. Lecture 3 Page 16 CS 239, Spring 2001 Source/Destination Naming by Links Sometimes we want multiple communications paths between pairs of processes –Connections We can name these connections as sources and destinations –Links

17. Lecture 3 Page 17 CS 239, Spring 2001 Characteristics of Links Unidirectional Multiple links allowed between two processes Each link independently nameable Similar to multiple process entry points Dynamically created and destroyed Direct communications between peers

18. Lecture 3 Page 18 CS 239, Spring 2001 Links AB link1 link2 link3 send(link1,message) create_link(B) create_link(A) send(link2,message)send(link3,message)

19. Lecture 3 Page 19 CS 239, Spring 2001 Source/Destination Naming by Mailboxes Links aren’t appropriate for all cases Sometimes sender doesn’t care about identity of receiver –E.g., obtaining service from one of pool of servers Or receiver doesn’t care about identity of sender Mailboxes don’t require matching one sender to one receiver

20. Lecture 3 Page 20 CS 239, Spring 2001 Mailbox Characteristics Multiple senders can put messages into the same mailbox Multiple receivers can retrieve messages from the same mailbox Varying semantics about how each of these works

21. Lecture 3 Page 21 CS 239, Spring 2001 Mailboxes Mailbox send(mailbox,message) receive(mailbox,&message) send(mailbox,message) receive(mailbox,&message)

22. Lecture 3 Page 22 CS 239, Spring 2001 Message Synchronization and Buffering Sending and receiving a message requires some synchronization –When is it OK to send? –When is it OK to receive? –When is the process complete? –When can the system free various buffers?

23. Lecture 3 Page 23 CS 239, Spring 2001 Blocking and Non-blocking Send and Receiver Blocking send blocks the sender Blocking receive blocks the receiver For how long? –Some options here Non-blocking versions allow send and receive to go ahead immediately

24. Lecture 3 Page 24 CS 239, Spring 2001 Blocking at the Receiver In most systems, receiver blocks Must wait for message to arrive before proceeding Sometimes, only blocks from moment of message arrival until message is in receiver buffer Sometimes from point when receiver needs message

25. Lecture 3 Page 25 CS 239, Spring 2001 Blocking on Arrival A B Interrupt! Block!

26. Lecture 3 Page 26 CS 239, Spring 2001 Blocking on Demand A B receive(A,&message) Block!

27. Lecture 3 Page 27 CS 239, Spring 2001 Common Options for Send Blocking Non-blocking send Blocking send Reliable blocking send Explicit blocking send Request and reply

28. Lecture 3 Page 28 CS 239, Spring 2001 Non-Blocking Send A B Block!

29. Lecture 3 Page 29 CS 239, Spring 2001 Blocking Send A B Block!

30. Lecture 3 Page 30 CS 239, Spring 2001 Reliable Blocking Send A B Block! ACK

31. Lecture 3 Page 31 CS 239, Spring 2001 Explicit Blocking Send A B Block! ACK

32. Lecture 3 Page 32 CS 239, Spring 2001 Request and Reply A B Block!

33. Lecture 3 Page 33 CS 239, Spring 2001 Buffering Any delay in duration of blocking implies a buffering capability –In the sending kernel –In the network –In the receiving kernel Buffering allows multiple simultaneous sends –Also provides reliability

34. Lecture 3 Page 34 CS 239, Spring 2001 Pipes and Sockets A common OS features providing message communications The interface between applications and the network Hides many details of the network

35. Lecture 3 Page 35 CS 239, Spring 2001 Pipes Originally built as IPC for single processor system Basically a FIFO byte stream buffer Bytes fed in one end Bytes read out the other end In the same order

36. Lecture 3 Page 36 CS 239, Spring 2001 Pipe Details Once set up, treated like a file Typically, one process writes, another reads –Usually parent and child processes Send is non-blocking –Assuming local buffer space available Receive is blocking –Receiver explicitly asks to receive bytes –And waits for them to arrive

37. Lecture 3 Page 37 CS 239, Spring 2001 Pipes, Buffering, and Blocking Pipes are implemented in the kernel by setting up a buffer –Of fixed size When writer fills the buffer, further writes block When reader asks for bytes from empty buffer, reads block

38. Lecture 3 Page 38 CS 239, Spring 2001 Named Pipes Normal pipes only usable by processes that share file descriptors –Usually parents and children Pipes can have names attached –In file system namespace Allows any process to name either end of pipe Implemented with kernel buffers or file system

39. Lecture 3 Page 39 CS 239, Spring 2001 Pipes and Distributed Processing Pipe implementations assume a shared buffer space Named pipes assume a shared namespace –And shared FS or buffer space Generally, distributed systems might not have these Can we provide a similar mechanism for that environment?

40. Lecture 3 Page 40 CS 239, Spring 2001 Sockets A general mechanism for distributed systems IPC Subsumes pipes Sets up communication channel named by two unique endpoints

41. Lecture 3 Page 41 CS 239, Spring 2001 Creating Sockets Both communicating processes must issue socket() calls Creating a logical communications endpoint (LCE) –Local to creating process Must bind LCE to a physical communications endpoint (PCE)

42. Lecture 3 Page 42 CS 239, Spring 2001 Example of Creating Sockets A B sA = socket(domain, type, protocol) domain describes protocol family to use type describes high-level characteristics of communications protocol specifies which protocol in the family to use sB = socket(domain, type, protocol)

43. Lecture 3 Page 43 CS 239, Spring 2001 Binding Sockets A PCE is a network host address plus a transport port pair –Network host address space is global –Transport port pair numbers generated locally The bind() system call performs the bindings

44. Lecture 3 Page 44 CS 239, Spring 2001 Example of Binding Sockets A B bind(sA,name,namelen) bind(sB,name,namelen) Now A can send a message to B using the socket

45. Lecture 3 Page 45 CS 239, Spring 2001 Connected Sockets Bound sockets can be used for connectionless message sending Sending process must know remote PCE, though Connecting sockets allows send calls without specifying remote PCE Typically, for servers with well-known ports

46. Lecture 3 Page 46 CS 239, Spring 2001 Broadcast and Multicast Methods for sending the same message to more than one destination Broadcast typically sends to everyone on the network Multicast typically sends to chosen set of recipients

47. Lecture 3 Page 47 CS 239, Spring 2001 Best Efforts Multicast Guarantees delivery to at least one member of the group Useful when trying to obtain service from set of interchangeable servers

48. Lecture 3 Page 48 CS 239, Spring 2001 Reliable Multicast Guarantees delivery to all servers –Or none of them Useful for distributing shared information to cooperating processes If several reliable multicasts going on simultaneously, participants may see messages in different orders

49. Lecture 3 Page 49 CS 239, Spring 2001 Handling Failures in Multicast Sender expects some feedback from each recipient –Perhaps explicit acknowledgement If not all received, re-multicast –Or send to just those who didn’t acknowledge Failure of sender harder to deal with

50. Lecture 3 Page 50 CS 239, Spring 2001 Multicast Message Ordering FIFO: Messages from single source delivered in same order as sent Causal order: Causally related messages from different sources delivered in causal order Total order: All messages from all sources delivered in single order

51. Lecture 3 Page 51 CS 239, Spring 2001 Achieving FIFO Ordering Sender keeps local sequence counter Each multicast stamped with incremented counter Each receiver keeps counter for each sender Only deliver messages with expected timestamp –Queue or reject others

52. Lecture 3 Page 52 CS 239, Spring 2001 Achieving Causal Order Use what amounts to vector clocks –One element for each sender Sender increments local element of vector each time it multicasts –Attaching new vector to message Each receiver stores vector of last message it accepted

53. Lecture 3 Page 53 CS 239, Spring 2001 Receiving Causally Ordered Messages Properly Compare incoming message’s vector to stored vector If one element of incoming vector is 1 greater than stored vector, accept it Delay it if more than one element greater, or any element greater by more than 1 Reject if vector is less than stored one

54. Lecture 3 Page 54 CS 239, Spring 2001 Example of Receiving Causally Ordered Messages 000000000100 100 100 A B C Should C accept this message? Yes! 1 = 0+1, 0 = 0, and 0 = 0 But B’s message is lost 101 101 Should B accept this message? No! 1 = 0+1, 0 = 0, and 1 = 0+1 B has missed a causal message 100 100 100101

55. Lecture 3 Page 55 CS 239, Spring 2001 Totally Ordering Multicasts Even harder than causal order Essentially, requires a commitment protocol –Everyone agrees they’ve received the message –The sender knows everyone’s agreed –Then the message is delivered Related to 2-phase commit