1. Operating Systems, 2011, Danny Hendler & Amnon Meisels 1 Distributed Synchronization: outline  Introduction  Causality and time o Lamport timestamps o Vector timestamps o Causal communication  Snapshots This presentation is based on the book: “Distributed operating-systems & algorithms” by Randy Chow and Theodore Johnson

2. 2 Operating Systems, 2011, Danny Hendler & Amnon Meisels  A distributed system is a collection of independent computational nodes, communicating over a network, that is abstracted as a single coherent system o Grid computing o Cloud computing (“software as a service”) o Peer-to-peer computing o Sensor networks o …  A distributed operating system allows sharing of resources and coordination of distributed computation in a transparent manner Distributed systems (a), (b) – a distributed system (c) – a multiprocessor

3. 3 Operating Systems, 2011, Danny Hendler & Amnon Meisels  Underlies distributed operating systems and algorithms  Processes communicate via message passing (no shared memory)  Inter-node coordination in distributed systems challenged by o Lack of global state o Lack of global clock o Communication links may fail o Messages may be delayed or lost o Nodes may fail  Distributed synchronization supports correct coordination in distributed systems o May no longer use shared-memory based locks and semaphores Distributed synchronization

4. 4 Operating Systems, 2011, Danny Hendler & Amnon Meisels Distributed Synchronization: outline  Introduction  Causality and time o Lamport timestamps o Vector timestamps o Causal communication  Snapshots

5. 5 Operating Systems, 2011, Danny Hendler & Amnon Meisels  Events o Sending a message o Receiving a message o Timeout, internal interrupt  Processors send control messages to each other o send(destination, action; parameters)  Processes may declare that they are waiting for events: o Wait for A 1, A 2, …., A n A 1 (source; parameters) code to handle A 1.. A n (source; parameters) code to handle A n Distributed computation model

6. 6 Operating Systems, 2011, Danny Hendler & Amnon Meisels  A distributed system has no global state nor global clock  no global order on all events may be determined  Each processor knows total orders on events occurring in it  There is a causal relation between the sending of a message and its receipt Causality and events ordering Lamport's happened-before relation H 1.e 1 < p e 2  e 1 < H e 2 (events within same processor are ordered) 2.e 1 < m e 2  e 1 < H e 2 (each message m is sent before it is received) 3.e 1 < H e 2 AND e 2 < H e 3  e 1 < H e 3 (transitivity) Leslie Lamport (1978): “Time, clocks, and the ordering of events in a distributed system”

7. 7 Operating Systems, 2011, Danny Hendler & Amnon Meisels Causality and events ordering (cont'd) Time P1P1 P2P2 P3P3 e1e1 e2e2 e3e3 e4e4 e5e5 e6e6 e7e7 e8e8 e 1 < H e 7 ? Yes. e 1 < H e 3 ? Yes. e 1 < H e 8 ? Yes. e 5 < H e 7 ? No.

8. 8 Operating Systems, 2011, Danny Hendler & Amnon Meisels 1 Initially my_TS=0 2 Upon event e, 3 if e is the receipt of message m 4 my_TS=max(m.TS, my_TS) 5 my_TS++ 6 e.TS=my_TS 7 If e is the sending of message m 8 m.TS=my_TS Lamport's timestamp algorithm To create a total order, ties are broken by process ID

9. 9 Operating Systems, 2011, Danny Hendler & Amnon Meisels Lamport's timestamps (cont'd) Time P1P1 P2P2 P3P3 e1e1 e2e2 e3e3 e4e4 e5e5 e6e6 e7e7 e8e8 1.1 2.1 3.1 1.3 2.2 3.2 4.3 1.2

10. 10 Operating Systems, 2011, Danny Hendler & Amnon Meisels Distributed Synchronization: outline  Introduction  Causality and time o Lamport timestamps o Vector timestamps o Causal communication  Snapshots

11. 11 Operating Systems, 2011, Danny Hendler & Amnon Meisels  Lamport's timestamps define a total order o e 1 < H e 2  e 1.TS < e 2.TS o However, e 1.TS < e 2.TS  e 1 < H e 2 does not hold, in general. Vector timestamps - motivation Definition: Message m 1 casually precedes message m 2 (written as m 1 < c m 2 ) if s(m 1 ) < H s(m 2 ) (sending m 1 happens before sending m 2 ) Definition: causality violation occurs if m 1 < c m 2 but r(m 2 ) < p r(m 1 ). In other words, m 1 is sent to processor p before m 2 but is received after it.  Lamport's timestamps do not allow to detect (nor prevent) causality violations.

12. 12 Operating Systems, 2011, Danny Hendler & Amnon Meisels Causality violations – an example P1P1 P2P2 P3P3 Migrate O to P 2 Where is O? M1 On p 2 Where is O? I don’t know M2 M3 ERROR! Causality violation between… M 1 and M 3.

13. 13 Operating Systems, 2011, Danny Hendler & Amnon Meisels Vector timestamps 1 Initially my_VT=[0,…,0] 2 Upon event e, 3 if e is the receipt of message m 4 for i=1 to M 5 my_VT[i]=max(m.VT[i], my_VT[i]) 6My_VT[self]++ 7e.VT=my_VT 8if e is the sending of message m 9 m.VT=my_VT For vector timestamps it does hold that: e 1 < VT e 2 e 1 < H e 2 e 1.VT ≤ V e 2.VT if and only if e 1.VT[i] ≤ e 2.VT[i], for every i=1,…,M e 1.VT < V e 2.VT if and only if e 1.VT ≤ V e 2.VT and e 1.VT ≠ e 2.VT

14. 14 Operating Systems, 2011, Danny Hendler & Amnon Meisels Vector timestamps (cont'd) P1P1 P2P2 P3P3 P4P4 1 1 1 1 2 2 2 3 2 3 3 4 5 4 3 5 4 6 e e.VT=(3,6,4,2)

15. 15 Operating Systems, 2011, Danny Hendler & Amnon Meisels VTs can be used to detect causality violations P1P1 P2P2 P3P3 Migrate O to P 2 Where is O? M1 On p 2 Where is O? I don’t know M2 M3 ERROR! Causality violation between… M 1 and M 3. (1,0,0) (0,0,1) (2,0,1) (3,0,1) (3,0,2) (3,0,3) (3,1,3) (3,2,3) (3,3,3) (3,2,4)

16. 16 Operating Systems, 2011, Danny Hendler & Amnon Meisels Distributed Synchronization: outline  Introduction  Causality and time o Lamport timestamps o Vector timestamps o Causal communication  Snapshots

17. 17 Operating Systems, 2011, Danny Hendler & Amnon Meisels A processor cannot control the order in which it receives messages… But it may control the order in which they are delivered to applications Preventing causality violations Application FIFO ordering Message passing Deliver in FIFO order Assign timestamps Message arrival order x1x1 Deliver x 1 x2x2 Deliver x 2 x4x4 Buffer x 4 x5x5 Buffer x 5 x3x3 Deliever x 3 x 4 x 5 Protocol for FIFO message delivery (as in, e.g., TCP)

18. 18 Operating Systems, 2011, Danny Hendler & Amnon Meisels Preventing causality violations (cont'd)  Senders attach a timestamp to each message  Destination delays the delivery of out-of-order messages  Hold back a message m until we are assured that no message m’ < H m will be delivered o For every other process p, maintain the earliest timestamp of a message m that may be delivered from p o Do not deliver a message if an earlier message may still be delivered from another process

19. 19 Operating Systems, 2011, Danny Hendler & Amnon Meisels Algorithm for preventing causality violations 1earliest[1..M] initially [,, …, ] 2blocked[1…M] initially [ {}, …, {} ] 3Upon the receipt of message m from processor p 4Delivery_list = {} 5If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7Add m to the tail of blocked[p] 8While (  k such that blocked[k] is non-empty AND  i  {1,…,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k]  m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14End While 15Deliver the messages in delivery_list, in causal order For each processor p, the earliest timestamp with which a message from p may still be delivered

20. 20 Operating Systems, 2011, Danny Hendler & Amnon Meisels Algorithm for preventing causality violations 1earliest[1..M] initially [,, …, ] 2blocked[1…M] initially [ {}, …, {} ] 3Upon the receipt of message m from processor p 4Delivery_list = {} 5If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7Add m to the tail of blocked[p] 8While (  k such that blocked[k] is non-empty AND  i  {1,…,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k]  m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14End While 15Deliver the messages in delivery_list, in causal order For each process p, the messages from p that were received but were not delivered yet

21. 21 Operating Systems, 2011, Danny Hendler & Amnon Meisels Algorithm for preventing causality violations 1earliest[1..M] initially [,, …, ] 2blocked[1…M] initially [ {}, …, {} ] 3Upon the receipt of message m from processor p 4Delivery_list = {} 5If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7Add m to the tail of blocked[p] 8While (  k such that blocked[k] is non-empty AND  i  {1,…,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k]  m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14End While 15Deliver the messages in delivery_list, in causal order List of messages to be delivered as a result of m’s receipt

22. 22 Operating Systems, 2011, Danny Hendler & Amnon Meisels Algorithm for preventing causality violations 1earliest[1..M] initially [,, …, ] 2blocked[1…M] initially [ {}, …, {} ] 3Upon the receipt of message m from processor p 4Delivery_list = {} 5If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7Add m to the tail of blocked[p] 8While (  k such that blocked[k] is non-empty AND  i  {1,…,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k]  m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14End While 15Deliver the messages in delivery_list, in causal order If no blocked messages from p, update earliest[p]

23. 23 Operating Systems, 2011, Danny Hendler & Amnon Meisels Algorithm for preventing causality violations 1earliest[1..M] initially [,, …, ] 2blocked[1…M] initially [ {}, …, {} ] 3Upon the receipt of message m from processor p 4Delivery_list = {} 5If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7Add m to the tail of blocked[p] 8While (  k such that blocked[k] is non-empty AND  i  {1,…,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k]  m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14End While 15Deliver the messages in delivery_list, in causal order m is now the most recent message from p not yet delivered

24. 24 Operating Systems, 2011, Danny Hendler & Amnon Meisels Algorithm for preventing causality violations 1earliest[1..M] initially [,, …, ] 2blocked[1…M] initially [ {}, …, {} ] 3Upon the receipt of message m from processor p 4Delivery_list = {} 5If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7Add m to the tail of blocked[p] 8While (  k such that blocked[k] is non-empty AND  i  {1,…,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k]  m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14End While 15Deliver the messages in delivery_list, in causal order If there is a process k with delayed messages for which it is now safe to deliver its earliest message…

25. 25 Operating Systems, 2011, Danny Hendler & Amnon Meisels Algorithm for preventing causality violations 1earliest[1..M] initially [,, …, ] 2blocked[1…M] initially [ {}, …, {} ] 3Upon the receipt of message m from processor p 4Delivery_list = {} 5If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7Add m to the tail of blocked[p] 8While (  k such that blocked[k] is non-empty AND  i  {1,…,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k]  m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14End While 15Deliver the messages in delivery_list, in causal order Remove k'th earliest message from blocked queue, make sure it is delivered

26. 26 Operating Systems, 2011, Danny Hendler & Amnon Meisels Algorithm for preventing causality violations 1earliest[1..M] initially [,, …, ] 2blocked[1…M] initially [ {}, …, {} ] 3Upon the receipt of message m from processor p 4Delivery_list = {} 5If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7Add m to the tail of blocked[p] 8While (  k such that blocked[k] is non-empty AND  i  {1,…,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k]  m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14End While 15Deliver the messages in delivery_list, in causal order If there are additional blocked messages of k, update earliest[k] to be the timestamp of the earliest such message

27. 27 Operating Systems, 2011, Danny Hendler & Amnon Meisels Algorithm for preventing causality violations 1earliest[1..M] initially [,, …, ] 2blocked[1…M] initially [ {}, …, {} ] 3Upon the receipt of message m from processor p 4Delivery_list = {} 5If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7Add m to the tail of blocked[p] 8While (  k such that blocked[k] is non-empty AND  i  {1,…,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k]  m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14End While 15Deliver the messages in delivery_list, in causal order Otherwise the earliest message that may be delivered from k would have previous timestamp with the k'th component incremented

28. 28 Operating Systems, 2011, Danny Hendler & Amnon Meisels Algorithm for preventing causality violations 1earliest[1..M] initially [,, …, ] 2blocked[1…M] initially [ {}, …, {} ] 3Upon the receipt of message m from processor p 4Delivery_list = {} 5If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7Add m to the tail of blocked[p] 8While (  k such that blocked[k] is non-empty AND  i  {1,…,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k]  m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14End While 15Deliver the messages in delivery_list, in causal order Finally, deliver set of messages that will not cause causality violation (if there are any).

29. 29 Operating Systems, 2011, Danny Hendler & Amnon Meisels Execution of the algorithm as multicast P1P1 P2P2 P3P3 (0,1,0) Since the algorithm is “interested” only in causal order of sending events, vector timestamp is incremented only upon send events. (1,1,0) p 3 state earliest[1] earliest[2] earliest[3] (1,0,0)(0,1,0)(0,0,1) m1m1 m2m2 (1,1,0) (0,1,0)(0,0,1) Upon receipt of m 2 Upon receipt of m 1 (1,1,0)(0,1,0)(0,0,1) deliver m 1 (1,1,0)(0,2,0)(0,0,1) deliver m 2 (2,1,0)(0,2,0)(0,0,1)

30. 30 Operating Systems, 2011, Danny Hendler & Amnon Meisels  Introduction  Causality and time o Lamport timestamps o Vector timestamps o Causal communication  Snapshots Distributed Synchronization: outline

31. 31 Operating Systems, 2011, Danny Hendler & Amnon Meisels  Assume we would like to implement a distributed deadlock-detector  We record process states to check if there is a waits-for-cycle Snapshots motivation – phantom deadlock P1P1 r1r1 r2r2 P2P2 request OK request OK release request OK 1 2 3 4 p2p2 r1r1 r2r2 p1p1 Actual waits-for graph p2p2 r1r1 r2r2 p1p1 Observed waits-for graph request

32.  Global system state: o S=(s 1, s 2, …, s M ) – local processor states o The contents L i,j =(m 1, m 2, …, m k ) of each communication channel C i,j (channels assumed to be FIFO) These are messages sent but not yet received  Global state must be consistent o If we observe in state s i that p i received message m from p k, then in observation s k,k must have sent m. o Each L i,j must contain exactly the set of messages sent by p i but not yet received by p j, as reflected by s i, s j. What is a snapshot? Snapshot state much be consistent, one that might have existed during the computation. Observations must be mutually concurrent that is, no observation casually precedes another observation

33.  Upon joining the algorithm a process records its local state  The process that initiates the snapshot sends snapshot tokens to its neighbors (before sending any other messages) o Neighbors send them to their neighbors – broadcast  Upon receiving a snapshot token: o a process must record its state prior to receiving additional messages o Must then send tokens to all its other neighbors  How shall we record sets L p,q ? o q receives token from p and that is the first time q receives token L p,q ={ } o q receives token from p but q received token before L p,q ={all messages received by q from p since q received token} Snapshot algorithm – informal description

34. 34 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – data structures  The algorithm supports multiple ongoing snapshots – one per process  Different snapshots from same process distinguished by version number Per process variables integer my_version initially 0 integer current_snap[1..M] initially [0,…,0] integer tokens_received[1..M] processor_state S[1…M] channel_state [1..M][1..M] The version number of my current snapshot

35. 35 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – data structures  The algorithm supports multiple ongoing snapshots – one per process  Different snapshots from same process distinguished by version number Per process variables integer my_version initially 0 integer current_snap[1..M] initially [0,…,0] integer tokens_received[1..M] processor_state S[1…M] channel_state [1..M][1..M] current_snap[r] contains version number of snapshot initiated by processor r

36. 36 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – data structures  The algorithm supports multiple ongoing snapshots – one per process  Different snapshots from same process distinguished by version number Per process variables integer my_version initially 0 integer current_snap[1..M] initially [0,…,0] integer tokens_received[1..M] processor_state S[1…M] channel_state [1..M][1..M] tokens_received[r] contains the number of tokens received for the snapshot initiated by processor r

37. 37 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – data structures  The algorithm supports multiple ongoing snapshots – one per process  Different snapshots from same process distinguished by version number Per process variables integer my_version initially 0 integer current_snap[1..M] initially [0,…,0] integer tokens_received[1..M] processor_state S[1…M] channel_state [1..M][1..M] processor_state[r] contains the state recorded for the snapshot of processor r

38. 38 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – data structures  The algorithm supports multiple ongoing snapshots – one per process  Different snapshots from same process distinguished by version number Per process variables integer my_version initially 0 integer current_snap[1..M] initially [0,…,0] integer tokens_received[1..M] processor_state S[1…M] channel_state [1..M][1..M] channel_state[r][q] records the state of channel from q to current processor for the snapshot initiated by processor r

39. 39 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self]  my_state 3 for each outgoing channel q, send(q, TOEKEN, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5.If current_snap[r] < version 6. S[r]  my state 7. current_snap[r]=version 8. L[r][q]  empty, send token(r, version) on each outgoing channel 9. tokens_received[r]  1 10.else 11. tokens_received[r]++ 12. L[r][q]  all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished

40. 40 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self]  my_state 3 for each outgoing channel q, send(q, TOEKEN, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5.If current_snap[r] < version 6. S[r]  my state 7. current_snap[r]=version 8. L[r][q]  empty, send token(r, version) on each outgoing channel 9. tokens_received[r]  1 10.else 11. tokens_received[r]++ 12. L[r][q]  all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished

41. 41 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self]  my_state 3 for each outgoing channel q, send(q, TOEKEN, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5.If current_snap[r] < version 6. S[r]  my state 7. current_snap[r]=version 8. L[r][q]  empty, send token(r, version) on each outgoing channel 9. tokens_received[r]  1 10.else 11. tokens_received[r]++ 12. L[r][q]  all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished Increment version number of this snapshot, record my local state

42. 42 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self]  my_state 3 for each outgoing channel q, send(q, TOEKEN; self, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5.If current_snap[r] < version 6. S[r]  my state 7. current_snap[r]=version 8. L[r][q]  empty, send token(r, version) on each outgoing channel 9. tokens_received[r]  1 10.else 11. tokens_received[r]++ 12. L[r][q]  all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished Send snapshot-token on all outgoing channels, initialize number of received tokens for my snapshot to 0

43. 43 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self]  my_state 3 for each outgoing channel q, send(q, TOEKEN; self, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5.If current_snap[r] < version 6. S[r]  my state 7. current_snap[r]=version 8. L[r][q]  empty, send token(r, version) on each outgoing channel 9. tokens_received[r]  1 10.else 11. tokens_received[r]++ 12. L[r][q]  all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished Upon receipt from q of TOKEN for snapshot (r,version)

44. 44 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self]  my_state 3 for each outgoing channel q, send(q, TOEKEN; self, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5.If current_snap[r] < version 6. S[r]  my state 7. current_snap[r]=version 8. L[r][q]  empty, send token(r, version) on each outgoing channel 9. tokens_received[r]  1 10.else 11. tokens_received[r]++ 12. L[r][q]  all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished If this is first token for snapshot (r,version)

45. 45 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self]  my_state 3 for each outgoing channel q, send(q, TOEKEN; self, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5.If current_snap[r] < version 6. S[r]  my state 7. current_snap[r]=version 8. L[r][q]  empty, send token(r, version) on each outgoing channel 9. tokens_received[r]  1 10.else 11. tokens_received[r]++ 12. L[r][q]  all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished Record local state for r'th snapshot Update version number of r'th snapshot

46. 46 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self]  my_state 3 for each outgoing channel q, send(q, TOEKEN; self, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5.If current_snap[r] < version 6. S[r]  my state 7. current_snap[r]=version 8. L[r][q]  empty, send token(r, version) on each outgoing channel 9. tokens_received[r]  1 10.else 11. tokens_received[r]++ 12. L[r][q]  all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished Set of messages on channel from q is empty Send token on all outgoing channels Initialize number of received tokens to 1

47. 47 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self]  my_state 3 for each outgoing channel q, send(q, TOEKEN; self, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5.If current_snap[r] < version 6. S[r]  my state 7. current_snap[r]=version 8. L[r][q]  empty, send token(r, version) on each outgoing channel 9. tokens_received[r]  1 10.else 11. tokens_received[r]++ 12. L[r][q]  all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished Not the first token for (r,version)

48. 48 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self]  my_state 3 for each outgoing channel q, send(q, TOEKEN; self, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5.If current_snap[r] < version 6. S[r]  my state 7. current_snap[r]=version 8. L[r][q]  empty, send token(r, version) on each outgoing channel 9. tokens_received[r]  1 10.else 11. tokens_received[r]++ 12. L[r][q]  all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished Yet another token for snapshot (r,version)

49. 49 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self]  my_state 3 for each outgoing channel q, send(q, TOEKEN; self, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5.If current_snap[r] < version 6. S[r]  my state 7. current_snap[r]=version 8. L[r][q]  empty, send token(r, version) on each outgoing channel 9. tokens_received[r]  1 10.else 11. tokens_received[r]++ 12. L[r][q]  all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished These messages are the state of the channel from q for snapshot (r,version)

50. 50 Operating Systems, 2011, Danny Hendler & Amnon Meisels Snapshot algorithm – pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self]  my_state 3 for each outgoing channel q, send(q, TOEKEN; self, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5.If current_snap[r] < version 6. S[r]  my state 7. current_snap[r]=version 8. L[r][q]  empty, send token(r, version) on each outgoing channel 9. tokens_received[r]  1 10.else 11. tokens_received[r]++ 12. L[r][q]  all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished If all tokens of snapshot (r,version) arrived, snapshot computation is over