1. Theoretical Foundations
Clocks
Logical clocks (Lamport)
Partially ordered clocks (Fidge)
Causally ordered message delivery
Broadcast-based (Birman et al.)
Unicast-based (Raynal, Schiper & Toueg)
System states
Global snapshots (Chandy & Lamport)
Termination detection (Huang)
Theoretical Foundations

2. States Process State: Channel State: Global State:
the state of a process’s variables after an event (e.g., SP = p1)
Channel State:
the set of messages currently in transit in a channel (e.g., CP,Q = {m1, m2})
Global State:
the collective state of all processes and channels (e.g., S = {SP, SQ, SR, CP,Q, CP,R, CQ,P, CQ,R, CR,P, CR,Q})
Theoretical Foundations > Global States

3. Inconsistent Global State
The recorded state of process q shows the delivery of message m sent by process p, but the recorded state of process p does not show that the sending of m has occurred yet
If record(Sp) → sendp(m) and deliverq(m) → record(Sq), then S is inconsistent
For each deliver event, there is not a corresponding send event
Theoretical Foundations > Global States

4. Inconsistent Example P Q R p2 q3 m4 m2 m3 r3
Theoretical Foundations > Global States

5. Consistent Global State
If the recorded state of process q shows the delivery of message m sent by process p, then the recorded state of process p shows the sending of m
If deliverq(m) → record(Sq), then sendp(m) → record(Sp)
For each deliver event, there is a corresponding send event
Theoretical Foundations > Global States

6. Consistent Example P Q R p2 q3 m2 m3 r2
Theoretical Foundations > Global States

7. Strongly Consistent Global State
A consistent global state with no messages in transit
If S is a consistent global state and there are no messages in the channel states, then S is strongly consistent
For each send event, there is a corresponding deliver event
Theoretical Foundations > Global States

8. Strongly Consistent Exampleq1 m2 q2 q3 q4 q5 q6 q7 Q m3 m2 r1 r2 r3 r4 R
Theoretical Foundations > Global States

9. GSD Protocol Purpose: Assumptions: Terminology:
To record a consistent global state for a system
Assumptions:
Channels causally order messages (FIFO)
Channels are not required to be bidirectional
Any process can initiate the algorithm by recording its local state
Terminology:
Messages exchanged by the computation are called “application messages” or just “messages”
Messages exchanged by the GSD algorithm are called “control messages” or “markers”
Theoretical Foundations > Global States > GSD Protocol

10. GSD Protocol Rules Rule 1: Marker-Sending
if(process p records its state) { for(each outgoing channel c) {
p sends one marker along c; }
Theoretical Foundations > Global States > GSD Protocol

11. GSD Protocol Rules Rule 2: Marker-Receiving
if(process q receives a marker from channel c) {
if(q has not recorded its state) {
q records its state; (see Rule 1)
q records the state of c as empty; }
else {
q records the state of c as the sequence of messages received from c since q recorded its state;
Theoretical Foundations > Global States > GSD Protocol

12. GSD Protocol Rules Rule 3: Termination Detection complete = true;
for(each incoming channel c) {
if(process q has not received a marker from c) {
complete = false; }
if(complete) {terminate GSD protocol at q; }
Theoretical Foundations > Global States > GSD Protocol

13. Stable Properties
A property is stable if once it becomes true, it remains true for all future global states
If y(S) = true and y(S’) = true for all future global states S’, then y is stable.
Examples:
Termination
Deadlock
Theoretical Foundations > Global States > Stable Properties

14. GSD Protocol Exercise P Q R Topology
Theoretical Foundations > Global States > GSD Protocol

15. GSD Protocol Exercise
Assume R initiates after r1 and all markers are received between [p2, p3], [q2, q3], and [r1, r2] p1 p2 p3 P m1 m3 q1 m2 q2 q3 Q m2 r1 r2 r3 R
Theoretical Foundations > Global States > GSD Protocol

16. GSD Protocol Exercise
R records the state of its variables after r1 
(i.e., SR = r1) p1 p2 p3 P m1 m3 q1 m2 q2 q3 Q m2 r1 r2 r3 R
Theoretical Foundations > Global States > GSD Protocol

17. GSD Protocol Exercise R sends a marker along each outgoing channel P Q
Theoretical Foundations > Global States > GSD Protocol

18. GSD Protocol Exercise
P records the state of its variables after p2 
(i.e., SP = p2) p1 p2 p3 P m1 m3 q1 m2 q2 q3 Q m2 r1 r2 r3 R
Theoretical Foundations > Global States > GSD Protocol

19. GSD Protocol Exercise P sends a marker along each outgoing channel P Q
Theoretical Foundations > Global States > GSD Protocol

20. GSD Protocol Exercise
P records the state of CR,P as empty
(i.e., CR,P = { } ) p1 p2 p3 P m1 m3 q1 m2 q2 q3 Q m2 r1 r2 r3 R
Theoretical Foundations > Global States > GSD Protocol

21. GSD Protocol Exercise
Q records the state of its variables after q2 
(i.e., SQ = q2) p1 p2 p3 P m1 m3 q1 m2 q2 q3 Q m2 r1 r2 r3 R
Theoretical Foundations > Global States > GSD Protocol

22. GSD Protocol Exercise Q sends a marker along each outgoing channel P Q
Theoretical Foundations > Global States > GSD Protocol

23. GSD Protocol Exercise
Q records the state of CP,Q as empty
(i.e., CP,Q = { } ) p1 p2 p3 P m1 m3 q1 m2 q2 q3 Q m2 r1 r2 r3 R
Theoretical Foundations > Global States > GSD Protocol

24. GSD Protocol Exercise Q has received all markers and terminates GSD P
Theoretical Foundations > Global States > GSD Protocol

25. GSD Protocol Exercise
R must receive m2 before Q’s marker due to the FIFO channel p1 p2 p3 P m1 m3 q1 m2 q2 q3 Q m2 r1 r2 r3 R
Theoretical Foundations > Global States > GSD Protocol

26. GSD Protocol Exercise
R records the state of CQ,R as the set of messages received from Q since SR (i.e., CQ,R = {m2} ) p1 p2 p3 P m1 m3 q1 m2 q2 q3 Q m2 r1 r2 r3 R
Theoretical Foundations > Global States > GSD Protocol

27. GSD Protocol Exercise R has received all markers and terminates GSD Pq1 m2 q2 q3 Q m2 r1 r2 r3 R
Theoretical Foundations > Global States > GSD Protocol

28. GSD Protocol Exercise
P records the state of CQ,P as empty
(i.e., CQ,P = { } ) p1 p2 p3 P m1 m3 q1 m2 q2 q3 Q m2 r1 r2 r3 R
Theoretical Foundations > Global States > GSD Protocol

29. GSD Protocol Exercise P has received all markers and terminates GSD Pq1 m2 q2 q3 Q m2 r1 r2 r3 R
Theoretical Foundations > Global States > GSD Protocol

30. GSD Protocol Exercise SC is the global snapshot recorded by GSD P Q Rm1 m3 q1 m2 q2 q3 Q m2 r1 r2 r3 R
Theoretical Foundations > Global States > GSD Protocol

31. GSD Protocol Exercise SC is the global snapshot recorded by GSD P Q Rm2 r1 R
Theoretical Foundations > Global States > GSD Protocol

32. GSD Protocol Exercise P Q R p2 q2 r1 m2
Theoretical Foundations > Global States > GSD Protocol

33. Theoretical Foundations
Clocks
Logical clocks (Lamport)
Partially ordered clocks (Fidge)
Causally ordered message delivery
Broadcast-based (Birman et al.)
Unicast-based (Raynal, Schiper & Toueg)
System states
Global snapshots (Chandy & Lamport)
Termination detection (Huang)
Theoretical Foundations

34. Theoretical Foundations: Termination Detection
CS/CE/TE Advanced Operating Systems

35. Activity Processes are either active or passive Active: Passive:
processing a computation or application
Only active processes send messages
May become passive at any time
Passive:
not processing a computation or application
Reactivated only upon receiving a message
Theoretical Foundations > Activity

36. Quiz Question When is an idle process reactivated?
When it computes an atomic action.
When it receives a message.
When it sends a message.
All of the above.
Theoretical Foundations > Activity

37. Activity At S1, only C is active C D E S1 m1 m2 m3
Theoretical Foundations > Activity

38. Activity At S2, D is reactivated and C remains active C D E S2 m1 m2
Theoretical Foundations > Activity

39. Activity At S3, E is reactivated, and C and D remain active C D E S3
Theoretical Foundations > Activity

40. Activity At S4, E becomes idle, and C and D remain active C D E S4 m1
Theoretical Foundations > Activity

41. Activity At S5, D becomes idle and C remains active C D E S5 m1 m2 m3
Theoretical Foundations > Activity

42. Activity At S6, E is reactivated and C remains active C D E S6 m1 m2
Theoretical Foundations > Activity

43. Activity At S7, E becomes idle and C remains active C D E S7 m1 m2 m3
Theoretical Foundations > Activity

44. Termination
A computation or application is said to be terminated iff all processes are idle and there are no messages in transit
Theoretical Foundations > Termination

45. Quiz Question
If all processes are idle and there are no messages in transit, what is a computation said to be?
Active
Idle
Terminated
Weighted
Theoretical Foundations > Termination

46. Quiz Question What type of global state is termination?
Inconsistent Strongly consistent
Strongly inconsistent
Consistent
Strongly consistent
Theoretical Foundations > Termination

47. Termination Where does termination occur? S5 C D E S1 S2 S3 S4 S5 m1
Theoretical Foundations > Termination

48. Huang’s Protocol Basic concept: Control process determines termination
Every active process has a weight
Idle processes have no weight
Processes send portion of weight with messages
Processes send entire weight to control process when becoming idle
Termination occurs when control has all the weight
Theoretical Foundations > Termination > Huang’s Protocol

49. Huang’s Protocol Assumptions:
Control process initiates the computation
Communication channels exist between every process pair (i.e., a complete network topology)
Message delay is arbitrary but finite (i.e., a lossless network)
No process failures
Theoretical Foundations > Termination > Huang’s Protocol

50. Huang’s Protocol
Rule 1: Sending an application message When a process p sends a message m, it divides it weight Wp into W1 and W2. It keeps W1 and sends W2 along with the message. send(m, W2); Wp = W1;
Theoretical Foundations > Termination > Huang’s Protocol

51. Huang’s Protocol Example
Control process C starts with a weight of 1 and other processes start with no weight 1
control C D E
Theoretical Foundations > Termination > Huang’s Protocol

52. Huang’s Protocol Example
C starts by sending a message to process D 1
1/2
control C m1
1/2 D E
Theoretical Foundations > Termination > Huang’s Protocol

53. Huang’s Protocol
Rule 2: Receiving an application message When process q receives a message, it adds the message’s weight Wm to its own weight Wq. If q is idle, q is reactivated. receive(m, Wm); Wq = Wq + Wm;
Theoretical Foundations > Termination > Huang’s Protocol

54. Huang’s Protocol Example
D receives message m1 and adds Wm to weight 1
1/2
control C m1
1/2
1/2 D E
Theoretical Foundations > Termination > Huang’s Protocol

55. Huang’s Protocol Example
D sends a message to process E 1
1/2
control C m1
1/2
1/2
1/4 D m2
1/4 E
Theoretical Foundations > Termination > Huang’s Protocol

56. Huang’s Protocol Example
E receives message m2 and adds Wm to weight 1
1/2
control C m1
1/2
1/2
1/4 D m2
1/4
1/4 E
Theoretical Foundations > Termination > Huang’s Protocol

57. Huang’s Protocol
Rule 3: Becoming idle When process q becomes idle, it sends its weight Wq along with a control message to the control process c. send(m, Wq); idle();
Theoretical Foundations > Termination > Huang’s Protocol

58. Huang’s Protocol Example
E becomes idle and sends weight to C 1
1/2
control C m1
1/2
1/2
1/4 D m2
1/4
1/4
1/4 E
Theoretical Foundations > Termination > Huang’s Protocol

59. Huang’s Protocol
Rule 4: Monitoring When the control process c receives a control message, it adds the message’s weight Wm to its own weight Wc. If Wc equals 1, the computation has terminated. receive(m, Wm); Wc = Wc + Wm; if(Wc == 1) termination = true;
Theoretical Foundations > Termination > Huang’s Protocol

60. Huang’s Protocol Example
C receives E’s idle message and adds Wm to WC 1
1/2
3/4
control C m1
1/2
1/2
1/4 D m2
1/4
1/4
1/4 E
Theoretical Foundations > Termination > Huang’s Protocol

61. Huang’s Protocol Example
WC is not 1 so computation has not terminated 1
1/2
3/4
control C m1
1/2
1/2
1/4 D m2
1/4
1/4
1/4 E
Theoretical Foundations > Termination > Huang’s Protocol

62. Huang’s Protocol Example
D sends a message to process E 1
1/2
3/4
control C m1
1/2
1/2
1/4
1/8 D m2 m3
1/4
1/4
1/8
1/4 E
Theoretical Foundations > Termination > Huang’s Protocol

63. Huang’s Protocol Example
D becomes idle and sends weight to C 1
1/2
3/4
control C m1
1/2
1/8
1/2
1/4
1/8 D m2 m3
1/4
1/4
1/8
1/4 E
Theoretical Foundations > Termination > Huang’s Protocol

64. Huang’s Protocol Example
E receives message m3 and adds Wm to weight 1
1/2
3/4
control C m1
1/2
1/8
1/2
1/4
1/8 D m2 m3
1/4
1/4
1/8
1/4
1/8 E
Theoretical Foundations > Termination > Huang’s Protocol

65. Huang’s Protocol Example
C receives D’s idle message and adds Wm to WC 1
1/2
3/4
7/8
control C m1
1/2
1/8
1/2
1/4
1/8 D m2 m3
1/4
1/4
1/8
1/4
1/8 E
Theoretical Foundations > Termination > Huang’s Protocol

66. Huang’s Protocol Example
E becomes idle and sends weight to C 1
1/2
3/4
7/8
control C m1
1/2
1/8
1/2
1/4
1/8 D m2 m3
1/4
1/8
1/4
1/8
1/4
1/8 E
Theoretical Foundations > Termination > Huang’s Protocol

67. Huang’s Protocol Example
C receives E’s idle message and adds Wm to WC 1
1/2
3/4
7/8 1
control C m1
1/2
1/8
1/2
1/4
1/8 D m2 m3
1/4
1/8
1/4
1/8
1/4
1/8 E
Theoretical Foundations > Termination > Huang’s Protocol

68. Huang’s Protocol Example
WC is 1 so the computation has terminated 1
1/2
3/4
7/8 1
control C m1
1/2
1/8
1/2
1/4
1/8 D m2 m3
1/4
1/8
1/4
1/8
1/4
1/8 E
Theoretical Foundations > Termination > Huang’s Protocol

69. Quiz Questions If WC = 0.5 at c2, and WC = 0.75 at c4,
what does WD equal at d2?
1.0
1.0
control
0.5
0.75
0.5
0.25
0.5
0.25
0.25
0.25
Theoretical Foundations > Termination > Huang’s Protocol

70. Quiz Questions If WC = 0.5 at c2, and WC = 0.75 at c4,
what does WE equal at e2?
1.0
1.0
control
0.5
0.75
0.5
0.25
0.5
0.25
0.25
0.25
Theoretical Foundations > Termination > Huang’s Protocol

71. Binary Representation
Can represent W as a binary number in an unbounded array of bits with the binary point before the first bit
(i.e., “100…” has a binary value of 0.100… or a decimal value of 0.5)
The control process c starts by sending a message with Wm = “1” and sets Wc = “1”
Termination occurs when Wc rolls over (i.e., the bit array Wc requires a new leading bit)
Theoretical Foundations > Termination > Huang’s Protocol

72. Binary Representation Example
Control process C starts by sending a message to process D 1
control C m1 1 D E
Theoretical Foundations > Termination > Huang’s Protocol

73. Binary Representation Example
D receives message m1 and adds Wm to weight 1
control C m1 1 1 D E
Theoretical Foundations > Termination > Huang’s Protocol

74. Binary Representation Example
D sends a message to process E 1
control C m1 1 1 01 D m2 01 E
Theoretical Foundations > Termination > Huang’s Protocol

75. Binary Representation Example
E receives message m2 and adds Wm to weight 1
control C m1 1 1 01 D m2 01 01 E
Theoretical Foundations > Termination > Huang’s Protocol

76. Binary Representation Example
E becomes idle and sends weight to C 1
control C m1 1 1 01 D m2 01 01 01 E
Theoretical Foundations > Termination > Huang’s Protocol

77. Binary Representation Example
C receives E’s idle message and adds Wm to WC 1 11
control C m1 1 1 01 D m2 01 01 01 E
Theoretical Foundations > Termination > Huang’s Protocol

78. Binary Representation Example
WC is not 1 so computation has not terminated 1 11
control C m1 1 1 01 D m2 01 01 01 E
Theoretical Foundations > Termination > Huang’s Protocol

79. Binary Representation Example
D sends a message to process E 1 11
control C m1 1 1 01
001 D m2 m3 01 01
001 01 E
Theoretical Foundations > Termination > Huang’s Protocol

80. Binary Representation Example
D becomes idle and sends weight to C 1 11
control C m1 1
001 1 01
001 D m2 m3 01 01
001 01 E
Theoretical Foundations > Termination > Huang’s Protocol

81. Binary Representation Example
E receives message m3 and adds Wm to weight 1 11
control C m1 1
001 1 01
001 D m2 m3 01 01
001 01
001 E
Theoretical Foundations > Termination > Huang’s Protocol

82. Binary Representation Example
C receives D’s idle message and adds Wm to WC 1 11
111
control C m1 1
001 1 01
001 D m2 m3 01 01
001 01
001 E
Theoretical Foundations > Termination > Huang’s Protocol

83. Binary Representation Example
E becomes idle and sends weight to C 1 11
111
control C m1 1
001 1 01
001 D m2 m3 01
001 01
001 01
001 E
Theoretical Foundations > Termination > Huang’s Protocol

84. Binary Representation Example
C receives E’s idle message and adds Wm to WC 1 11
111
1000
control C m1 1
001 1 01
001 D m2 m3 01
001 01
001 01
001 E
Theoretical Foundations > Termination > Huang’s Protocol

85. Binary Representation Example
WC is 1 so the computation has terminated 1 11
111
1000
control C m1 1
001 1 01
001 D m2 m3 01
001 01
001 01
001 E
Theoretical Foundations > Termination > Huang’s Protocol

86. Space Efficiency Issue
Assume D keeps reactivating E, which then goes idle and sends weight to the control C
control C D
m12 E
Theoretical Foundations > Termination > Huang’s Protocol

87. Space Efficiency Issue
In this example, 13 bits are required after only 12 application messages!
control C D
m12 E
Theoretical Foundations > Termination > Huang’s Protocol

88. Space-Efficient Representation
Evenly divide the binary array into window slots
Every weight W is represented by (…)i, where (…) is a window of bits and i is the slot number
Let H be the maximum array size (e.g., H = 16 bits)
Let h be the window size (e.g., h = 4 bits)
Since H = 4h, we need 2 bits (because 22 = 4) to represent the slot number
For each slot i, it is assumed that all the bits in windows other than slot I have value 0
Hence, only 6 bits is required to represent a slot (e.g., = 0001|11)
Theoretical Foundations > Termination > Huang’s Protocol

89. Space-Efficient Scheme
When a process p has weight Wp = (0…01)i and needs to send a message, it slides the window right one slot and divides Wp into W1 and W2
Example:
Wp = 0001|01 (i.e., )
p sets W1 to 1000|10 (i.e., )
p sets W2 to 1000|10 (i.e., )
When a process p receives a message with weight Wm and the sum of Wp and Wm do not fit in one window slot, p forwards the smaller weight to the control process
Hence, only the weight of the control (WC) needs 16 bits
Theoretical Foundations > Termination > Huang’s Protocol

90. Space-Efficient Example
D shifts window and divides weight
control C
0001|10
1000|11 D
m32
1000|11
0011|10 E
Theoretical Foundations > Termination > Huang’s Protocol

91. Space-Efficient Example
E can’t keep the sum of WE and Wm in one slot
control C
0001|10
1000|11 D
m32
1000|11
0011|10 E
Theoretical Foundations > Termination > Huang’s Protocol

92. Space-Efficient Example
E forwards smaller weight (Wm) on to C
control C
0001|10
1000|11 D
m32
1000|11
1000|11
0011|10
0011|10 E
Theoretical Foundations > Termination > Huang’s Protocol

93. Space-Efficient Scheme
When a process p needs to send a message and has weight Wp = (0…01)i, where i is the last window slot, p must first send a weight request to the control with its weight attached
When the control process c receives a weight request of form (0…01)i, it will send a weight supply message to the requesting process of the form (0…01)j, where j < i
Theoretical Foundations > Termination > Huang’s Protocol

94. Space-Efficient Example
D needs to send m52 but can’t divide WD
control C
0001|11 D
m52
0101|11 E
Theoretical Foundations > Termination > Huang’s Protocol

95. Space-Efficient Example
D sends a request to C and goes idle
control C
request 0001|11
0001|11
0000|00 D
m52
0101|11 E
Theoretical Foundations > Termination > Huang’s Protocol

96. Space-Efficient Example
C receives D’s request and sends a supply back
control C
supply 0001|10
request 0001|11
0001|11
0000|00
0001|10 D
m52
0101|11 E
Theoretical Foundations > Termination > Huang’s Protocol

97. Space-Efficient Example
D can now send m52
control C
supply 0001|10
request 0001|11
0001|11
0000|00
0001|10
1000|11 D
m52
1000|11
0101|11 E
Theoretical Foundations > Termination > Huang’s Protocol

98. Space-Efficient Example
E receives m52
control C
supply 0001|10
request 0001|11
0001|11
0000|00
0001|10
1000|11 D
m52
1000|11
0101|11
1101|11 E
Theoretical Foundations > Termination > Huang’s Protocol

99. Upcoming Classes Permission-based Distributed Mutual Exclusion
Quorum-based Distributed Mutual Exclusion
Token-based Distributed Mutual Exclusion