1. Distributed Computing
Adam Morrison
Some slides based on “Art of Multiprocessor Programming” 1

2. Outline
Administration
Background
Mutual exclusion 2

3. Administration Mandatory attendance in 11 of the 13 lectures
Grade:
5% class participation
40% homework (~5 in the semester)
55% project 3

4. Project Analyze some topic or papers
Submit 2-5 pages summarizing your findings
Give a 15-minute talk 4

5. Outline
Administration
Background
Mutual exclusion 5

6. Distributed computing
code 6

7. Distributed computing
code
code
code
code 7

8. Models Message-passing: Communicate by messages over the network
code
code
network
code 8

9. Models Message-passing: Communicate by messages over the network
Shared-memory:
Communicate by reading/writing shared memory
code
code
network
code
code
code
memory
code 9

10. Models Message-passing & shared-memory are closely connected
Shared-memory can simulate message-passing
Proof: Implement message queues in software
Message-passing can simulate shared-memory*
*under assumptions on # of failures
Proof: “Sharing Memory Robustly in Message-Passing Systems” [Attiya, Bar-Noy, Dolev 1990]
Dijkstra award 10

11. This course Message-passing: Communicate by messages over the network
Shared-memory:
Communicate by reading/writing shared memory
code
code
network
code
code
code
memory
code 11

12. This course Foundations of distributed computing
Mainly about communication/synchronization between processors
Not about parallelism
Problems of agreement will come up a lot
New this year: blockchains (theory of) 12

13. Outline
Administration
Background
Mutual exclusion 13

14. Shared-memory model
code
code
code
memory
read
write
read
read
read 14

15. Shared-memory model Execution consists of a sequence of steps
Each step is a read/write of some memory location
(Don’t care about local computation!) P1 P2
time 15

16. Shared-memory model Execution consists of a sequence of steps
Each step is a read/write of some memory location
(Don’t care about local computation!)
Asynchronous system
Sudden unpredictable delays
Some scheduler picks next step in an arbitrary way P1 P2
time 16

17. Mutual exclusion Lock is an object (variable) with basic methods:
Lock() (acquire)
Unlock() (release)
Code between Lock(&L) and
Unlock(&L) is a critical section of L
The lock algorithm guarantees mutual exclusion: of all callers to lock(), only one can finish and enter the critical section until it exits the CS by calling unlock()
Lock L;
Lock(&L); …
Unlock(&L); 17

18. Mutual exclusion Process execution consists of repeat forever:
remainder section
entry section
critical section
exit section
Progress assumption: process can only halt while in the remainder section
defined by mutual exclusion algorithm 18

19. Mutual exclusion formalism
Interval (a0,a1) is the subsequence of events starting with a0 & ending with a1
time a0 a1 19

20. Mutual exclusion formalism
Overlapping intervals
time 20

21. Mutual exclusion formalism
Disjoint intervals
We write A -> B (A precedes B)
End event of A precedes start event of B
Precedence is a partial order
A->B and B->A might both be false
time A B 21

22. Mutual exclusion property
Let CSik be process i's k-th critical section execution
and CSjm be process j's m-th critical section execution
Then either:
CSik -> CSjm
CSjm -> CSik
CSik
CSjm
time 22

23. Plan
2-process solution
N-process solution
Fairness
Inherent costs 23

24. First attempt: flag principle
Entry (“lock()”) P0 P1
flag0 := 1
while (flag1){}
-- CS –-
flag0 := 0
flag1 := 1
while (flag0){}
-- CS –-
flag1 := 0
flag1 24

25. First attempt: flag principle
Exit (“unlock()”) P0 P1
flag0 := 1
while (flag1){}
-- CS –-
flag0 := 0
flag1 := 1
while (flag0){}
-- CS –-
flag1 := 0
flag1 25

26. First attempt: flag principle
while (flag1){}
-- CS –-
flag0 := 0
flag1 := 1
while (flag0){}
-- CS –-
flag1 := 0
flag0
flag1
No other flag up
My flag up
time 26

27. Mutual exclusion proof
Assume CSik overlaps CSjm
Consider each process's last (kth and mth) read and write before entering
Derive a contradiction 27

28. Proof From code: write0(flag0=true)  read0(flag1==false)  CS0
while (flag1){}
-- CS –-
flag0 := 0
flag1 := 1
while (flag0){}
-- CS –-
flag1 := 0
From code:
write0(flag0=true)  read0(flag1==false)  CS0
write1(flag1=true)  read1(flag0==false)  CS1 28

29. Proof From code: write0(flag0=true)  read0(flag1==false)  CS0
while (flag1){}
-- CS –-
flag0 := 0
flag1 := 1
while (flag0){}
-- CS –-
flag1 := 0
From code:
write0(flag0=true)  read0(flag1==false)  CS0
write1(flag1=true)  read1(flag0==false)  CS1
From assumption:
read0(flag1==false)  write1(flag1=true)
read1(flag0==false)  write0(flag0=true) 29

30. Proof From code: write0(flag0=true)  read0(flag1==false)  CS0
while (flag1){}
-- CS –-
flag0 := 0
flag1 := 1
while (flag0){}
-- CS –-
flag1 := 0
From code:
write0(flag0=true)  read0(flag1==false)  CS0
write1(flag1=true)  read1(flag0==false)  CS1
From assumption:
read0(flag1==false)  write1(flag1=true)
read1(flag0==false)  write0(flag0=true) 30

31. Proof From code: write0(flag0=true)  read0(flag1==false)  CS0
while (flag1){}
-- CS –-
flag0 := 0
flag1 := 1
while (flag0){}
-- CS –-
flag1 := 0
From code:
write0(flag0=true)  read0(flag1==false)  CS0
write1(flag1=true)  read1(flag0==false)  CS1
From assumption:
read0(flag1==false)  write1(flag1=true)
read1(flag0==false)  write0(flag0=true) 31

32. Proof From code: write0(flag0=true)  read0(flag1==false)  CS0
while (flag1){}
-- CS –-
flag0 := 0
flag1 := 1
while (flag0){}
-- CS –-
flag1 := 0
From code:
write0(flag0=true)  read0(flag1==false)  CS0
write1(flag1=true)  read1(flag0==false)  CS1
From assumption:
read0(flag1==false)  write1(flag1=true)
read1(flag0==false)  write0(flag0=true) 32

33. Proof From code: write0(flag0=true)  read0(flag1==false)  CS0
while (flag1){}
-- CS –-
flag0 := 0
flag1 := 1
while (flag0){}
-- CS –-
flag1 := 0
From code:
write0(flag0=true)  read0(flag1==false)  CS0
write1(flag1=true)  read1(flag0==false)  CS1
From assumption:
read0(flag1==false)  write1(flag1=true)
read1(flag0==false)  write0(flag0=true) 33

34. Impossible in a total order (of events)
Proof
A cycle!
Impossible in a total order (of events)
flag0 := 1
while (flag1){}
-- CS –-
flag0 := 0
flag1 := 1
while (flag0){}
-- CS –-
flag1 := 0
From code:
write0(flag0=true)  read0(flag1==false)  CS0
write1(flag1=true)  read1(flag0==false)  CS1
From assumption:
read0(flag1==false)  write1(flag1=true)
read1(flag0==false)  write0(flag0=true) 34

35. Problem: progress P0 P1 flag0 := 1 flag1 := 1 while (flag1){}
-- CS –-
flag0 := 0
flag1 := 1
while (flag0){}
-- CS –-
flag1 := 0
flag1 35

36. Deadlock and progress
Deadlock is a state in which no thread can complete its operation, because they’re all waiting for some condition that will never happen
Previous slide is an example
Mutual exclusion progress guarantees:
Deadlock-freedom:
If a thread is trying to enter the critical section then some thread must eventually enter the critical section
Starvation-freedom:
If a thread is trying to enter the critical section then this thread must eventually enter the critical section 36

37. 2nd attempt victim := 0 victim := 1 flag0 := 1 flag1 := 1
while (flag1 && victim==0){}
-- CS –-
flag0 := 0
victim := 1
flag1 := 1
while (flag0 && victim==1){}
-- CS –-
flag1 := 0
flag1 37

38. Peterson’s algorithm flag0 := 1 flag1 := 1 victim := 0 victim := 1
while (flag1 && victim==0){}
-- CS –-
flag0 := 0
flag1 := 1
victim := 1
while (flag0 && victim==1){}
-- CS –-
flag1 := 0
flag1 38

39. Peterson’s mutual exclusion proof
flag[i] := 1
victim := i
while (flag[1-i] && victim==i) {}
-- CS –-
flag[i] := 0
Assume both in CS 39

40. Peterson’s mutual exclusion proof
flag[i] := 1
victim := i
while (flag[1-i] && victim==i) {}
-- CS –-
flag[i] := 0
Assume both in CS
Suppose P0 is last to
write victim (writes 0)
P0 writes
victim:=0 40

41. Peterson’s mutual exclusion proof
flag[i] := 1
victim := i
while (flag[1-i] && victim==i) {}
-- CS –-
flag[i] := 0
Assume both in CS
Suppose P0 is last to
write victim (writes 0)
So it reads flag1==0
P0 writes
victim:=0
P0 reads
flag1==0 41

42. Peterson’s mutual exclusion proof
flag[i] := 1
victim := i
while (flag[1-i] && victim==i) {}
-- CS –-
flag[i] := 0
Assume both in CS
Suppose P0 is last to
write victim (writes 0)
So it reads flag1==0
So P1 writes flag1 later
P0 writes
victim:=0
P0 reads
flag1==0
P1 writes
flag1:=1 42

43. Peterson’s mutual exclusion proof
flag[i] := 1
victim := i
while (flag[1-i] && victim==i) {}
-- CS –-
flag[i] := 0
Assume both in CS
Suppose P0 is last to
write victim (writes 0)
So it reads flag1==0
So P1 writes flag1 later
But then it writes victim=> contradiction
P0 writes
victim:=0
P0 reads
flag1==0
P1 writes
flag1:=1
P1 writes
victim:=1 43

44. Peterson’s deadlock-freedom proof
flag[i] := 1
victim := i
while (flag[1-i] && victim==i) {}
-- CS –-
flag[i] := 0
Process blocked:
Only at while loop
Only if other’s flag==1
Only if it is victim
In solo execution: other’s flag is false
Otherwise: somebody isn’t the victim 44

45. Peterson’s starvation-freedom proof
flag[i] := 1
victim := i
while (flag[1-i] && victim==i) {}
-- CS –-
flag[i] := 0
Process i blocked only if 1-i repeatedly enters so that
flag[1-i] && victim==i
But when 1-i re-enters:
It sets victim to 1-i
So i gets in 45

46. Plan
2-process solution
N-process solution
Fairness
Inherent costs 46

47. Filter algorithm Generalization of Peterson’s
N-1 levels (waiting rooms) that a process has to go through to enter CS
At each level:
At least one enters
At least one blocked if many try
I.e., at most N-i pass into level i
Only one process makes it to CS (level N-1)
remainder cs
Art of Multiprocessor Programming 47

48. Filter algorithm int level[N] // level of process i
int victim[N] // victim at level L
for (L = 1; L < N; L++) {
level[i] = L
victim[L] = i
while (($ k!=i:level[k]>=L) && victim[L] == i) {} }
-- CS –-
level[i] = 0
flag1 48

49. Filter algorithm int level[N] // level of process i
int victim[N] // victim at level L
for (L = 1; L < N; L++) {
level[i] = L
victim[L] = i
while (($ k!=i:level[k]>=L) && victim[L] == i) {} }
-- CS –-
level[i] = 0
flag1
One level at a time 49

50. Announce intention to enter level L
Filter algorithm
int level[N] // level of process i
int victim[N] // victim at level L
for (L = 1; L < N; L++) {
level[i] = L
victim[L] = i
while (($ k!=i:level[k]>=L) && victim[L] == i) {} }
-- CS –-
level[i] = 0
flag1
Announce intention to enter level L 50

51. Give priority to anyone but me
Filter algorithm
int level[N] // level of process i
int victim[N] // victim at level L
for (L = 1; L < N; L++) {
level[i] = L
victim[L] = i
while (($ k!=i:level[k]>=L) && victim[L] == i) {} }
-- CS –-
level[i] = 0
flag1
Give priority to anyone but me 51

52. Filter algorithm int level[N] // level of process i
int victim[N] // victim at level L
for (L = 1; L < N; L++) {
level[i] = L
victim[L] = i
while (($ k!=i:level[k]>=L) && victim[L] == i) {} }
-- CS –-
level[i] = 0
flag1
Wait as long as someone else is at same or higher level, and I’m the victim 52

53. Process enters level L when it completes the loop
Filter algorithm
int level[N] // level of process i
int victim[N] // victim at level L
for (L = 1; L < N; L++) {
level[i] = L
victim[L] = i
while (($ k!=i:level[k]>=L) && victim[L] == i) {} }
-- CS –-
level[i] = 0
flag1
Process enters level L when it completes the loop 53

54. . . . 1 5
level[.] 2 1 1 1 1 2
victim[.]
. .
while(($ k != i level[k] >= L)&&
(victim[L] == i)) {};
while(($ k != i level[k] >= L)&&
(victim[L] == i)) {}; 5
Art of Multiprocessor Programming 54

55. . . . 1 5
level[.] 1 2
victim[.]
. . 5 CS
Art of Multiprocessor Programming 55

56. Claim Start at level L=0 At most n-L threads enter level L
© 2003 Herlihy and Shavit
Claim
Start at level L=0
At most n-L threads enter level L
Mutual exclusion at level L=n-1
remainder
L=0
L=1
L=n-2 cs
L=n-1
Art of Multiprocessor Programming

57. Induction hypothesis No more than n-(L-1) at level L-1
© 2003 Herlihy and Shavit
Induction hypothesis
No more than n-(L-1) at level L-1
Induction step: by contradiction
Assume all at level L-1 enter level L
A last to write victim[L]
B is any other thread at level L
remainder
assume
L-1 has n-(L-1)
L has n-L cs
prove
Art of Multiprocessor Programming

58. Proof structure A B remainder Assumed to enter L-1 n-(L-1) = 4
© 2003 Herlihy and Shavit
Proof structure
remainder
Assumed to enter L-1 A B
n-(L-1) = 4
n-(L-1) = 4
Last to
write
victim[L] cs
By way of contradiction
all enter L
Show that A must have seen
B in level[L] and since victim[L] == A
could not have entered
Art of Multiprocessor Programming

59. Just like Peterson (1) writeB(level[B]=L)writeB(victim[L]=B)
© 2003 Herlihy and Shavit
Just like Peterson
(1) writeB(level[B]=L)writeB(victim[L]=B)
for (L = 1; L < N; L++) {
level[i] = L
victim[L] = i
while (($ k!=i:level[k]>=L) && victim[L] == i) {} }
From the Code 59 59

60. From the code (2) writeA(victim[L]=A)readA(level[B])
© 2003 Herlihy and Shavit
From the code
(2) writeA(victim[L]=A)readA(level[B])
readA(victim[L])
for (L = 1; L < N; L++) {
level[i] = L
victim[L] = i
while (($ k!=i:level[k]>=L) && victim[L] == i) {} } 60 60

61. By assumption, A is the last thread to write victim[L]
© 2003 Herlihy and Shavit
By assumption
(3) writeB(victim[L]=B)writeA(victim[L]=A)
By assumption, A is the last thread to write victim[L] 61 61

62. Combining observations
© 2003 Herlihy and Shavit
Combining observations
(1) writeB(level[B]=L)writeB(victim[L]=B)
(3) writeB(victim[L]=B)writeA(victim[L]=A)
(2) writeA(victim[L]=A)readA(level[B])
readA(victim[L]) 62 62

63. Combining observations
© 2003 Herlihy and Shavit
Combining observations
(1) writeB(level[B]=L)writeB(victim[L]=B)
(3) writeB(victim[L]=B)writeA(victim[L]=A)
(2) writeA(victim[L]=A)readA(level[B])
readA(victim[L])
A read level[B] ≥ L, and victim[L] = A, so it could not have entered level L! 63 63

64. No starvation Filter Lock satisfies properties:
© 2003 Herlihy and Shavit
No starvation
Filter Lock satisfies properties:
Just like Peterson algorithm at any level
So no one starves
But what about fairness?
Threads can be overtaken by others
Art of Multiprocessor Programming 64 64

65. Bounded waiting Want stronger fairness guarantees
© 2003 Herlihy and Shavit
Bounded waiting
Want stronger fairness guarantees
Thread not “overtaken” too much
If A starts before B, then A enters before B?
But what does “start” mean?
Need to adjust definitions…
Art of Multiprocessor Programming 65 65

66. Bounded waiting Divide entry section into 2 parts: Doorway interval:
© 2003 Herlihy and Shavit
Bounded waiting
Divide entry section into 2 parts:
Doorway interval:
Written DA
Always finishes in finite steps
Waiting interval:
Written WA
May take unbounded steps
Art of Multiprocessor Programming

67. r-Bounded waiting For threads A and B: If DAk  DB m
© 2003 Herlihy and Shavit
r-Bounded waiting
For threads A and B:
If DAk  DB m
A's k-th doorway precedes B's m-th doorway
Then CSAk  CSBm+r
A's k-th critical section precedes B's m+r-th critical section
B cannot overtake A more than r times
Art of Multiprocessor Programming

68. What’s r for Peterson’s algorithm?
© 2003 Herlihy and Shavit
What’s r for Peterson’s algorithm?
flag[i] := 1
victim := i
while (flag[1-i] && victim==i) {}
-- CS –-
flag[i] := 0
Answer: r = 0

69. What’s r for Filter lock?
© 2003 Herlihy and Shavit
What’s r for Filter lock?
for (L = 1; L < N; L++) {
level[i] = L
victim[L] = i
while (($ k!=i:level[k]>=L) && victim[L] == i) {} }
Answer: there is no value of “r”

70. Fairness Filter Lock satisfies properties: No one starves
© 2003 Herlihy and Shavit
Fairness
Filter Lock satisfies properties:
No one starves
But very weak fairness
Can be overtaken arbitrary # of times
So being fair is stronger than avoiding starvation
And filter is pretty lame…
Art of Multiprocessor Programming

71. First-come-first-served
© 2003 Herlihy and Shavit
First-come-first-served
For threads A and B:
If DAk  DB m
A's k-th doorway precedes B's m-th doorway
Then CSAk  CSBm
A's k-th critical section precedes B's m-th critical section
B cannot overtake A
Art of Multiprocessor Programming

72. Bakery algorithm [Lamport]
© 2003 Herlihy and Shavit
Bakery algorithm [Lamport]
Provides First-Come-First-Served for n threads
How?
Take a “number”
Wait until lower numbers have been served
Lexicographic order
(a,i) > (b,j)
If a > b, or a = b and i > j
Art of Multiprocessor Programming

73. Bakery algorithm flag1 Bool flag[N] // initially false
Label label[N] // initially 0
flag[i] = true;
t = max(label[0], …, label[N-1])
label[i] = t+1
while ($ k:flag[k] &&
(label[i],i) > (label[k],k)) {}
-- CS –-
flag[i] = false
flag1 73

74. Bakery algorithm flag1 Bool flag[N] // initially false
Label label[N] // initially 0
flag[i] = true;
t = max(label[0], …, label[N-1])
label[i] = t+1
while ($ k:flag[k] &&
(label[i],i) > (label[k],k)) {}
-- CS –-
flag[i] = false
flag1
Doorway 74

75. Bakery algorithm flag1 Bool flag[N] // initially false
Label label[N] // initially 0
flag[i] = true;
t = max(label[0], …, label[N-1])
label[i] = t+1
while ($ k:flag[k] &&
(label[i],i) > (label[k],k)) {}
-- CS –-
flag[i] = false
flag1
I’m interested 75

76. Bakery algorithm flag1 Bool flag[N] // initially false
Label label[N] // initially 0
flag[i] = true;
t = max(label[0], …, label[N-1])
label[i] = t+1
while ($ k:flag[k] &&
(label[i],i) > (label[k],k)) {}
-- CS –-
flag[i] = false
flag1
Someone is interested 76

77. Someone is interested, whose label is lexicographically lower
Bakery algorithm
Bool flag[N] // initially false
Label label[N] // initially 0
flag[i] = true;
t = max(label[0], …, label[N-1])
label[i] = t+1
while ($ k:flag[k] &&
(label[i],i) > (label[k],k)) {}
-- CS –-
flag[i] = false
flag1
Someone is interested, whose label is lexicographically lower 77

78. No longer interested. Labels keep increasing.
Bakery algorithm
Bool flag[N] // initially false
Label label[N] // initially 0
flag[i] = true;
t = max(label[0], …, label[N-1])
label[i] = t+1
while ($ k:flag[k] &&
(label[i],i) > (label[k],k)) {}
-- CS –-
flag[i] = false
flag1
No longer interested. Labels keep increasing. 78

79. No deadlock There is always one thread with earliest label
© 2003 Herlihy and Shavit
No deadlock
There is always one thread with earliest label
Ties are impossible (why?)
Art of Multiprocessor Programming

80. FCFS If DA  DB then A's label is smaller And:
© 2003 Herlihy and Shavit
flag[i] = true;
t = max(label[0], …, label[N-1])
label[i] = t+1
while ($ k:flag[k] &&
(label[i],i) > (label[k],k)) {}
FCFS
If DA  DB then
A's label is smaller
And:
writeA(label[A])  readB(label[A]) 
writeB(label[B])  readB(flag[A])
So B sees
smaller label for A
locked out while flag[A] is true

81. Mutex Suppose A and B in CS together Suppose A has earlier label
© 2003 Herlihy and Shavit
flag[i] = true;
t = max(label[0], …, label[N-1])
label[i] = t+1
while ($ k:flag[k] &&
(label[i],i) > (label[k],k)) {}
Mutex
Suppose A and B in CS together
Suppose A has earlier label
When B entered, it must have seen
flag[A] is false, or label[A] > label[B]

82. Mutex Suppose A and B in CS together Suppose A has earlier label
© 2003 Herlihy and Shavit
flag[i] = true;
t = max(label[0], …, label[N-1])
label[i] = t+1
while ($ k:flag[k] &&
(label[i],i) > (label[k],k)) {}
Mutex
Suppose A and B in CS together
Suppose A has earlier label
When B entered, it must have seen
flag[A] is false, or label[A] > label[B]
Could B see label[A] > label[B] ?
By assumption, A now has earlier label
Labels are strictly increasing => So, no

83. Mutex Suppose A and B in CS together Suppose A has earlier label
© 2003 Herlihy and Shavit
flag[i] = true;
t = max(label[0], …, label[N-1])
label[i] = t+1
while ($ k:flag[k] &&
(label[i],i) > (label[k],k)) {}
Mutex
Suppose A and B in CS together
Suppose A has earlier label
When B entered, it must have seen
flag[A] is false, or label[A] > label[B]
Could B see label[A] > label[B] ?
By assumption, A now has earlier label
Labels are strictly increasing => So, no
LabelingB  readB(flag[A]==false)  writeA(flag[A]=true)  LabelingA
Contradiction

84. Mutual exclusion breaks if label overflow (and decreases)
Bakery overflow bug
Bool flag[N] // initially false
Label label[N] // initially 0
flag[i] = true;
t = max(label[0], …, label[N-1])
label[i] = t+1
while ($ k:flag[k] &&
(label[i],i) > (label[k],k)) {}
-- CS –-
flag[i] = false
flag1
Mutual exclusion breaks if label overflow (and decreases) 84

85. Timestamps Label variable is really a timestamp Need ability to
© 2003 Herlihy and Shavit
Timestamps
Label variable is really a timestamp
Need ability to
Read others' timestamps
Compare them
Generate a later timestamp
Can we do this without overflow?
Art of Multiprocessor Programming

86. The good news One can construct a Wait-free (no mutual exclusion)
© 2003 Herlihy and Shavit
The good news
One can construct a
Wait-free (no mutual exclusion)
Concurrent
Timestamping system
That never overflows
Art of Multiprocessor Programming

87. The good news Bad This part is hard One can construct a
© 2003 Herlihy and Shavit
The good news
Bad
One can construct a
Wait-free (no mutual exclusion)
Concurrent
Timestamping system
That never overflows
This part is hard
Art of Multiprocessor Programming

88. Philosophical question
© 2003 Herlihy and Shavit
Philosophical question
The Bakery Algorithm is
Succinct,
Elegant, and
Fair.
Q: So why isn't it practical?
A: Well, you have to read N distinct variables
Art of Multiprocessor Programming

89. Shared memory variables
© 2003 Herlihy and Shavit
Shared memory variables
Shared read/write memory locations called Registers (historical reasons)
Come in different flavors
Multi-Reader-Single-Writer (flag[])
Multi-Reader-Multi-Writer (victim[])
Art of Multiprocessor Programming

90. © 2003 Herlihy and Shavit
Theorem (lower bound)
At least N MRSW (multi-reader/single-writer) registers are needed to solve deadlock-free mutual exclusion.
Art of Multiprocessor Programming

91. Proving algorithmic impossibility
© 2003 Herlihy and Shavit
Proving algorithmic impossibility C
To show no algorithm exists:
Assume by way of contradiction one does,
Show a bad execution that violates properties:
In our case assume an algorithm for deadlock free mutual exclusion using < N registers
write CS
Art of Multiprocessor Programming

92. Proof a0 a3 a2 a1 Threads are state machines
© 2003 Herlihy and Shavit
Proof
Threads are state machines
Execution events are transitions a0 a3 a2 a1
Art of Multiprocessor Programming

93. …can't tell whether A is in critical section
© 2003 Herlihy and Shavit
Proof
Each thread must write to some register A B C
write
write CS CS CS
…can't tell whether A is in critical section
Art of Multiprocessor Programming

94. Upper bound Bakery algorithm Uses 2N MRSW registers
© 2003 Herlihy and Shavit
Upper bound
Bakery algorithm
Uses 2N MRSW registers
So the bound is (pretty) tight
But what if we use MRMW registers?
Like victim[] ?
Art of Multiprocessor Programming

95. Philosophical motivation
© 2003 Herlihy and Shavit
Philosophical motivation
MRMW register: hardware implements mutual exclusion for us
Two concurrent writes always get ordered
Does this buy us anything?
Art of Multiprocessor Programming 95 95

96. © 2003 Herlihy and Shavit
Bad news theorem
At least N MRMW multi-reader/multi-writer registers are needed to solve deadlock-free mutual exclusion.
(So multiple writers don't help)
Art of Multiprocessor Programming

97. © 2003 Herlihy and Shavit
Theorem (for 2 threads)
Theorem: Deadlock-free mutual exclusion for 2 threads requires at least 2 multi-reader multi-writer registers
Proof: assume one register suffices and derive a contradiction
Art of Multiprocessor Programming

98. Two-thread execution R A B CS CS Threads run, reading and writing R
© 2003 Herlihy and Shavit
Two-thread execution A B R
Write(R) CS CS
Threads run, reading and writing R
Deadlock free so at least one gets in
Art of Multiprocessor Programming

99. Covering state Covering state: B
© 2003 Herlihy and Shavit
Covering state
Covering state:
at least 1 thread about to write to each register
but memory
looks as if CS
is empty (and no one is trying to enter) B
Write(R)
Art of Multiprocessor Programming 99 99

100. Register looks as if CS empty
© 2003 Herlihy and Shavit
Covering state for 1 register B
Write(R)
In any protocol B has to write to the register before entering CS, so stop it just before.
Register looks as if CS empty
Art of Multiprocessor Programming

101. A runs, possibly writes to the register, enters CS
© 2003 Herlihy and Shavit
Proof: assume cover of 1 A B
Write(R)
A runs, possibly writes to the register, enters CS CS
Art of Multiprocessor Programming

102. Proof: assume cover of 1 A B B runs, first obliterating
© 2003 Herlihy and Shavit
Proof: assume cover of 1 A B
B runs,
first obliterating
any trace of A,
then also enters
the critical
section
Write(R) CS CS
Art of Multiprocessor Programming

103. © 2003 Herlihy and Shavit
Theorem
Deadlock-free mutual exclusion for 3 threads requires at least 3 multi-reader multi-writer registers
Art of Multiprocessor Programming

104. Registers look as if CS empty
© 2003 Herlihy and Shavit
Proof: assume cover of 2 A B C
Write(RB)
Write(RC)
Only 2 registers
Registers look as if CS empty
Art of Multiprocessor Programming

105. Writes to one or both registers, enters CS
© 2003 Herlihy and Shavit
Run A solo A B C
Write(RB)
Write(RC)
Writes to one or both registers, enters CS CS
Art of Multiprocessor Programming

106. Other threads obliterate evidence that A entered CS
© 2003 Herlihy and Shavit
Obliterate traces of A A B C
Write(RB)
Write(RC)
Other threads obliterate evidence that A entered CS CS
Art of Multiprocessor Programming

107. CS looks empty, so another thread gets in
© 2003 Herlihy and Shavit
Mutual exclusion fails A B C
Write(RB)
Write(RC)
CS looks empty, so another thread gets in CS CS
Art of Multiprocessor Programming

108. Constructing a covering state
© 2003 Herlihy and Shavit
Constructing a covering state
Proved: a contradiction starting from a covering state for 2 registers
Claim: a covering state for 2 registers is reachable from any state where CS is empty and no process is trying to enter it
Art of Multiprocessor Programming

109. © 2003 Herlihy and Shavit
Covering state for 2 A B
Write(RA)
Write(RB)
If we run B through CS 3 times, first write of B
must twice be to some register, say RB
Art of Multiprocessor Programming

110. © 2003 Herlihy and Shavit
Covering state for 2 A B
Write(RA)
Write(RB)
Start with B covering register RB for the 1st time
Run A until it is about to write to uncovered RA
Are we done?
Art of Multiprocessor Programming

111. Covering state for 2 A B NO! A could have written to RB
© 2003 Herlihy and Shavit
Covering state for 2 A B
Write(RA)
Write(RB)
NO! A could have written to RB
So CS no longer looks empty to thread C
Art of Multiprocessor Programming

112. Covering state for 2 A B Run B obliterating traces of A in RB
© 2003 Herlihy and Shavit
Covering state for 2 A B
Write(RA)
Write(RB)
Run B obliterating traces of A in RB
Run B again until it is about to write to RB
Now we are done
Art of Multiprocessor Programming

113. Inductively we can show
© 2003 Herlihy and Shavit
Inductively we can show A B C
Write(RA)
Write(RB)
Write(RC)
There is a covering state
Where k threads not in CS cover k distinct registers
Proof follows when k = N-1
Art of Multiprocessor Programming

114. © 2003 Herlihy and Shavit
Summary
In the 1960s several incorrect solutions to starvation-free mutual exclusion using RW-registers were published…
Today we know how to solve FIFO N thread mutual exclusion using 2N RW-Registers
Art of Multiprocessor Programming