1. The Relative Power of Synchronization Operations
Companion slides for
The Art of Multiprocessor Programming
by Maurice Herlihy & Nir Shavit

2. Wait-Free Implementation
Every method call completes in finite number of steps
Implies no mutual exclusion
We looked at wait-free implementations because they imply no mutual exclusion.
(2)
Art of Multiprocessor Programming

3. Rationale for wait-freedom
We wanted atomic registers to implement mutual exclusion
So we couldn’t use mutual exclusion to implement atomic registers
But wait, there’s more!
Art of Multiprocessor Programming

4. Basic Questions
Wait-Free synchronization might be a good idea in principle
But how do you do it
Systematically?
Correctly?
Efficiently?
Art of Multiprocessor Programming

5. FIFO Queue: Enqueue Method
q.enq( )
Art of Multiprocessor Programming

6. FIFO Queue: Dequeue Method
q.deq()/
Art of Multiprocessor Programming

7. Two-Thread Wait-Free Queue
public class LockFreeQueue {
int head = 0, tail = 0;
Item[QSIZE] items;
public void enq(Item x) {
while (tail-head == QSIZE) {};
items[tail % QSIZE] = x; tail++; }
public Item deq() {
while (tail-head == 0) {}
Item item = items[head % QSIZE];
head++; return item; }}
Art of Multiprocessor Programming

8. What About Multiple Dequeuers?
Art of Multiprocessor Programming

9. Grand Challenge Only new aspect Implement a FIFO queue Wait-free
Linearizable
From atomic read-write registers
Multiple dequeuers
(1)
Art of Multiprocessor Programming

10. Consensus While you are ruminating on the grand challenge…
We will give you another puzzle
Consensus
Will be important …
Art of Multiprocessor Programming

11. Consensus: Each Thread has a Private Input19 32 21
Art of Multiprocessor Programming

12. They Communicate
Art of Multiprocessor Programming

13. They Agree on One Thread’s Input19 19 19
Art of Multiprocessor Programming

14. Formally: Consensus Consistent: all threads decide the same value
Valid: the common decision value is some thread's input
As we will see later consensus can be formulated as a linearizable object in which threads in the sequential
execution decide on the value of the first thread to complete.
Art of Multiprocessor Programming

15. No Wait-Free Implementation of Consensus using Registers
???
???
Art of Multiprocessor Programming

16. Formally
Theorem [adapted from Fischer, Lynch, Paterson]: There is no wait- free implementation of n-thread consensus, n>1, from read-write registers
Implication: asynchronous computability fundamentally different from Turing computability
This is one of the most fundamental theorems of distributed computing, proved originally by Fischer, Lynch, and Paterson in the 1980’s. It says that there are problems that are easily solvable by reading and writing to memory on a single processor, a turing machine, but are not solvable on a multiprocessor. As we will see, this will have profound implications on multiprocessor hardware design.
We will give here a proof of their theorem in the context of shared memory as proven by Herlihy.
Art of Multiprocessor Programming

17. Proof Strategy Assume otherwise
Reason about the properties of any such protocol
Derive a contradiction
Quod Erat Demonstrandum
Suffices to prove for binary consensus and n=2
Art of Multiprocessor Programming

18. Wait-Free Computation
A moves
B moves
Either A or B “moves”
Moving means
Register read
Register write
It suffices to prove the claim for two threads. Here is how we model computation histories.
Art of Multiprocessor Programming

19. The Two-Move Tree Final states Initial state
(2)
Art of Multiprocessor Programming

20. Decision Values 1 1 1 1
Art of Multiprocessor Programming

21. Bivalent: Both Possible1 1 1 1
Art of Multiprocessor Programming

22. Univalent: Single Value Possible1 1 1 1
Art of Multiprocessor Programming

23. x-valent: x Only Possible Decision1 1 1 1
Art of Multiprocessor Programming

24. Summary Wait-free computation is a tree Bivalent system states
Outcome not fixed
Univalent states
Outcome is fixed
May not be “known” yet
1-Valent and 0-Valent states
Art of Multiprocessor Programming

25. Claim Some initial state is bivalent Outcome depends on
Chance
Whim of the scheduler
Multiprocessor gods do play dice …
Lets prove this claim
Art of Multiprocessor Programming

26. Univalent: all executions must decide 0
Both Inputs 0
Univalent: all executions must decide 0
(2)
Art of Multiprocessor Programming
(1)

27. Including this solo execution by A
Both Inputs 0
Including this solo execution by A
(1)
Art of Multiprocessor Programming
(1)

28. All executions must decide 1
Both Inputs 1 1 1
All executions must decide 1
(2)
Art of Multiprocessor Programming
(1)

29. Including this solo execution by B
Both Inputs 1 1
Including this solo execution by B
(1)
Art of Multiprocessor Programming
(1)

30. What if inputs differ? 1 By Way of contradiction: If univalent1
Imagine that all the executions still decide on one input or another…assume the claim is false and
lets show the set of possible executions must then include two executions with different values.
By Way of contradiction: If univalent
all executions must decide on same value
(2)
Art of Multiprocessor Programming

31. The Possible Executions1
Include the solo execution by A
that decides 0
(2)
Art of Multiprocessor Programming

32. The Possible Executions1
Also include the solo execution by B
which we know decides 1
(2)
Art of Multiprocessor Programming

33. Possible Executions Include
How univalent is that?
(QED) 1
Since e
Solo execution by A must decide 0
Solo execution by B must decide 1
Art of Multiprocessor Programming

34. Critical States critical 0-valent 1-valent
A critical state is a bivalent state in which the next step by one thread will lead to a 0-valent state and the next step by the other will lead to a 1-valent state
0-valent
1-valent
(3)
Art of Multiprocessor Programming
(3)

35. From a Critical State c 0-valent 1-valent
If A goes first, protocol decides 0
If B goes first, protocol decides 1
Art of Multiprocessor Programming

36. Reaching Critical State
initially bivalent CB
univalent CA CB
univalent CA
univalent
Why must there be such a critical state? Start from a bivalent state. If it is not critical, run A until about to make state 0-valent, if not a critical state, that is, if B’s step will not take the state to a 1-valent state, then run B until in is about to take system to a 1-valent state. If this state is not critical, run A again, then B, and so on. Since this is a wai-free execution, within a finite number of steps each will finish, so somewhere before they must reach a critical state. c
univalent
0-valent
1-valent
Art of Multiprocessor Programming

37. Critical States Starting from a bivalent initial state
The protocol can reach a critical state
Otherwise we could stay bivalent forever
And the protocol is not wait-free
Art of Multiprocessor Programming

38. Model Dependency So far, memory-independent! True for Registers
Message-passing
Carrier pigeons
Any kind of asynchronous computation
Art of Multiprocessor Programming

39. Read-Write Memory Reads and/or writes To same/different registers
Art of Multiprocessor Programming

40. Completing the Proof Lets look at executions that:
Start from a critical state
Threads cause state to become univalent by reading or writing to same/different registers
End within a finite number of steps deciding either 0 or 1
Show this leads to a contradiction
Art of Multiprocessor Programming

41. Possible Interactions
A reads x
x.read()
y.read()
x.write()
y.write() ?
A reads y
Tp prove that we cannot solve consensus using read-write registers, lets go over all the possible interactions among the threads and show that for each such execution they lead to a contradiction.
Art of Multiprocessor Programming

42. Some Thread Reads Contradiction c 1 A runs solo, eventually decides 0
B reads x
A runs solo, eventually decides 1 1
Contradiction
This is the case in which some thread reads (and the other either reads or writes). Two possible executions, one in which A runs solo and decides 0 and another in which A runs after B takes its critical read step. Leads to a contradiction since A cannot tell that B has read…so two executions lead to the same state but the system is supposed to have two different valencies for it, a contradiction.
States look the same to A
Art of Multiprocessor Programming

43. Possible Interactions
x.read()
y.read()
x.write()
y.write() no ?
Art of Multiprocessor Programming

44. Writing Distinct Registers
A writes y
B writes x
A writes y
B writes x
Contradiction 1
The song is the same
Art of Multiprocessor Programming

45. Possible Interactions
x.read()
y.read()
x.write()
y.write() no ?
Art of Multiprocessor Programming

46. Writing Same Registersc
B writes x
A writes x
A runs solo, eventually decides 0
A writes x
Contradiction
A runs solo, eventually decides 1 1
States look the same to A
Art of Multiprocessor Programming

47. That’s All, Folks! QED no Art of Multiprocessor Programming x.read()
y.read()
x.write()
y.write() no
QED
Art of Multiprocessor Programming

48. Recap: Atomic Registers Can’t Do Consensus
If protocol exists
It has a bivalent initial state
Leading to a critical state
What’s up with the critical state?
Case analysis for each pair of methods
As we showed, all lead to a contradiction
Art of Multiprocessor Programming

49. What Does Consensus have to do with Concurrent Objects?
Consensus can be formulated as a linearizable object in which threads in the sequential
execution decide on the value of the first thread to complete.
Art of Multiprocessor Programming

50. Consensus Object public interface Consensus {
Object decide(Object value); }
(4)
Art of Multiprocessor Programming
(4)

51. Concurrent Consensus Object
We consider only one time objects: each thread can execute a method only once
Linearizable to sequential consensus object in which
the thread who’s input was decided on completed its method first
A linearizable object in which threads in the sequential
execution decide on the value of the first thread to complete. The consensus objects we will consider will be one time objects, that is, each thread executes a method once.
Art of Multiprocessor Programming

52. Java Jargon Watch Define Consensus protocol as an abstract class
We implement some methods
Leave you to do the rest …
Art of Multiprocessor Programming

53. Generic Consensus Protocol
abstract class ConsensusProtocol implements Consensus {
protected Object[] proposed =
new Object[N];
private void propose(Object value) {
proposed[ThreadID.get()] = value; }
abstract public Object
decide(object value); }}
Note the use of a protected for proposed. A protected method is visible to inheriting classes, even not part of the same package. A package scope (default) method is not. That is the only difference between protected and package scope.
method are restricted in accessing it. See example below:
package one;
public class A {
protected int p; }
////////////////////////////////
package two;
import one. A;
class B extends A
void myMethod()
p = 1; // ok
A a = new A();
a.p = 1; // not ok, p would have to be public for this to work.
(4)
Art of Multiprocessor Programming
(4)

54. Generic Consensus Protocol
abstract class ConsensusProtocol implements Consensus {
protected Object[] proposed =
new Object[N];
private void propose(Object value) {
proposed[ThreadID.get()] = value; }
abstract public Object
decide(object value); }}
Each thread’s proposed value
(4)
Art of Multiprocessor Programming
(4)

55. Generic Consensus Protocol
abstract class ConsensusProtocol implements Consensus {
protected Object[] proposed =
new Object[N];
private void propose(Object value) {
proposed[ThreadID.get()] = value; }
abstract public Object
decide(object value); }}
Propose a value
(4)
Art of Multiprocessor Programming
(4)

56. Generic Consensus Protocol
abstract class ConsensusProtocol implements Consensus {
protected Object[] proposed =
new Object[N];
private void propose(Object value) {
proposed[ThreadID.get()] = value; }
abstract public Object
decide(object value); }}
Decide a value: abstract method means subclass does the heavy lifting (real work)
(4)
Art of Multiprocessor Programming
(4)

57. Can a FIFO Queue Implement Consensus?

58. FIFO Queue with red and black balls
FIFO Consensus
proposed array
FIFO Queue with red and black balls 8
Coveted red ball
Dreaded black ball
Art of Multiprocessor Programming

59. Protocol: Write Value to Array1
Art of Multiprocessor Programming

60. Protocol: Take Next Item from Queue8 1
Art of Multiprocessor Programming

61. Protocol: Take Next Item from Queue
I got the dreaded black ball, so I will decide the other’s value from the array
I got the coveted red ball, so I will decide my value 1 8
Art of Multiprocessor Programming

62. Consensus Using FIFO Queue
public class QueueConsensus
extends ConsensusProtocol {
private Queue queue;
public QueueConsensus() {
queue = new Queue();
queue.enq(Ball.RED);
queue.enq(Ball.BLACK); } …
Art of Multiprocessor Programming

63. Initialize Queue public class QueueConsensus
extends ConsensusProtocol {
private Queue queue;
public QueueConsensus() {
this.queue = new Queue();
this.queue.enq(Ball.RED);
this.queue.enq(Ball.BLACK); } … 8
Art of Multiprocessor Programming

64. Who Won? public class QueueConsensus extends ConsensusProtocol {
private Queue queue; …
public decide(object value) {
propose(value);
Ball ball = this.queue.deq();
if (ball == Ball.RED)
return proposed[i];
else
return proposed[1-i]; }
Art of Multiprocessor Programming

65. Race to dequeue first queue item
Who Won?
public class QueueConsensus
extends ConsensusProtocol {
private Queue queue; …
public decide(object value) {
propose(value);
Ball ball = this.queue.deq();
if (ball == Ball.RED)
return proposed[i];
else
return proposed[1-ij]; }
Race to dequeue first queue item
Art of Multiprocessor Programming

66. Who Won? I win if I was first public class QueueConsensus
extends ConsensusProtocol {
private Queue queue; …
public decide(object value) {
propose(value);
Ball ball = this.queue.deq();
if (ball == Ball.RED)
return proposed[i];
else
return proposed[1-i]; }
i = ThreadID.get();
I win if I was first
Art of Multiprocessor Programming

67. Other thread wins if I was second
Who Won?
public class QueueConsensus
extends ConsensusProtocol {
private Queue queue; …
public decide(object value) {
propose(value);
Ball ball = this.queue.deq();
if (ball == Ball.RED)
return proposed[i];
else
return proposed[1-i]; }
Other thread wins if I was second
Art of Multiprocessor Programming

68. Why does this Work? If one thread gets the red ball
Then the other gets the black ball
Winner decides her own value
Loser can find winner’s value in array
Because threads write array
Before dequeueing from queue
Art of Multiprocessor Programming

69. Theorem We can solve 2-thread consensus using only
A two-dequeuer queue, and
Some atomic registers
Art of Multiprocessor Programming

70. Implications contradiction Given Assume there exists
A consensus protocol from queue and registers
Assume there exists
A queue implementation from atomic registers
Substitution yields:
A wait-free consensus protocol from atomic registers
contradiction
(1)
Art of Multiprocessor Programming
(1)

71. Corollary It is impossible to implement
a two-dequeuer wait-free FIFO queue
from read/write memory.
Art of Multiprocessor Programming

72. Consensus Numbers An object X has consensus number n
If it can be used to solve n-thread consensus
Take any number of instances of X
together with atomic read/write registers
and implement n-thread consensus
But not (n+1)-thread consensus
Art of Multiprocessor Programming

73. Consensus Numbers Theorem
Atomic read/write registers have consensus number 1
Multi-dequeuer FIFO queues have consensus number 2
Similar argument about other data types such as sets, stacks, double-ended queues, priority queues.
Art of Multiprocessor Programming

74. Consensus Numbers Measure Synchronization Power
Theorem
If you can implement X from Y
And X has consensus number c
Then Y has consensus number at least c
Art of Multiprocessor Programming

75. Multiple Assignment Theorem
Atomic registers cannot implement multiple assignment
(1)
Art of Multiprocessor Programming
(1)

76. Proof Strategy If we can write to 2/3 array elements Therefore
We can solve 2-consensus
Impossible with atomic registers
Therefore
Cannot implement multiple assignment with atomic registers
Art of Multiprocessor Programming
(1)

77. Proof Strategy Take a 3-element array
A writes atomically to slots 0 and 1
B writes atomically to slots 1 and 2
Any thread can scan any set of locations
Art of Multiprocessor Programming
(1)

78. Double Assignment Interface
interface Assign2 {
public void assign(int i1, int v1,
int i2, int v2);
public int read(int i); }
(4)
Art of Multiprocessor Programming
(4)

79. Double Assignment Interface
interface Assign2 {
public void assign(int i1, int v1,
int i2, int v2);
public int read(int i); }
Atomically assign value[i1]= v1
value[i2]= v2
(4)
Art of Multiprocessor Programming
(4)

80. Double Assignment Interface
interface Assign2 {
public void assign(int i1, int v1,
int i2, int v2);
public int read(int i); }
Return i-th value
(4)
Art of Multiprocessor Programming
(4)

81. Initially A B Writes to 0 and 1 Writes to 1 and 2
Art of Multiprocessor Programming

82. Thread A wins if A Thread B didn’t move B
Art of Multiprocessor Programming
(1)

83. Thread A wins if A Thread B moved later B
Art of Multiprocessor Programming
(1)

84. Thread A loses if A Thread B moved earlier B
Art of Multiprocessor Programming
(1)

85. Multi-Consensus Code class MultiConsensus extends …{
Assign2 a = new Assign2(3, EMPTY);
public Object decide(Object value) {
propose(value);
int i = ThreadID.get(), j = 1 – i;
a.assign(i, i, i+1, i);
int other = a.read((i+2) % 3);
if (other==EMPTY||other==a.read(1))
return proposed[i];
else
return proposed[j]; }}
(4)
Art of Multiprocessor Programming
(4)

86. Extends ConsensusProtocol
Multi-Consensus Code
class MultiConsensus extends …{
Assign2 a = new Assign2(3, EMPTY);
public Object decide(object value) {
propose(value);
int i = ThreadID.get(), j = 1 – i;
a.assign(i, i, i+1, i);
int other = a.read((i+2) % 3);
if (other==EMPTY||other==a.read(1))
return proposed[i];
else
return proposed[j]; }}
To save space and make the code clearer we wrote that it extends consensus while it actually extends ConsensusProtocol and omitted that Decide sets j=i-1 and then proposes value. Assume this appears in all the code.
Extends ConsensusProtocol
(4)
Art of Multiprocessor Programming
(4)

87. Three slots initialized to EMPTY
Multi-Consensus Code
class MultiConsensus extends …{
Assign2 a = new Assign2(3, EMPTY);
public Object decide(object value) {
propose(value);
int i = ThreadID.get(), j = 1 – i;
a.assign(i, i, i+1, i);
int other = a.read((i+2) % 3);
if (other==EMPTY||other==a.read(1))
return proposed[i];
else
return proposed[j]; }}
Three slots initialized to EMPTY
(4)
Art of Multiprocessor Programming
(4)

88. Assign id 0 to entries 0,1 (or id 1 to entries 1,2)
Multi-Consensus Code
class MultiConsensus extends …{
Assign2 a = new Assign2(3, EMPTY);
public Object decide(object value) {
propose(value);
int i = ThreadID.get(), j = 1 – i;
a.assign(i, i, i+1, i);
int other = a.read((i+2) % 3);
if (other==EMPTY||other==a.read(1))
return proposed[i];
else
return proposed[j]; }}
Assign id 0 to entries 0,1 (or id 1 to entries 1,2)
(4)
Art of Multiprocessor Programming
(4)

89. Read the register my thread didn’t assign
Multi-Consensus Code
class MultiConsensus extends …{
Assign2 a = new Assign2(3, EMPTY);
public Object decide(object value) {
propose(value);
int i = ThreadID.get(), j = 1 – i;
a.assign(i, i, i+1, i);
int other = a.read((i+2) % 3);
if (other==EMPTY||other==a.read(1))
return proposed[i];
else
return proposed[j]; }}
Read the register my thread didn’t assign
(4)
Art of Multiprocessor Programming
(4)

90. Other thread didn’t move, so I win
Multi-Consensus Code
class MultiConsensus extends …{
Assign2 a = new Assign2(3, EMPTY);
public Object decide(object value) {
a.assign(i, i, i+1, i);
int other = a.read((i+2) % 3);
if (other==EMPTY||other==a.read(1))
return proposed[i];
else
return proposed[j]; }}
Other thread didn’t move, so I win
(4)
Art of Multiprocessor Programming
(4)

91. Other thread moved later so I win
Multi-Consensus Code
class MultiConsensus extends …{
Assign2 a = new Assign2(3, EMPTY);
public Object decide(object value) {
a.assign(i, i, i+1, i);
int other = a.read((i+2) % 3);
if (other==EMPTY||other==a.read(1))
return proposed[i];
else
return proposed[j]; }}
Other thread moved later so I win
(4)
Art of Multiprocessor Programming
(4)

92. Multi-Consensus Code class MultiConsensus extends …{
Assign2 a = new Assign2(3, EMPTY);
public Object decide(object value) {
a.assign(i, i, i+1, i);
int other = a.read((i+2) % 3);
if (other==EMPTY||other==a.read(1))
return proposed[i];
else
return proposed[j]; }}
OK, I win.
(4)
Art of Multiprocessor Programming
(4)

93. Other thread moved first, so I lose
Multi-Consensus Code
class MultiConsensus extends …{
Assign2 a = new Assign2(3, EMPTY);
public Object decide(object value) {
a.assign(i, i, i+1, i);
int other = a.read((i+2) % 3);
if (other==EMPTY||other==a.read(1))
return proposed[i];
else
return proposed[j]; }}
(1)
Other thread moved first, so I lose
(4)
Art of Multiprocessor Programming
(4)

94. Summary
If a thread can assign atomically to 2 out of 3 array locations
Then we can solve 2-consensus
Therefore
No wait-free multi-assignment
From read/write registers
Art of Multiprocessor Programming

95. Read-Modify-Write Objects
Method call
Returns object’s prior value x
Replaces x with mumble(x)
Art of Multiprocessor Programming

96. Read-Modify-Write public abstract class RMWRegister {
private int value;
public int synchronized
getAndMumble() {
int prior = this.value;
this.value = mumble(this.value);
return prior; }
(1)
Art of Multiprocessor Programming

97. Read-Modify-Write Return prior value
public abstract class RMWRegister {
private int value;
public int synchronized
getAndMumble() {
int prior = this.value;
this.value = mumble(this.value);
return prior; }
Return prior value
(1)
Art of Multiprocessor Programming

98. Apply function to current value
Read-Modify-Write
public abstract class RMWRegister {
private int value;
public int synchronized
getAndMumble() {
int prior = this.value;
this.value = mumble(this.value);
return prior; }
Apply function to current value
(1)
Art of Multiprocessor Programming

99. RMW Everywhere! Most synchronization instructions The rest
are RMW methods
The rest
Can be trivially transformed into RMW methods
Art of Multiprocessor Programming

100. Example: Read public abstract class RMWRegister { private int value;
public int synchronized read() {
int prior = this.value;
this.value = this.value;
return prior; }
(1)
Art of Multiprocessor Programming

101. Apply f(v)=v, the identity function
Example: Read
public abstract class RMW {
private int value;
public int synchronized read() {
int prior = this.value;
this.value = this.value;
return prior; }
Apply f(v)=v, the identity function
(1)
Art of Multiprocessor Programming

102. Example: getAndSet public abstract class RMWRegister {
private int value;
public int synchronized
getAndSet(int v) {
int prior = this.value;
this.value = v;
return prior; } …
(1)
Art of Multiprocessor Programming

103. Example: getAndSet (swap)
public abstract class RMWRegister {
private int value;
public int synchronized
getAndSet(int v) {
int prior = this.value;
this.value = v;
return prior; } …
F(x)=v is constant function
(1)
Art of Multiprocessor Programming

104. getAndIncrement public abstract class RMWRegister { private int value;
public int synchronized
getAndIncrement() {
int prior = this.value;
this.value = this.value + 1;
return prior; } …
(1)
Art of Multiprocessor Programming

105. getAndIncrement F(x) = x+1 public abstract class RMWRegister {
private int value;
public int synchronized
getAndIncrement() {
int prior = this.value;
this.value = this.value + 1;
return prior; } …
F(x) = x+1
(1)
Art of Multiprocessor Programming

106. getAndAdd public abstract class RMWRegister { private int value;
public int synchronized
getAndAdd(int a) {
int prior = this.value;
this.value = this.value + a;
return prior; } …
(1)
Art of Multiprocessor Programming

107. Example: getAndAdd F(x) = x+a public abstract class RMWRegister {
private int value;
public int synchronized
getAndIncrement(int a) {
int prior = this.value;
this.value = this.value + a;
return prior; } …
F(x) = x+a
(1)
Art of Multiprocessor Programming

108. compareAndSet public abstract class RMWRegister { private int value;
public boolean synchronized
compareAndSet(int expected,
int update) {
int prior = this.value;
if (this.value==expected) {
this.value = update; return true; }
return false;
} … }
(1)
Art of Multiprocessor Programming

109. If value is what was expected, …
compareAndSet
public abstract class RMWRegister {
private int value;
public boolean synchronized
compareAndSet(int expected,
int update) {
int prior = this.value;
if (this.value==expected) {
this.value = update; return true; }
return false;
} … }
If value is what was expected, …
(1)
Art of Multiprocessor Programming

110. compareAndSet … replace it public abstract class RMWRegister {
private int value;
public boolean synchronized
compareAndSet(int expected,
int update) {
int prior = this.value;
if (this.value==expected) {
this.value = update; return true; }
return false;
} … }
… replace it
(1)
Art of Multiprocessor Programming

111. compareAndSet Report success public abstract class RMWRegister {
private int value;
public boolean synchronized
compareAndSet(int expected,
int update) {
int prior = this.value;
if (this.value==expected) {
this.value = update; return true; }
return false;
} … }
Report success
(1)
Art of Multiprocessor Programming

112. Otherwise report failure
compareAndSet
public abstract class RMWRegister {
private int value;
public boolean synchronized
compareAndSet(int expected,
int update) {
int prior = this.value;
if (this.value==expected) {
this.value = update; return true; }
return false;
} … }
Otherwise report failure
(1)
Art of Multiprocessor Programming

113. Lets characterize F(x)…
Read-Modify-Write
public abstract class RMWRegister {
private int value;
public void synchronized
getAndMumble() {
int prior = this.value;
this.value = mumble(this.value);
return prior; }
Lets characterize F(x)…
(1)
Art of Multiprocessor Programming

114. Definition A RMW method With function mumble(x)
is non-trivial if there exists a value v
Such that v ≠ mumble(v)
Art of Multiprocessor Programming

115. Par Example Identity(x) = x getAndIncrement(x) = x+1 is trivial
is non-trivial
Art of Multiprocessor Programming

116. Theorem Any non-trivial RMW object has consensus number at least 2
No wait-free implementation of RMW registers from atomic registers
Hardware RMW instructions not just a convenience
Art of Multiprocessor Programming

117. Reminder Subclasses of consensus have propose(x) method
which just stores x into proposed[i]
built-in method
decide(object value) method
which determines winning value
customized, class-specific method
Art of Multiprocessor Programming

118. Proof public class RMWConsensus extends ConsensusProtocol {
private RMWRegister r = v;
public Object decide(object value) {
propose(value);
if (r.getAndMumble() == v)
return proposed[i];
else
return proposed[j]; }}
(4)
Art of Multiprocessor Programming

119. Proof Initialized to v public class RMWConsensus
extends ConsensusProtocol {
private RMWRegister r = v;
public Object decide(object value) {
propose(value);
if (r.getAndMumble() == v)
return proposed[i];
else
return proposed[j]; }}
Initialized to v
(4)
Art of Multiprocessor Programming

120. Proof Am I first? public class RMWConsensus extends Consensus {
private RMWRegister r = v;
public Object decide(object value) {
if (r.getAndMumble() == v)
return proposed[i];
else
return proposed[j]; }}
Am I first?
(4)
Art of Multiprocessor Programming

121. Proof Yes, return my input public class RMWConsensus
extends ConsensusProtocol {
private RMWRegister r = v;
public Object decide(object value) {
propose(value);
if (r.getAndMumble() == v)
return proposed[i];
else
return proposed[j]; }}
Yes, return my input
(4)
Art of Multiprocessor Programming

122. No, return other’s input
Proof
public class RMWConsensus
extends ConsensusProtocol {
private RMWRegister r = v;
public Object decide(object value) {
propose(value);
if (r.getAndMumble() == v)
return proposed[i];
else
return proposed[j]; }}
No, return other’s input
(4)
Art of Multiprocessor Programming

123. Proof We have displayed A two-thread consensus protocol
Using any non-trivial RMW object
Art of Multiprocessor Programming

124. Interfering RMW
Let F be a set of functions such that for all fi and fj, either
Commute: fi(fj(v))=fj(fi(v))
Overwrite: fi(fj(v))=fi(v)
Claim: Any set of RMW objects that commutes or overwrites has consensus number exactly 2
Art of Multiprocessor Programming

125. Examples “test-and-set” getAndSet(1) f(v)=1
“swap” getAndSet(x) f(v,x)=x
“fetch-and-inc” getAndIncrement() f(v)=v+1
Overwrite fi(fj(v))=fi(v)
Overwrite fi(fj(v))=fi(v)
Commute fi(fj(v))= fj(fi(v))
Art of Multiprocessor Programming

126. Meanwhile Back at the Critical State
A about to apply fA
B about to apply fB c
0-valent
1-valent
Art of Multiprocessor Programming

127. Maybe the Functions Commute
A applies fA
B applies fB
B applies fB
A applies fA
C runs solo
C runs solo 1
0-valent
1-valent
Art of Multiprocessor Programming

128. Maybe the Functions Commute
A applies fA
B applies fB
These states look the same to C
B applies fB
A applies fA
Contradiction
C runs solo
C runs solo 1
0-valent
1-valent
Art of Multiprocessor Programming

129. Maybe the Functions Overwrite
A applies fA
B applies fB
A applies fA
C runs solo
C runs solo 1
0-valent
1-valent
Art of Multiprocessor Programming

130. Maybe the Functions Overwrite
These states look the same to C c
A applies fA
B applies fB
A applies fA
C runs solo
Contradiction
C runs solo 1
0-valent
1-valent
Art of Multiprocessor Programming

131. Impact Many early machines provided these “weak” RMW instructions
Test-and-set (IBM 360)
Fetch-and-add (NYU Ultracomputer)
Swap (Original SPARCs)
We now understand their limitations
But why do we want consensus anyway?
Art of Multiprocessor Programming

132. compareAndSet public abstract class RMWRegister { private int value;
public boolean synchronized
compareAndSet(int expected,
int update) {
int prior = this.value;
if (this.value==expected) {
this.value = update; return true; }
return false;
} … }
(1)
Art of Multiprocessor Programming

133. replace value if its what we expected, …
compareAndSet
public abstract class RMWRegister {
private int value;
public boolean synchronized
compareAndSet(int expected,
int update) {
int prior = this.value;
if (this.value==expected) {
this.value = update; return true; }
return false;
} … }
replace value if its what we expected, …
(1)
Art of Multiprocessor Programming

134. compareAndSet Has ∞ Consensus Number
public class RMWConsensus
extends ConsensusProtocol {
private AtomicInteger r =
new AtomicInteger(-1);
public Object decide(object value) {
propose(value);
r.compareAndSet(-1,i);
return proposed[r.get()]; }
(4)
Art of Multiprocessor Programming

135. compareAndSet Has ∞ Consensus Number
public class RMWConsensus
extends ConsensusProtocol {
private AtomicInteger r =
new AtomicInteger(-1);
public Object decide(object value) {
propose(value)
r.compareAndSet(-1,i);
return proposed[r.get()]; }
Initialized to -1
(4)
Art of Multiprocessor Programming

136. compareAndSet Has ∞ Consensus Number
public class RMWConsensus
extends ConsensusProtocol {
private AtomicInteger r =
new AtomicInteger(-1);
public Object decide(object value) {
propose(value);
r.compareAndSet(-1,i);
return proposed[r.get()]; }
Try to swap in my id
(4)
Art of Multiprocessor Programming

137. compareAndSet Has ∞ Consensus Number
public class RMWConsensus
extends ConsensusProtocol {
private AtomicInteger r =
new AtomicInteger(-1);
public Object decide(object value) {
propose(value);
r.compareAndSet(-1,i);
return proposed[r.get()]; }
Decide winner’s preference
(4)
Art of Multiprocessor Programming

138. The Consensus Hierarchy
1 Read/Write Registers, Snapshots…
2 getAndSet, getAndIncrement, … .
∞ compareAndSet,…
Art of Multiprocessor Programming

139. Multiple Assignment Atomic k-assignment
Solves consensus for 2k-2 threads
Every even consensus number has an object (can be extended to odd numbers)
Art of Multiprocessor Programming

140. Lock-Freedom
Lock-free: in an infinite execution infinitely often some method call finishes (obviously, in a finite number of steps)
Pragmatic approach
Implies no mutual exclusion
NOTE:
Lock-free is the same as wait-free if the execution is finite.
“Lock-free is to wait-free as deadlock-free is to lockout-free.” In other words, Lock-free and Deadlock-free are “stalinistic”, preferring group progress, while Wait-free and Lockout-free guarantee individual progress.
Any wait-free implementation is lock-free.
Art of Multiprocessor Programming

141. Lock-Free vs. Wait-free
Wait-Free: each method call takes a finite number of steps to finish
Lock-free: infinitely often some method call finishes
NOTE:
Lock-free is the same as wait-free if the execution is finite.
“Lock-free is to wait-free as deadlock-free is to lockout-free.” In other words, Lock-free and Deadlock-free are “stalinistic”, preferring group progress, while Wait-free and Lockout-free guarantee individual progress.
Any wait-free implementation is lock-free.
Art of Multiprocessor Programming

142. Lock-Freedom Any wait-free implementation is lock-free.
Lock-free is the same as wait- free if the execution is finite.
Old saying: “Lock-free is to wait-free as deadlock-free is to lockout-free.”
NOTE:
Lock-free is the same as wait-free if the execution is finite.
“Lock-free is to wait-free as deadlock-free is to lockout-free.” In other words, Lock-free and Deadlock-free are “stalinistic”, preferring group progress, while Wait-free and Lockout-free guarantee individual progress.
Any wait-free implementation is lock-free.
Art of Multiprocessor Programming

143. Lock-Free Implementations
Lock-free consensus is as impossible as wait-free consensus
All the results we presented hold for lock-free algorithms also.
(2)
Art of Multiprocessor Programming

144. There is More: Universality
Consensus is universal
From n-thread consensus
Wait-free/Lock-free
Linearizable
n-threaded
Implementation
Of any sequentially specified object
Stay tuned…
Art of Multiprocessor Programming

145. Art of Multiprocessor Programming
          This work is licensed under a Creative Commons Attribution-ShareAlike 2.5 License.
You are free:
to Share — to copy, distribute and transmit the work
to Remix — to adapt the work
Under the following conditions:
Attribution. You must attribute the work to “The Art of Multiprocessor Programming” (but not in any way that suggests that the authors endorse you or your use of the work).
Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license.
For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to
Any of the above conditions can be waived if you get permission from the copyright holder.
Nothing in this license impairs or restricts the author's moral rights.
Art of Multiprocessor Programming