The Relative Power of Synchronization Methods Nir Shavit Multiprocessor Synchronization Spring 2003

Wait-Free Implementation Every method call completes in finite number of steps Implies no mutual exclusion 4-Feb-19 (2) © 2003 Herlihy and Shavit

Wait-Free Constructions Wait-free atomic registers From safe registers Two-threaded FIFO queue From atomic registers And indirectly from safe registers 4-Feb-19 © 2003 Herlihy and Shavit

Rationale We wanted atomic registers to implement mutual exclusion So we couldn’t use mutual exclusion to implement atomic registers But wait, there’s more! 4-Feb-19 © 2003 Herlihy and Shavit

Unexpected Delay ??? ??? Swapped out back at 4-Feb-19 (2) © 2003 Herlihy and Shavit

Heterogeneous Architectures ??? ??? yawn 286 Pentium Pentium 4-Feb-19 (1) © 2003 Herlihy and Shavit

Fault-Tolerance ??? ??? 4-Feb-19 (2) © 2003 Herlihy and Shavit

Basic Questions Might be a good idea in principle But how do you do it Systematically? Correctly? Efficiently? 4-Feb-19 © 2003 Herlihy and Shavit

Concurrent FIFO Queue 4-Feb-19 © 2003 Herlihy and Shavit

Two-Thread Wait-Free Queue public class WaitFreeQueue { int head = 0, tail = 0; Item[QSIZE] items; public void enq(Item x) { while (tail-head == QSIZE); // busy-wait items[tail++ % QSIZE] = x; } public Item deq() { while (tail == head); // busy-wait return items[head++ % QSIZE]; }} 4-Feb-19 © 2003 Herlihy and Shavit

What About Multiple Dequeuers? 4-Feb-19 © 2003 Herlihy and Shavit

Grand Challenge Only new aspect Implement a FIFO queue Wait-free Linearizable From atomic read-write registers Multiple dequeuers 4-Feb-19 (1) © 2003 Herlihy and Shavit

Consensus While you are ruminating on the grand challenge… We will give you another puzzle Consensus Pretty important … 4-Feb-19 © 2003 Herlihy and Shavit

Consensus: Each Thread has a Private Input 19 32 21 4-Feb-19 © 2003 Herlihy and Shavit

They Communicate 4-Feb-19 © 2003 Herlihy and Shavit

They Agree on Some Thread’s Input 19 19 19 4-Feb-19 © 2003 Herlihy and Shavit

Formally: Consensus Consistent: all threads decide the same value Valid: the common decision value is some thread's input Wait-free: each thread decides after a finite number of steps 4-Feb-19 © 2003 Herlihy and Shavit

Theorem: No Consensus from Registers if any Thread can Halt ??? ??? 4-Feb-19 © 2003 Herlihy and Shavit

Proof Strategy Assume otherwise Reason about the properties of any such protocol Derive a contradiction Quod Erat Demonstrandum 4-Feb-19 © 2003 Herlihy and Shavit

Wait-Free Computation A moves B moves Either A or B “moves” Moving means Register read Register write 4-Feb-19 © 2003 Herlihy and Shavit

The Two-Move Tree Final states Initial state 4-Feb-19 (2) © 2003 Herlihy and Shavit

Decision Values 1 1 1 1 4-Feb-19 © 2003 Herlihy and Shavit

Bivalent: Both Possible 1 1 1 1 4-Feb-19 © 2003 Herlihy and Shavit

Univalent: Single Value Possible 1 1 1 1 4-Feb-19 © 2003 Herlihy and Shavit

1-valent: Only 1 Possible 1 1 1 4-Feb-19 © 2003 Herlihy and Shavit

0-valent: Only 0 possible 1 1 1 1 4-Feb-19 © 2003 Herlihy and Shavit

Summary Wait-free computation is a tree Bivalent system states Outcome not fixed Univalent states Outcome is fixed May not be “known” yet 1-Valent and 0-Valent states 4-Feb-19 © 2003 Herlihy and Shavit

Claim Some initial system state is bivalent If A starts with input 0 and B starts with input 1, then initially A should be able to decide 0 and B should be able to decide 1. Lets see why? 4-Feb-19 © 2003 Herlihy and Shavit

A 0-Valent Initial State All executions lead to decision of 0 4-Feb-19 (2) © 2003 Herlihy and Shavit (1)

A 0-Valent Initial State Solo execution by A also decides 0 4-Feb-19 (1) © 2003 Herlihy and Shavit (1)

A 1-Valent Initial State All executions lead to decision of 1 4-Feb-19 (2) © 2003 Herlihy and Shavit (1)

A 1-Valent Initial State Solo execution by B also decides 1 4-Feb-19 (1) © 2003 Herlihy and Shavit (1)

A Univalent Initial State? 1 Can all executions lead to the same decision? 4-Feb-19 (2) © 2003 Herlihy and Shavit

State is Bivalent 1 Solo execution by A must decide 0 1 Solo execution by A must decide 0 Solo execution by B must decide 1 4-Feb-19 © 2003 Herlihy and Shavit

Critical States critical 0-valent 1-valent 4-Feb-19 (3) © 2003 Herlihy and Shavit (3)

Critical States Starting from a bivalent initial state The protocol can reach a critical state Otherwise we could stay bivalent forever And the protocol is not wait-free 4-Feb-19 © 2003 Herlihy and Shavit

From a Critical State c 0-valent 1-valent If A goes first, protocol decides 0 If B goes first, protocol decides 1 4-Feb-19 © 2003 Herlihy and Shavit

Model Dependency So far, memory-independent! True for Registers Message-passing Carrier pigeons Any kind of asynchronous computation 4-Feb-19 © 2003 Herlihy and Shavit

What are the Threads Doing? Reads and/or writes To same/different registers 4-Feb-19 © 2003 Herlihy and Shavit

Possible Interactions A reads x x.read() y.read() x.write() y.write() ? A reads y 4-Feb-19 © 2003 Herlihy and Shavit

States look the same to A Reading Registers A runs solo, decides 0 c B reads x A runs solo, decides 1 1 Contradiction States look the same to A 4-Feb-19 © 2003 Herlihy and Shavit

Possible Interactions x.read() y.read() x.write() y.write() no ? 4-Feb-19 © 2003 Herlihy and Shavit

Writing Distinct Registers A writes y B writes x A writes y B writes x Contradiction 1 The song remains the same 4-Feb-19 © 2003 Herlihy and Shavit

Possible Interactions x.read() y.read() x.write() y.write() no ? 4-Feb-19 © 2003 Herlihy and Shavit

Writing Same Registers c B writes x A writes x A runs solo, decides 0 A writes x Contradiction A runs solo, decides 1 1 States look the same to A 4-Feb-19 © 2003 Herlihy and Shavit

That’s All, Folks! no x.read() y.read() x.write() y.write() 4-Feb-19 © 2003 Herlihy and Shavit

Theorem It is impossible to solve consensus using read/write atomic registers Assume protocol exists It has a bivalent initial state Must be able to reach a critical state Case analysis of interactions Reads vs others Writes vs writes 4-Feb-19 © 2003 Herlihy and Shavit

What Does Consensus have to do with Concurrent Objects? 4-Feb-19 © 2003 Herlihy and Shavit

Always announce intentions Figure out which one won The Consensus Object public abstract class Consensus implements Consensus { private Object[] announce; public ConsensusImpl(int n) { announce = new Object[n]; } public void propose(Object value) { announce[Thread.myIndex()] = value; abstract public Object decide(); }} Always announce intentions Figure out which one won 4-Feb-19 (4) © 2003 Herlihy and Shavit (4)

Can FIFO Queue Implement A Consensus Object? proposed array FIFO Queue with red and black balls 8 4-Feb-19 © 2003 Herlihy and Shavit

FIFO Queue with red and black balls A Consensus Protocol Preferred array FIFO Queue with red and black balls 8 Coveted red ball Dreaded black ball 4-Feb-19 © 2003 Herlihy and Shavit

Protocol: Write Value to Array 1 4-Feb-19 © 2003 Herlihy and Shavit

Protocol: Take Next Item from Queue 8 1 4-Feb-19 © 2003 Herlihy and Shavit

Protocol: Take Next Item from Queue I got the dreaded black ball, so I will decide the other’s value from the array I got the coveted red ball, so I will decide my value 1 8 4-Feb-19 © 2003 Herlihy and Shavit

Consensus Using FIFO Queue public class QueueConsensus extends Consensus { Queue queue; public QueueConsensus() { queue = new Queue(); queue.enq(“red ball"); queue.enq(“black ball");} public Object decide() { String status = queue.deq(); int i = Thread.myIndex(); if (status == “red ball") return proposed[i]; else return proposed[1-i];} Initialize Queue i won i lost 4-Feb-19 © 2003 Herlihy and Shavit

Why does this Work? If one thread gets the red ball Then the other gets the black ball Winner can take her own value Loser can find winner’s value in array Because threads write array Before dequeueing from queue 4-Feb-19 © 2003 Herlihy and Shavit

Implication We can solve 2-thread consensus using only A two-dequeuer queue, and A couple of atomic registers 4-Feb-19 © 2003 Herlihy and Shavit

Implications contradiction Given Assume there exists A consensus protocol from queue and registers Assume there exists A queue implementation from atomic registers Substitution yields: A wait-free consensus protocol from atomic registers contradiction 4-Feb-19 (1) © 2003 Herlihy and Shavit (1)

Corollary It is impossible to implement a two-dequeuer wait-free FIFO queue from read/write memory. 4-Feb-19 © 2003 Herlihy and Shavit

Consensus Numbers An object has consensus number n If it can be used Together with atomic read/write registers To implement n-thread consensus But not (n+1)-thread consensus 4-Feb-19 © 2003 Herlihy and Shavit

Consensus Numbers Theorem Atomic read/write registers have consensus number 1 4-Feb-19 © 2003 Herlihy and Shavit

Consensus Numbers Consensus numbers are a useful way of measuring synchronization power Theorem If you can implement X from Y And X has consensus number c Then Y has consensus number at least c 4-Feb-19 © 2003 Herlihy and Shavit

Synchronization Speed Limit Conversely If X has consensus number c And Y has consensus number d < c Then there is no way to construct a wait-free implementation of X by Y This theorem will be very useful Unforeseen practical implications! 4-Feb-19 © 2003 Herlihy and Shavit

Earlier Grand Challenge Snapshot means Write any array element Read multiple array elements atomically What about the dual problem: Write multiple array elements atomically Scan any array elements Call this problem multiple assignment 4-Feb-19 © 2003 Herlihy and Shavit

Multiple Assignment Theorem No protocol Using only read/write registers Implements multiple assignment Weird or what? Single write/multiple read OK Multi write/multiple read impossible In fact: Multi write/single read also impossible!!! 4-Feb-19 (1) © 2003 Herlihy and Shavit (1)

Proof Strategy Given a 3-element array Then we can solve 2-consensus A writes atomically to slots 0 and 1 B writes atomically to slots 1 and 2 Any thread can scan any set of locations Then we can solve 2-consensus So multiple assignment has consensus number at least 2 No read/write register implementation 4-Feb-19 © 2003 Herlihy and Shavit (1)

Initially A B Writes to 0 and 1 Writes to 1 and 2 4-Feb-19 © 2003 Herlihy and Shavit

Thread A wins if it sees A B 4-Feb-19 © 2003 Herlihy and Shavit (1)

Thread A wins if it sees A B 4-Feb-19 (1) © 2003 Herlihy and Shavit

Thread A loses if it sees B 4-Feb-19 (1) © 2003 Herlihy and Shavit

Figure out which one won Multi-Consensus class MultiConsensus { MultiAssign multi; public Object decide() { int i = Thread.myIndex(); // my index int j = 1 - i; // other index multi.assign(i, input, i+1, input); Object[] result = multi.scan(); if (result[(j + 1) % 3] == null || (result[j] == result[(j + 1) % 3])) return proposed[i]; else return proposed[j]; } Write/Scan locations Figure out which one won 4-Feb-19 (4) © 2003 Herlihy and Shavit (4)

Summary If a thread can assign atomically to 2 out of 3 array locations Then we can solve 2-consensus Implying No wait-free implementation From read/write registers 4-Feb-19 © 2003 Herlihy and Shavit

Read-Modify-Write Method takes 2 arguments: Method call: Variable x Function f Method call: Returns value of x Replaces x with f(x) 4-Feb-19 © 2003 Herlihy and Shavit

Read-Modify-Write Return prior value Apply function f() public abstract class RMW { private int value; public void synchronized rmw(function f) { int prior = this.value; this.value = f(this.value); return prior; } Return prior value Apply function f() 4-Feb-19 (1) © 2003 Herlihy and Shavit

f(v)=v is the identity function Example: Read public abstract class RMW { private int value; public void synchronized read() { int prior = this.value; this.value = this.value; return prior; } f(v)=v is the identity function 4-Feb-19 (1) © 2003 Herlihy and Shavit

F(v)=1 is constant function Example: test&set public abstract class RMW { private int value; public synchronized void TAS() { int prior = this.value; this.value = 1; return prior; } F(v)=1 is constant function 4-Feb-19 (1) © 2003 Herlihy and Shavit

f(v)=v+1 is increment function Example: fetch&inc public abstract class RMW { private int value; public synchronized void FAI() { int prior = this.value; this.value = this.value+1; return prior; } f(v)=v+1 is increment function 4-Feb-19 (1) © 2003 Herlihy and Shavit

F(v,x)=v+x is the addition function Example: fetch&add public abstract class RMW { private int value; public synchronized void FAA(int x) { int prior = this.value; this.value = this.value+x; return prior; } F(v,x)=v+x is the addition function 4-Feb-19 (1) © 2003 Herlihy and Shavit

F(v,x) = x is a constant function Example: swap public abstract class RMW { private int value; public synchronized void swap(int x) { int prior = this.value; this.value = x; return prior; } F(v,x) = x is a constant function 4-Feb-19 (1) © 2003 Herlihy and Shavit

Example: compare&swap public abstract class RMW { private int value; public synchronized void CAS(int old, int new) { int prior = this.value; if (this.value == old) this.value = new; return prior; } f(old,new) is a complex function 4-Feb-19 (1) © 2003 Herlihy and Shavit

Definition A RMW is non-trivial if there exists a value v Such that v ≠ f(v) read is trivial FAA is not 4-Feb-19 © 2003 Herlihy and Shavit

Theorem Any non-trivial RMW object has consensus number at least 2 Implies no wait-free implementation of RMW registers from read/write registers Hardware RMW instructions not just a convenience 4-Feb-19 © 2003 Herlihy and Shavit

Reminder All objects that inherit from Consensus have propose method which just stores input into this.announce[i] 4-Feb-19 © 2003 Herlihy and Shavit

No, return other’s input Proof Initialized to v public class RMWConsensus implements Consensus { private RMW r; public Object decide() { int i = Thread.myIndex(); if (r.rmw(f) == v) return this.announce[i]; else return this.announce[1-i]; }} Am I first? Yes, return my input No, return other’s input 4-Feb-19 (4) © 2003 Herlihy and Shavit

Proof We have displayed A two-thread consensus protocol Using any non-trivial RMW object 4-Feb-19 © 2003 Herlihy and Shavit

Interfering RMW Let F be a set of functions such that for all fi and fj, either They commute: fi(fj(v))=fj(fi(v)) They overwrite: fi(fj(v))=fi(v) Claim: Any such set of RMW objects has consensus number exactly 2 4-Feb-19 © 2003 Herlihy and Shavit

Examples Test-and-Set f(v)=1 Swap f(v,x)=x Fetch-and-inc f(v)=v+1 Overwrite fi(fj(v))=fi(v) Overwrite fi(fj(v))=fi(v) Commute fi(fj(v))= fj(fi(v)) 4-Feb-19 © 2003 Herlihy and Shavit

Meanwhile Back at the Critical State A about to apply fA B about to apply fB c 0-valent 1-valent 4-Feb-19 © 2003 Herlihy and Shavit

Maybe the Functions Commute A applies fA B applies fB B applies fB A applies fA C runs solo C runs solo 1 0-valent 1-valent 4-Feb-19 © 2003 Herlihy and Shavit

Maybe the Functions Commute A applies fA B applies fB These states look the same to C B applies fB A applies fA C runs solo C runs solo 1 0-valent 1-valent 4-Feb-19 © 2003 Herlihy and Shavit

Maybe the Functions Overwrite A applies fA B applies fB A applies fA C runs solo C runs solo 1 0-valent 1-valent 4-Feb-19 © 2003 Herlihy and Shavit

Maybe the Functions Overwrite These states look the same to C c A applies fA B applies fB A applies fA C runs solo C runs solo 1 0-valent 1-valent 4-Feb-19 © 2003 Herlihy and Shavit

Impact Many early machines provided these “weak” RMW instructions Test-and-set (IBM 360) Fetch-and-add (NYU Ultracomputer) Swap (Original SPARCs) We now understand their limitations But why do we want consensus anyway? 4-Feb-19 © 2003 Herlihy and Shavit

Example: compare&swap public abstract class RMW { private int value; public void CAS(int old, int new) { int prior = this.value; if (this.value == old) this.value = new; return prior; } 4-Feb-19 (1) © 2003 Herlihy and Shavit

CAS Has Unbounded Consensus Number Initialized to -1 public class RMWConsensus implements Consensus { private RMW r; public Object decide() { int i = Thread.myIndex(); int j = r.CAS(-1,i); if (j == -1) return this.announce[i]; else return this.announce[j]; }} Am I first? Yes, return my input No, return other’s input 4-Feb-19 (4) © 2003 Herlihy and Shavit

The Consensus Hierarchy 1 Read/Write Registers, Snapshots… 2 T&S, F&I, Swap… . ∞ CAS,… 4-Feb-19 © 2003 Herlihy and Shavit

Multiple Assignment Atomic k-assignment Solves consensus for 2k-2 threads Every even consensus number has an object (can be extended to odd numbers) 4-Feb-19 © 2003 Herlihy and Shavit

Lock-Free Implementations Infinitely often some method call completes in a finite number of steps Stalinistic real world systems approach Implies no mutual exclusion 4-Feb-19 (2) © 2003 Herlihy and Shavit

Lock-Free Implementations Lock-free = Wait-free for finite executions Lock-free consensus is just as impossible All the results we presented hold for lock-free algorithms also. 4-Feb-19 (2) © 2003 Herlihy and Shavit

There is More: Universality Consensus is universal From n-thread consensus Wait-free/Lock-free Linearizable n-threaded Implementation Of any sequential object 4-Feb-19 © 2003 Herlihy and Shavit