Concurrent Queues and Stacks Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

Art of Multiprocessor Programming2 2 The Five-Fold Path Coarse-grained locking Fine-grained locking Optimistic synchronization Lazy synchronization Lock-free synchronization

Art of Multiprocessor Programming3 3 Another Fundamental Problem We told you about –Sets implemented by linked lists –Hash Tables Next: queues Next: stacks

Art of Multiprocessor Programming4 4 Queues & Stacks pool of items

Art of Multiprocessor Programming5 5 Queues deq()/ enq( ) Total order First in First out

Art of Multiprocessor Programming6 6 Stacks pop()/ push( ) Total order Last in First out

Art of Multiprocessor Programming7 7 Bounded Fixed capacity Good when resources an issue

Art of Multiprocessor Programming8 8 Unbounded Unlimited capacity Often more convenient …

Art of Multiprocessor Programming9 Blocking zzz … Block on attempt to remove from empty stack or queue

Art of Multiprocessor Programming10 Blocking zzz … Block on attempt to add to full bounded stack or queue

Art of Multiprocessor Programming11 Non-Blocking Ouch! Throw exception on attempt to remove from empty stack or queue

Art of Multiprocessor Programming12 This Lecture Queue –Bounded, blocking, lock-based –Unbounded, non-blocking, lock-free Stack –Unbounded, non-blocking lock-free –Elimination-backoff algorithm

Art of Multiprocessor Programming13 Queue: Concurrency enq(x) y=deq() enq() and deq() work at different ends of the object tailhead

Art of Multiprocessor Programming14 Concurrency enq(x) Challenge: what if the queue is empty or full? y=deq() tail head

Art of Multiprocessor Programming15 Bounded Queue Sentinel head tail

Art of Multiprocessor Programming16 Bounded Queue head tail First actual item

Art of Multiprocessor Programming17 Bounded Queue head tail Lock out other deq() calls deqLock

Art of Multiprocessor Programming18 Bounded Queue head tail Lock out other enq() calls deqLock enqLock

Art of Multiprocessor Programming 1919 Not Done Yet head tail deqLock enqLock Need to tell whether queue is full or empty

Art of Multiprocessor Programming20 Not Done Yet head tail deqLock enqLock Max size is 8 items size 1

Art of Multiprocessor Programming21 Not Done Yet head tail deqLock enqLock Incremented by enq() Decremented by deq() size 1

Art of Multiprocessor Programming22 Enqueuer 1 Lock enqLock head tail deqLock enqLock size

Art of Multiprocessor Programming23 Enqueuer 1 Read size OK head tail deqLock enqLock size

Art of Multiprocessor Programming24 Enqueuer 1 No need to lock tail head tail deqLock enqLock size

Art of Multiprocessor Programming25 Enqueuer 1 Enqueue Node head tail deqLock enqLock size

Art of Multiprocessor Programming26 Enqueuer size 1 2 getAndincrement() head tail deqLock enqLock

Art of Multiprocessor Programming27 Enqueuer 8 Release lock 2 head tail deqLock enqLock size

Art of Multiprocessor Programming28 Enqueuer 2 If queue was empty, notify waiting dequeuers head tail deqLock enqLock size

Art of Multiprocessor Programming29 Unsuccesful Enqueuer 8 Uh-oh Read size … head tail deqLock enqLock size

Art of Multiprocessor Programming30 Dequeuer 2 Lock deqLock head tail deqLock enqLock size

Art of Multiprocessor Programming31 Dequeuer 2 Read sentinel’s next field OK head tail deqLock enqLock size

Art of Multiprocessor Programming32 Dequeuer siz 2 Read value head tail deqLock enqLock size

Art of Multiprocessor Programming33 Dequeuer 2 Make first Node new sentinel head tail deqLock enqLock size

Art of Multiprocessor Programming34 Dequeuer 1 Decrement size head tail deqLock enqLock size

Art of Multiprocessor Programming35 Dequeuer 1 Release deqLock head tail deqLock enqLock size

Art of Multiprocessor Programming36 Unsuccesful Dequeuer 0 Read sentinel’s next field uh-oh head tail deqLock enqLock size

Art of Multiprocessor Programming37 Bounded Queue public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger size; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); }

Art of Multiprocessor Programming38 Bounded Queue public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger size; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } enq & deq locks

Art of Multiprocessor Programming39 Digression: Monitor Locks Java synchronized objects and ReentrantLocks are monitors Allow blocking on a condition rather than spinning Threads: –acquire and release lock –wait on a condition

Art of Multiprocessor Programming40 public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } The Java Lock Interface Acquire lock

Art of Multiprocessor Programming41 public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } The Java Lock Interface Release lock

Art of Multiprocessor Programming42 public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } The Java Lock Interface Try for lock, but not too hard

Art of Multiprocessor Programming43 public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } The Java Lock Interface Create condition to wait on

Art of Multiprocessor Programming44 The Java Lock Interface public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } Never mind what this method does

Art of Multiprocessor Programming45 Lock Conditions public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); }

Art of Multiprocessor Programming46 public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); } Lock Conditions Release lock and wait on condition

Art of Multiprocessor Programming47 public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); } Lock Conditions Wake up one waiting thread

Art of Multiprocessor Programming48 public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); } Lock Conditions Wake up all waiting threads

Art of Multiprocessor Programming49 Await Releases lock associated with q Sleeps (gives up processor) Awakens (resumes running) Reacquires lock & returns q.await()

Art of Multiprocessor Programming50 Signal Awakens one waiting thread –Which will reacquire lock q.signal();

Art of Multiprocessor Programming51 Signal All Awakens all waiting threads –Which will each reacquire lock q.signalAll();

Art of Multiprocessor Programming52 A Monitor Lock Critical Section waiting room lock() unlock()

Art of Multiprocessor Programming53 Unsuccessful Deq Critical Section waiting room lock() await() deq() Oh no, empty!

Art of Multiprocessor Programming54 Another One Critical Section waiting room lock() await() deq() Oh no, empty!

Art of Multiprocessor Programming55 Enqueuer to the Rescue Critical Section waiting room lock() signalAll() enq( ) unlock() Yawn!

Art of Multiprocessor Programming56 Yawn! Monitor Signalling Critical Section waiting room Yawn! Awakened thread might still lose lock to outside contender…

Art of Multiprocessor Programming57 Dequeuers Signalled Critical Section waiting room Found it Yawn!

Art of Multiprocessor Programming58 Yawn! Dequeuers Signaled Critical Section waiting room Still empty!

Art of Multiprocessor Programming59 Dollar Short + Day Late Critical Section waiting room

Art of Multiprocessor Programming60 public class Queue { int head = 0, tail = 0; T[QSIZE] items; public synchronized T deq() { while (tail – head == 0) wait(); T result = items[head % QSIZE]; head++; notifyAll(); return result; } … }} Java Synchronized Methods

Art of Multiprocessor Programming61 public class Queue { int head = 0, tail = 0; T[QSIZE] items; public synchronized T deq() { while (tail – head == 0) wait(); T result = items[head % QSIZE]; head++; notifyAll(); return result; } … }} Java Synchronized Methods Each object has an implicit lock with an implicit condition

Art of Multiprocessor Programming62 public class Queue { int head = 0, tail = 0; T[QSIZE] items; public synchronized T deq() { while (tail – head == 0) wait(); T result = items[head % QSIZE]; head++; notifyAll(); return result; } … }} Java Synchronized Methods Lock on entry, unlock on return

Art of Multiprocessor Programming63 public class Queue { int head = 0, tail = 0; T[QSIZE] items; public synchronized T deq() { while (tail – head == 0) wait(); T result = items[head % QSIZE]; head++; this.notifyAll(); return result; } … }} Java Synchronized Methods Wait on implicit condition

Art of Multiprocessor Programming64 public class Queue { int head = 0, tail = 0; T[QSIZE] items; public synchronized T deq() { while (tail – head == 0) this.wait(); T result = items[head % QSIZE]; head++; notifyAll(); return result; } … }} Java Synchronized Methods Signal all threads waiting on condition

Art of Multiprocessor Programming65 (Pop!) The Bounded Queue public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger size; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); }

Art of Multiprocessor Programming66 Bounded Queue Fields public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger size; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } Enq & deq locks

Art of Multiprocessor Programming67 Bounded Queue Fields public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger size; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } Enq lock’s associated condition

Art of Multiprocessor Programming68 Bounded Queue Fields public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger size; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } size: 0 to capacity

Art of Multiprocessor Programming69 Bounded Queue Fields public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger size; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } Head and Tail

Art of Multiprocessor Programming70 Enq Method Part One public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (size.get() == Capacity) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = tail.next; if (size.getAndIncrement() == 0) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … }

Art of Multiprocessor Programming71 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (size.get() == capacity) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = tail.next; if (size.getAndIncrement() == 0) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One Lock and unlock enq lock

Art of Multiprocessor Programming72 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (size.get() == capacity) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = tail.next; if (size.getAndIncrement() == 0) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One Wait while queue is full …

Art of Multiprocessor Programming73 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (size.get() == capacity) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = tail.next; if (size.getAndIncrement() == 0) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One when await() returns, you might still fail the test !

Art of Multiprocessor Programming74 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (size.get() == capacity) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = tail.next; if (size.getAndIncrement() == 0) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Be Afraid After the loop: how do we know the queue won’t become full again?

Art of Multiprocessor Programming75 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (size.get() == capacity) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = tail.next; if (size.getAndIncrement() == 0) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One Add new node

Art of Multiprocessor Programming76 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (size.get() == capacity) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = tail.next; if (size.getAndIncrement() == 0) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One If queue was empty, wake frustrated dequeuers

Art of Multiprocessor Programming77 Beware Lost Wake-Ups Critical Section waiting room lock() Queue empty so signal () enq( ) unlock() Yawn!

Art of Multiprocessor Programming78 Lost Wake-Up Critical Section waiting room lock() enq( ) unlock() Yawn! Queue not empty so no need to signal

Art of Multiprocessor Programming79 Lost Wake-Up Critical Section waiting room Yawn!

Art of Multiprocessor Programming80 Lost Wake-Up Critical Section waiting room Found it

Art of Multiprocessor Programming81 What’s Wrong Here? Critical Section waiting room Still waiting ….!

Art of Multiprocessor Programming82 Solution to Lost Wakeup Always use –signalAll() and notifyAll() Not –signal() and notify() 82

Art of Multiprocessor Programming83 Enq Method Part Deux public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); }

Art of Multiprocessor Programming84 Enq Method Part Deux public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); } Are there dequeuers to be signaled?

Art of Multiprocessor Programming85 public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); } Enq Method Part Deux Lock and unlock deq lock

Art of Multiprocessor Programming86 public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); } Enq Method Part Deux Signal dequeuers that queue is no longer empty

Art of Multiprocessor Programming87 The enq() & deq() Methods Share no locks –That’s good But do share an atomic counter –Accessed on every method call –That’s not so good Can we alleviate this bottleneck?

Art of Multiprocessor Programming88 Split the Counter The enq() method –Increments only –Cares only if value is capacity The deq() method –Decrements only –Cares only if value is zero

Art of Multiprocessor Programming89 Split Counter Enqueuer increments enqSize Dequeuer increments deqSize When enqueuer hits capacity –Locks deqLock –Sets size = enqSize - DeqSize Intermittent synchronization –Not with each method call –Need both locks! (careful …)

Art of Multiprocessor Programming90 A Lock-Free Queue Sentinel head tail

Art of Multiprocessor Programming91 Compare and Set CAS

Art of Multiprocessor Programming92 Enqueue head tail enq( )

Art of Multiprocessor Programming93 Enqueue head tail

Art of Multiprocessor Programming94 Logical Enqueue head tail CAS

Art of Multiprocessor Programming95 Physical Enqueue head tail CAS

Art of Multiprocessor Programming96 Enqueue These two steps are not atomic The tail field refers to either –Actual last Node (good) –Penultimate Node (not so good) Be prepared!

Art of Multiprocessor Programming97 Enqueue What do you do if you find –A trailing tail? Stop and help fix it –If tail node has non-null next field –CAS the queue’s tail field to tail.next As in the universal construction

Art of Multiprocessor Programming98 When CASs Fail During logical enqueue –Abandon hope, restart –Still lock-free (why?) During physical enqueue –Ignore it (why?)

Art of Multiprocessor Programming99 Dequeuer head tail Read value

Art of Multiprocessor Programming100 Dequeuer head tail Make first Node new sentinel CAS

Art of Multiprocessor Programming101 Memory Reuse? What do we do with nodes after we dequeue them? Java: let garbage collector deal? Suppose there is no GC, or we prefer not to use it?

Art of Multiprocessor Programming102 Dequeuer head tail CAS Can recycle

Art of Multiprocessor Programming103 Simple Solution Each thread has a free list of unused queue nodes Allocate node: pop from list Free node: push onto list Deal with underflow somehow …

Art of Multiprocessor Programming104 Why Recycling is Hard Free pool headtail Want to redirect head from gray to red zzz…

Art of Multiprocessor Programming105 Both Nodes Reclaimed Free pool zzz headtail

Art of Multiprocessor Programming106 One Node Recycled Free pool Yawn! headtail

Art of Multiprocessor Programming107 Why Recycling is Hard Free pool CAS headtail OK, here I go!

Art of Multiprocessor Programming108 Recycle FAIL Free pool zOMG what went wrong? headtail

Art of Multiprocessor Programming109 The Dreaded ABA Problem headtail Head reference has value A Thread reads value A Head reference has value A Thread reads value A

Art of Multiprocessor Programming110 Dreaded ABA continued zzz headtail Head reference has value B Node A freed Head reference has value B Node A freed

Art of Multiprocessor Programming111 Dreaded ABA continued Yawn! headtail Head reference has value A again Node A recycled and reinitialized Head reference has value A again Node A recycled and reinitialized

Art of Multiprocessor Programming112 Dreaded ABA continued CAS headtail CAS succeeds because references match, even though reference’s meaning has changed CAS succeeds because references match, even though reference’s meaning has changed

Art of Multiprocessor Programming113 The Dreaded ABA FAIL Is a result of CAS() semantics –Oracle, Intel, AMD, … Not with Load-Locked/Store-Conditional –IBM …

Art of Multiprocessor Programming114 Dreaded ABA – A Solution Tag each pointer with a counter Unique over lifetime of node Pointer size vs word size issues Overflow? –Don’t worry be happy? –Bounded tags? AtomicStampedReference class

Art of Multiprocessor Programming115 Atomic Stamped Reference AtomicStampedReference class –Java.util.concurrent.atomic package address S Stamp Reference Can get reference & stamp atomically

Art of Multiprocessor Programming116 Concurrent Stack Methods –push(x) –pop() Last-in, First-out (LIFO) order Lock-Free!

Art of Multiprocessor Programming117 Empty Stack Top

Art of Multiprocessor Programming118 Push Top

Art of Multiprocessor Programming119 Push Top CAS

Art of Multiprocessor Programming120 Push Top

Art of Multiprocessor Programming121 Push Top

Art of Multiprocessor Programming122 Push Top

Art of Multiprocessor Programming123 Push Top CAS

Art of Multiprocessor Programming124 Push Top

Art of Multiprocessor Programming125 Pop Top

Art of Multiprocessor Programming126 Pop Top CAS

Art of Multiprocessor Programming127 Pop Top CAS mine!

Art of Multiprocessor Programming128 Pop Top CAS

Art of Multiprocessor Programming129 Pop Top

Art of Multiprocessor Programming130 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff(); }} Lock-free Stack

Art of Multiprocessor Programming131 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public Boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack tryPush attempts to push a node

Art of Multiprocessor Programming132 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack Read top value

Art of Multiprocessor Programming133 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack current top will be new node’s successor

Art of Multiprocessor Programming134 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack Try to swing top, return success or failure

Art of Multiprocessor Programming135 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack Push calls tryPush

Art of Multiprocessor Programming136 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack Create new node

Art of Multiprocessor Programming137 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack If tryPush() fails, back off before retrying

Art of Multiprocessor Programming138 Lock-free Stack Good –No locking Bad –Without GC, fear ABA –Without backoff, huge contention at top –In any case, no parallelism

Art of Multiprocessor Programming139 Big Question Are stacks inherently sequential? Reasons why –Every pop() call fights for top item Reasons why not –Stay tuned …

Art of Multiprocessor Programming140 Elimination-Backoff Stack How to –“turn contention into parallelism” Replace familiar –exponential backoff With alternative –elimination-backoff

Art of Multiprocessor Programming141 Observation Push( ) Pop() linearizable stack After an equal number of pushes and pops, stack stays the same After an equal number of pushes and pops, stack stays the same Yes!

Art of Multiprocessor Programming142 Idea: Elimination Array Push( ) Pop() stack Pick at random Pick at random Elimination Array

Art of Multiprocessor Programming143 Push Collides With Pop Push( ) Pop() stack continue No need to access stack No need to access stack Yes!

Art of Multiprocessor Programming144 No Collision Push( ) Pop() stack If no collision, access stack If no collision, access stack If pushes collide or pops collide access stack

Art of Multiprocessor Programming145 Elimination-Backoff Stack Lock-free stack + elimination array Access Lock-free stack, –If uncontended, apply operation –if contended, back off to elimination array and attempt elimination

Art of Multiprocessor Programming146 Elimination-Backoff Stack Push( ) Pop() Top CAS If CAS fails, back off

Art of Multiprocessor Programming147 Dynamic Range and Delay Push( ) Pick random range and max waiting time based on level of contention encountered

50-50, Random Slots

Asymmetric Rendevous pop 1 () pop 2 () Push( ) Pops find first vacant slot and spin. Pushes hunt for pops.

Asymmetric vs. Symmetric

Effect of Backoff and Slot Choice

Effect of Slot Choice Random choice Sequential choice Darker shades mean more exchanges

Effect of Slot Choice

Art of Multiprocessor Programming155 Linearizability Un-eliminated calls –linearized as before Eliminated calls: –linearize pop() immediately after matching push() Combination is a linearizable stack

Art of Multiprocessor Programming156 Un-Eliminated Linearizability push(v 1 ) time linearizable push(v 1 ) pop(v 1 )

Art of Multiprocessor Programming157 Eliminated Linearizability pop(v 2 )push(v 1 ) push(v 2 ) time push(v 2 ) pop(v 2 ) push(v 1 ) pop(v 1 ) Collision Point Red calls are eliminated pop(v 1 ) linearizable

Art of Multiprocessor Programming158 Backoff Has Dual Effect Elimination introduces parallelism Backoff to array cuts contention on lock- free stack Elimination in array cuts down number of threads accessing lock-free stack

Art of Multiprocessor Programming159 public class EliminationArray { private static final int duration =...; private static final int timeUnit =...; Exchanger [] exchanger; public EliminationArray(int capacity) { exchanger = new Exchanger[capacity]; for (int i = 0; i < capacity; i++) exchanger[i] = new Exchanger (); … } … } Elimination Array

Art of Multiprocessor Programming160 public class EliminationArray { private static final int duration =...; private static final int timeUnit =...; Exchanger [] exchanger; public EliminationArray(int capacity) { exchanger = new Exchanger[capacity]; for (int i = 0; i < capacity; i++) exchanger[i] = new Exchanger (); … } … } Elimination Array An array of Exchangers

Art of Multiprocessor Programming161 public class Exchanger { AtomicStampedReference slot = new AtomicStampedReference (null, 0); Digression: A Lock-Free Exchanger

Art of Multiprocessor Programming162 public class Exchanger { AtomicStampedReference slot = new AtomicStampedReference (null, 0); A Lock-Free Exchanger Atomically modifiable reference + status

Art of Multiprocessor Programming163 Atomic Stamped Reference AtomicStampedReference class –Java.util.concurrent.atomic package In C or C++: address S reference stamp

Art of Multiprocessor Programming164 Extracting Reference & Stamp public T get(int[] stampHolder);

Art of Multiprocessor Programming165 Extracting Reference & Stamp public T get(int[] stampHolder); Returns reference to object of type T Returns stamp at array index 0!

Art of Multiprocessor Programming166 Exchanger Status enum Status {EMPTY, WAITING, BUSY};

Art of Multiprocessor Programming167 Exchanger Status enum Status {EMPTY, WAITING, BUSY}; Nothing yet

enum Status {EMPTY, WAITING, BUSY}; Art of Multiprocessor Programming168 Exchange Status Nothing yet One thread is waiting for rendez-vous

Art of Multiprocessor Programming169 Exchange Status enum Status {EMPTY, WAITING, BUSY}; Nothing yet One thread is waiting for rendez-vous Other threads busy with rendez-vous

Art of Multiprocessor Programming170 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {EMPTY}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: … // slot is free case WAITING: … // someone waiting for me case BUSY: … // others exchanging } The Exchange

Art of Multiprocessor Programming171 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {EMPTY}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: … // slot is free case WAITING: … // someone waiting for me case BUSY: … // others exchanging } The Exchange Item and timeout

Art of Multiprocessor Programming172 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {EMPTY}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: … // slot is free case WAITING: … // someone waiting for me case BUSY: … // others exchanging } The Exchange Array holds status

Art of Multiprocessor Programming173 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {0}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: // slot is free case WAITING: // someone waiting for me case BUSY: // others exchanging } }} The Exchange Loop until timeout

Art of Multiprocessor Programming174 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {0}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: // slot is free case WAITING: // someone waiting for me case BUSY: // others exchanging } }} The Exchange Get other’s item and status

Art of Multiprocessor Programming175 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {0}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: … // slot is free case WAITING: … // someone waiting for me case BUSY: … // others exchanging } }} The Exchange An Exchanger has three possible states

Art of Multiprocessor Programming176 Lock-free Exchanger

Art of Multiprocessor Programming177 Lock-free Exchanger CAS

Art of Multiprocessor Programming178 Lock-free Exchanger

Art of Multiprocessor Programming179 Lock-free Exchanger In search of partner …

Art of Multiprocessor Programming180 Lock-free Exchanger Slot Still waiting … Try to exchange item and set status to BUSY CAS

Art of Multiprocessor Programming181 Lock-free Exchanger Slot Partner showed up, take item and reset to EMPTY item status

Art of Multiprocessor Programming182 EMPTY Lock-free Exchanger Slot item status Partner showed up, take item and reset to EMPTY

Art of Multiprocessor Programming183 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY

Art of Multiprocessor Programming184 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY Try to insert myItem and change state to WAITING

Art of Multiprocessor Programming185 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY Spin until either myItem is taken or timeout Spin until either myItem is taken or timeout

Art of Multiprocessor Programming186 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY myItem was taken, so return herItem that was put in its place myItem was taken, so return herItem that was put in its place

Art of Multiprocessor Programming187 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY Otherwise we ran out of time, try to reset status to EMPTY and time out

Art of Multiprocessor Programming188 Art of Multiprocessor Programming© Herlihy-Shavit case EMPTY: // slot is free if (slot.compareAndSet(herItem, myItem, WAITING, BUSY)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.compareAndSet(myItem, null, WAITING, EMPTY)){throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY If reset failed, someone showed up after all, so take that item If reset failed, someone showed up after all, so take that item

Art of Multiprocessor Programming189 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY Clear slot and take that item

Art of Multiprocessor Programming190 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY If initial CAS failed, then someone else changed status from EMPTY to WAITING, so retry from start If initial CAS failed, then someone else changed status from EMPTY to WAITING, so retry from start

Art of Multiprocessor Programming191 case WAITING: // someone waiting for me if (slot.CAS(herItem, myItem, WAITING, BUSY)) return herItem; break; case BUSY: // others in middle of exchanging break; default: // impossible break; } States WAITING and BUSY

Art of Multiprocessor Programming192 case WAITING: // someone waiting for me if (slot.CAS(herItem, myItem, WAITING, BUSY)) return herItem; break; case BUSY: // others in middle of exchanging break; default: // impossible break; } States WAITING and BUSY someone is waiting to exchange, so try to CAS my item in and change state to BUSY someone is waiting to exchange, so try to CAS my item in and change state to BUSY

Art of Multiprocessor Programming193 case WAITING: // someone waiting for me if (slot.CAS(herItem, myItem, WAITING, BUSY)) return herItem; break; case BUSY: // others in middle of exchanging break; default: // impossible break; } States WAITING and BUSY If successful, return other’s item, otherwise someone else took it, so try again from start If successful, return other’s item, otherwise someone else took it, so try again from start

Art of Multiprocessor Programming194 case WAITING: // someone waiting for me if (slot.CAS(herItem, myItem, WAITING, BUSY)) return herItem; break; case BUSY: // others in middle of exchanging break; default: // impossible break; } States WAITING and BUSY If BUSY, other threads exchanging, so start again If BUSY, other threads exchanging, so start again

Art of Multiprocessor Programming195 The Exchanger Slot Exchanger is lock-free Because the only way an exchange can fail is if others repeatedly succeeded or no-one showed up The slot we need does not require symmetric exchange

Art of Multiprocessor Programming196 public class EliminationArray { … public T visit(T value, int range) throws TimeoutException { int slot = random.nextInt(range); int nanodur = convertToNanos(duration, timeUnit)); return (exchanger[slot].exchange(value, nanodur) }} Back to the Stack: the Elimination Array

Art of Multiprocessor Programming197 public class EliminationArray { … public T visit(T value, int range) throws TimeoutException { int slot = random.nextInt(range); int nanodur = convertToNanos(duration, timeUnit)); return (exchanger[slot].exchange(value, nanodur) }} Elimination Array visit the elimination array with fixed value and range visit the elimination array with fixed value and range

Art of Multiprocessor Programming198 public class EliminationArray { … public T visit(T value, int range) throws TimeoutException { int slot = random.nextInt(range); int nanodur = convertToNanos(duration, timeUnit)); return (exchanger[slot].exchange(value, nanodur) }} Elimination Array Pick a random array entry

Art of Multiprocessor Programming199 public class EliminationArray { … public T visit(T value, int range) throws TimeoutException { int slot = random.nextInt(range); int nanodur = convertToNanos(duration, timeUnit)); return (exchanger[slot].exchange(value, nanodur) }} Elimination Array Exchange value or time out

Art of Multiprocessor Programming200 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push

Art of Multiprocessor Programming201 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push First, try to push

Art of Multiprocessor Programming202 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push If I failed, backoff & try to eliminate

Art of Multiprocessor Programming203 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push Value pushed and range to try

Art of Multiprocessor Programming204 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push Only pop() leaves null, so elimination was successful Only pop() leaves null, so elimination was successful

Art of Multiprocessor Programming205 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push Otherwise, retry push() on lock-free stack

Art of Multiprocessor Programming206 public T pop() {... while (true) { if (tryPop()) { return returnNode.value; } else try { T otherValue = eliminationArray.visit(null,policy.range; if (otherValue != null) { return otherValue; } }} Elimination Stack Pop

Art of Multiprocessor Programming207 public T pop() {... while (true) { if (tryPop()) { return returnNode.value; } else try { T otherValue = eliminationArray.visit(null,policy.range; if ( otherValue != null) { return otherValue; } }} Elimination Stack Pop If value not null, other thread is a push(), so elimination succeeded If value not null, other thread is a push(), so elimination succeeded

Art of Multiprocessor Programming208 Summary We saw both lock-based and lock-free implementations of queues and stacks Don’t be quick to declare a data structure inherently sequential –Linearizable stack is not inherently sequential (though it is in the worst case) ABA is a real problem, pay attention

Art of Multiprocessor Programming209 This work is licensed under a Creative Commons Attribution- ShareAlike 2.5 License.Creative Commons Attribution- ShareAlike 2.5 License You are free: –to Share — to copy, distribute and transmit the work –to Remix — to adapt the work Under the following conditions: –Attribution. You must attribute the work to “The Art of Multiprocessor Programming” (but not in any way that suggests that the authors endorse you or your use of the work). –Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license. For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to – Any of the above conditions can be waived if you get permission from the copyright holder. Nothing in this license impairs or restricts the author's moral rights.