Concurrent Stacks Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

Art of Multiprocessor Programming22 Another Fundamental Problem We told you about –Sets implemented by linked lists –Queues Next: stacks

Art of Multiprocessor Programming33 Concurrent Stack Methods –push(x) –pop() Last-in, First-out (LIFO) order Lock-Free!

Art of Multiprocessor Programming44 Empty Stack Top

Art of Multiprocessor Programming55 Push Top

Art of Multiprocessor Programming66 Push Top CAS

Art of Multiprocessor Programming77 Push Top

Art of Multiprocessor Programming88 Push Top

Art of Multiprocessor Programming99 Push Top

Art of Multiprocessor Programming10 Push Top CAS

Art of Multiprocessor Programming11 Push Top

Art of Multiprocessor Programming12 Pop Top

Art of Multiprocessor Programming13 Pop Top CAS

Art of Multiprocessor Programming14 Pop Top CAS mine!

Art of Multiprocessor Programming15 Pop Top CAS

Art of Multiprocessor Programming16 Pop Top

Art of Multiprocessor Programming17 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff(); }} Lock-free Stack

Art of Multiprocessor Programming18 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public Boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack tryPush attempts to push a node

Art of Multiprocessor Programming19 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack Read top value

Art of Multiprocessor Programming20 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack current top will be new node’s successor

Art of Multiprocessor Programming21 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack Try to swing top, return success or failure

Art of Multiprocessor Programming22 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack Push calls tryPush

Art of Multiprocessor Programming23 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack Create new node

Art of Multiprocessor Programming24 public class LockFreeStack { private AtomicReference top = new AtomicReference(null); public boolean tryPush(Node node){ Node oldTop = top.get(); node.next = oldTop; return(top.compareAndSet(oldTop, node)) } public void push(T value) { Node node = new Node(value); while (true) { if (tryPush(node)) { return; } else backoff.backoff() }} Lock-free Stack If tryPush() fails, back off before retrying

Art of Multiprocessor Programming25 Lock-free Stack Good –No locking Bad –Without GC, fear ABA –Without backoff, huge contention at top –In any case, no parallelism

Art of Multiprocessor Programming26 Big Question Are stacks inherently sequential? Reasons why –Every pop() call fights for top item Reasons why not –Stay tuned …

Art of Multiprocessor Programming27 Elimination-Backoff Stack How to –“turn contention into parallelism” Replace familiar –exponential backoff With alternative –elimination-backoff

Art of Multiprocessor Programming28 Observation Push( ) Pop() linearizable stack After an equal number of pushes and pops, stack stays the same After an equal number of pushes and pops, stack stays the same Yes!

Art of Multiprocessor Programming29 Idea: Elimination Array Push( ) Pop() stack Pick at random Pick at random Elimination Array

Art of Multiprocessor Programming30 Push Collides With Pop Push( ) Pop() stack continue No need to access stack No need to access stack Yes!

Art of Multiprocessor Programming31 No Collision Push( ) Pop() stack If no collision, access stack If no collision, access stack If pushes collide or pops collide access stack

Art of Multiprocessor Programming32 Elimination-Backoff Stack Lock-free stack + elimination array Access Lock-free stack, –If uncontended, apply operation –if contended, back off to elimination array and attempt elimination

Art of Multiprocessor Programming33 Elimination-Backoff Stack Push( ) Pop() Top CAS If CAS fails, back off

Art of Multiprocessor Programming34 Dynamic Range and Delay Push( ) Pick random range and max waiting time based on level of contention encountered

Art of Multiprocessor Programming35 Linearizability Un-eliminated calls –linearized as before Eliminated calls: –linearize pop() immediately after matching push() Combination is a linearizable stack

Art of Multiprocessor Programming36 Backoff Has Dual Effect Elimination introduces parallelism Backoff to array cuts contention on lock- free stack Elimination in array cuts down number of threads accessing lock-free stack

Art of Multiprocessor Programming37 public class EliminationArray { private static final int duration =...; private static final int timeUnit =...; Exchanger [] exchanger; public EliminationArray(int capacity) { exchanger = new Exchanger[capacity]; for (int i = 0; i < capacity; i++) exchanger[i] = new Exchanger (); … } … } Elimination Array

Art of Multiprocessor Programming38 public class EliminationArray { private static final int duration =...; private static final int timeUnit =...; Exchanger [] exchanger; public EliminationArray(int capacity) { exchanger = new Exchanger[capacity]; for (int i = 0; i < capacity; i++) exchanger[i] = new Exchanger (); … } … } Elimination Array An array of Exchangers

Art of Multiprocessor Programming39 public class Exchanger { AtomicStampedReference slot = new AtomicStampedReference (null, 0); Digression: A Lock-Free Exchanger

Art of Multiprocessor Programming40 public class Exchanger { AtomicStampedReference slot = new AtomicStampedReference (null, 0); A Lock-Free Exchanger Atomically modifiable reference + status

Art of Multiprocessor Programming41 Atomic Stamped Reference AtomicStampedReference class –Java.util.concurrent.atomic package In C or C++: address S reference stamp

Art of Multiprocessor Programming42 Extracting Reference & Stamp public T get(int[] stampHolder);

Art of Multiprocessor Programming43 Extracting Reference & Stamp public T get(int[] stampHolder); Returns reference to object of type T Returns stamp at array index 0!

Art of Multiprocessor Programming44 Exchanger Status enum Status {EMPTY, WAITING, BUSY};

Art of Multiprocessor Programming45 Exchanger Status enum Status {EMPTY, WAITING, BUSY}; Nothing yet

enum Status {EMPTY, WAITING, BUSY}; Art of Multiprocessor Programming46 Exchange Status Nothing yet One thread is waiting for rendez-vous

Art of Multiprocessor Programming47 Exchange Status enum Status {EMPTY, WAITING, BUSY}; Nothing yet One thread is waiting for rendez-vous Other threads busy with rendez-vous

Art of Multiprocessor Programming48 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {EMPTY}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: … // slot is free case WAITING: … // someone waiting for me case BUSY: … // others exchanging } The Exchange

Art of Multiprocessor Programming49 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {EMPTY}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: … // slot is free case WAITING: … // someone waiting for me case BUSY: … // others exchanging } The Exchange Item and timeout

Art of Multiprocessor Programming50 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {EMPTY}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: … // slot is free case WAITING: … // someone waiting for me case BUSY: … // others exchanging } The Exchange Array holds status

Art of Multiprocessor Programming51 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {EMPTY}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: // slot is free case WAITING: // someone waiting for me case BUSY: // others exchanging } }} The Exchange Loop until timeout

Art of Multiprocessor Programming52 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {EMPTY}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: // slot is free case WAITING: // someone waiting for me case BUSY: // others exchanging } }} The Exchange Get other’s item and status

Art of Multiprocessor Programming53 public T Exchange(T myItem, long nanos) throws TimeoutException { long timeBound = System.nanoTime() + nanos; int[] stampHolder = {0}; while (true) { if (System.nanoTime() > timeBound) throw new TimeoutException(); T herItem = slot.get(stampHolder); int stamp = stampHolder[0]; switch(stamp) { case EMPTY: … // slot is free case WAITING: … // someone waiting for me case BUSY: … // others exchanging } }} The Exchange An Exchanger has three possible states

Art of Multiprocessor Programming54 Lock-free Exchanger

Art of Multiprocessor Programming55 Lock-free Exchanger CAS

Art of Multiprocessor Programming56 Lock-free Exchanger

Art of Multiprocessor Programming57 Lock-free Exchanger In search of partner …

Art of Multiprocessor Programming58 Lock-free Exchanger Slot Still waiting … Try to exchange item and set status to BUSY CAS

Art of Multiprocessor Programming59 Lock-free Exchanger Slot Partner showed up, take item and reset to EMPTY item status

Art of Multiprocessor Programming60 EMPTY Lock-free Exchanger Slot item status

Art of Multiprocessor Programming61 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY

Art of Multiprocessor Programming62 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY Try to insert myItem and change state to WAITING

Art of Multiprocessor Programming63 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY Spin until either myItem is taken or timeout Spin until either myItem is taken or timeout

Art of Multiprocessor Programming64 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY myItem was taken, so return herItem that was put in its place myItem was taken, so return herItem that was put in its place

Art of Multiprocessor Programming65 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY Otherwise we ran out of time, try to reset status to EMPTY and time out

Art of Multiprocessor Programming66Art of Multiprocessor Programming© Herlihy-Shavit case EMPTY: // slot is free if (slot.compareAndSet(herItem, myItem, WAITING, BUSY)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.compareAndSet(myItem, null, WAITING, EMPTY)){throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY If reset failed, someone showed up after all, so take that item If reset failed, someone showed up after all, so take that item

Art of Multiprocessor Programming67 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY Clear slot and take that item

Art of Multiprocessor Programming68 case EMPTY: // slot is free if (slot.CAS(herItem, myItem, EMPTY, WAITING)) { while (System.nanoTime() < timeBound){ herItem = slot.get(stampHolder); if (stampHolder[0] == BUSY) { slot.set(null, EMPTY); return herItem; }} if (slot.CAS(myItem, null, WAITING, EMPTY)){ throw new TimeoutException(); } else { herItem = slot.get(stampHolder); slot.set(null, EMPTY); return herItem; } } break; Exchanger State EMPTY If initial CAS failed, then someone else changed status from EMPTY to WAITING, so retry from start If initial CAS failed, then someone else changed status from EMPTY to WAITING, so retry from start

Art of Multiprocessor Programming69 case WAITING: // someone waiting for me if (slot.CAS(herItem, myItem, WAITING, BUSY)) return herItem; break; case BUSY: // others in middle of exchanging break; default: // impossible break; } States WAITING and BUSY

Art of Multiprocessor Programming70 case WAITING: // someone waiting for me if (slot.CAS(herItem, myItem, WAITING, BUSY)) return herItem; break; case BUSY: // others in middle of exchanging break; default: // impossible break; } States WAITING and BUSY someone is waiting to exchange, so try to CAS my item in and change state to BUSY someone is waiting to exchange, so try to CAS my item in and change state to BUSY

Art of Multiprocessor Programming71 case WAITING: // someone waiting for me if (slot.CAS(herItem, myItem, WAITING, BUSY)) return herItem; break; case BUSY: // others in middle of exchanging break; default: // impossible break; } States WAITING and BUSY If successful, return other’s item, otherwise someone else took it, so try again from start If successful, return other’s item, otherwise someone else took it, so try again from start

Art of Multiprocessor Programming72 case WAITING: // someone waiting for me if (slot.CAS(herItem, myItem, WAITING, BUSY)) return herItem; break; case BUSY: // others in middle of exchanging break; default: // impossible break; } States WAITING and BUSY If BUSY, other threads exchanging, so start again If BUSY, other threads exchanging, so start again

Art of Multiprocessor Programming73 The Exchanger Slot Exchanger is lock-free Because the only way an exchange can fail is if others repeatedly succeeded or no-one showed up The slot we need does not require symmetric exchange

Art of Multiprocessor Programming74 public class EliminationArray { … public T visit(T value, int range) throws TimeoutException { int slot = random.nextInt(range); int nanodur = convertToNanos(duration, timeUnit)); return (exchanger[slot].exchange(value, nanodur) }} Back to the Stack: the Elimination Array

Art of Multiprocessor Programming75 public class EliminationArray { … public T visit(T value, int range) throws TimeoutException { int slot = random.nextInt(range); int nanodur = convertToNanos(duration, timeUnit)); return (exchanger[slot].exchange(value, nanodur) }} Elimination Array visit the elimination array with fixed value and range visit the elimination array with fixed value and range

Art of Multiprocessor Programming76 public class EliminationArray { … public T visit(T value, int range) throws TimeoutException { int slot = random.nextInt(range); int nanodur = convertToNanos(duration, timeUnit)); return (exchanger[slot].exchange(value, nanodur) }} Elimination Array Pick a random array entry

Art of Multiprocessor Programming77 public class EliminationArray { … public T visit(T value, int range) throws TimeoutException { int slot = random.nextInt(range); int nanodur = convertToNanos(duration, timeUnit)); return (exchanger[slot].exchange(value, nanodur) }} Elimination Array Exchange value or time out

Art of Multiprocessor Programming78 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push

Art of Multiprocessor Programming79 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push First, try to push

Art of Multiprocessor Programming80 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push If I failed, backoff & try to eliminate

Art of Multiprocessor Programming81 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push Value pushed and range to try

Art of Multiprocessor Programming82 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push Only pop() leaves null, so elimination was successful Only pop() leaves null, so elimination was successful

Art of Multiprocessor Programming83 public void push(T value) {... while (true) { if (tryPush(node)) { return; } else try { T otherValue = eliminationArray.visit(value,policy.range); if (otherValue == null) { return; } Elimination Stack Push Otherwise, retry push() on lock-free stack

Art of Multiprocessor Programming84 public T pop() {... while (true) { if (tryPop()) { return returnNode.value; } else try { T otherValue = eliminationArray.visit(null,policy.range; if (otherValue != null) { return otherValue; } }} Elimination Stack Pop

Art of Multiprocessor Programming85 public T pop() {... while (true) { if (tryPop()) { return returnNode.value; } else try { T otherValue = eliminationArray.visit(null,policy.range; if (otherValue != null) { return otherValue; } }} Elimination Stack Pop If value not null, other thread is a push(), so elimination succeeded If value not null, other thread is a push(), so elimination succeeded

Art of Multiprocessor Programming86 Summary We saw both lock-based and lock-free implementations of queues and stacks Don’t be quick to declare a data structure inherently sequential –Linearizable stack is not inherently sequential (though it is in worst case) ABA is a real problem, pay attention

Art of Multiprocessor Programming87 This work is licensed under a Creative Commons Attribution- ShareAlike 2.5 License.Creative Commons Attribution- ShareAlike 2.5 License You are free: –to Share — to copy, distribute and transmit the work –to Remix — to adapt the work Under the following conditions: –Attribution. You must attribute the work to “The Art of Multiprocessor Programming” (but not in any way that suggests that the authors endorse you or your use of the work). –Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license. For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to – Any of the above conditions can be waived if you get permission from the copyright holder. Nothing in this license impairs or restricts the author's moral rights.