Linked Lists: Locking, Lock-Free, and Beyond … The Art of Multiprocessor Programming Spring 2007

© Herlihy-Shavit Last Lecture: Spin-Locks CS Resets lock upon exit spin lock critical section...

© Herlihy-Shavit Today: Concurrent Objects AddingThreads Should not lower throughput –Contention effects –Mostly fixed by Queue locks Should increase throughput –Not possible if inherently sequential –Surprising things are parallelizable

© Herlihy-Shavit Coarse-Grained Synchronization Each method locks the object –Avoid contention using queue locks –Easy to reason about In simple cases –Standard Java model Synchronized blocks and methods So, are we done?

© Herlihy-Shavit Coarse-Grained Synchronization Sequential bottleneck –All threads “stand in line” Adding more threads –Does not improve throughput –Struggle to keep it from getting worse So why even use a multiprocessor? –Well, some apps inherently parallel …

© Herlihy-Shavit This Lecture Introduce four “patterns” –Bag of tricks … –Methods that work more than once … For highly-concurrent objects Goal: –Concurrent access –More threads, more throughput

© Herlihy-Shavit First: Fine-Grained Synchronization Instead of using a single lock.. Split object into –Independently-synchronized components Methods conflict when they access –The same component … –At the same time

© Herlihy-Shavit Second: Optimistic Synchronization Object = linked set of components Search without locking … If you find it, lock and check … –OK, we are done –Oops, try again Evaluation –cheaper than locking –mistakes are expensive

© Herlihy-Shavit Third: Lazy Synchronization Postpone hard work Removing components is tricky –Logical removal Mark component to be deleted –Physical removal Do what needs to be done

© Herlihy-Shavit Fourth: Lock-Free Synchronization Don’t use locks at all –Use compareAndSet() & relatives … Advantages –Robust against asynchrony Disadvantages –Complex –Sometimes high overhead

© Herlihy-Shavit Linked List Illustrate these patterns … Using a list-based Set –Common application –Building block for other apps

© Herlihy-Shavit The Evolution of Concurrent List-based Sets Course Grained Fine Lock- Coupling Course Optimistic Fine Optim- istic Lock- free Lazy Fine Optim- istic

© Herlihy-Shavit Set Interface Unordered collection of objects No duplicates Methods –Add() a new object –Remove() an object –Test if set Contains() object

© Herlihy-Shavit List-Based Sets public interface Set { public boolean add(Object x); public boolean remove(Object x); public boolean contains(Object x); }

© Herlihy-Shavit List-Based Sets public interface Set { public boolean add(Object x); public boolean remove(Object x); public boolean contains(Object x); } Add object to set

© Herlihy-Shavit List-Based Sets public interface Set { public boolean add(Object x); public boolean remove(Object x); public boolean contains(Object x); } Remove object from set

© Herlihy-Shavit List-Based Sets public interface Set { public boolean add(Object x); public boolean remove(Object x); public boolean contains(Object x); } Is object in set?

© Herlihy-Shavit List Node public class Node { public Object object; public int key; public Node next; }

© Herlihy-Shavit List Node public class Node { public Object object; public int key; public Node next; } Object of interest

© Herlihy-Shavit List Node public class Node { public Object object; public int key; public Node next; } Usually hash code

© Herlihy-Shavit List Node public class Node { public Object object; public int key; public Node next; } Reference to next node

© Herlihy-Shavit The List-Based Set abc Ordered + Sentinel nodes (min & max possible keys)

© Herlihy-Shavit Reasoning about Concurrent Objects Invariant –Property that always holds Established by –True when object is created –Truth preserved by each method Each step of each method

© Herlihy-Shavit Specifically … Invariants preserved by –add() –remove() –contains() Most steps are trivial –Usually one step tricky –Often linearization point

© Herlihy-Shavit Interference Proof that invariants preserved works only if –methods considered –are the only modifiers Language encapsulation helps –List nodes not visible outside class

© Herlihy-Shavit Interference Freedom from interference neeeded even for removed nodes –Some algorithms traverse removed nodes –Careful with malloc() & free() ! Garbage-collection helps here

© Herlihy-Shavit Abstract Data Types Concrete representation Abstract Type –{a, b} ab

© Herlihy-Shavit Abstract Data Types Meaning of rep given by abstraction map –S( ) = {a,b} a b

© Herlihy-Shavit Rep Invariant Which concrete values are meaningful? –Sorted? Duplicates? Rep invariant –Characterizes legal concrete reps –Preserved by methods –Relied on by methods

© Herlihy-Shavit Blame Game Rep invariant is a contract Suppose –add() leaves behind 2 copies of x –remove() removes only 1 Which one is incorrect?

© Herlihy-Shavit Blame Game Suppose –add() leaves behind 2 copies of x –remove() removes only 1 Which one is incorrect? –If rep invariant says no duplicates add() is incorrect –Otherwise remove() is incorrect

© Herlihy-Shavit Rep Invariant (partly) Sentinel nodes –tail reachable from head Sorted, no duplicates

© Herlihy-Shavit Abstraction Map S(head) = –{ x | there exists a such that A reachable from head and a.object = x –}

© Herlihy-Shavit Sequential List Based Set a c d b a b c Add() Remove()

© Herlihy-Shavit Course Grained Locking a b d c Simple but hotspot + bottleneck

© Herlihy-Shavit Coarse-Grained Locking Easy, same as synchronized methods –“One lock to rule them all …” Simple, clearly correct –Deserves respect! Works poorly with contention –Queue locks help –But bottleneck still an issue

© Herlihy-Shavit Fine-grained Locking Requires careful thought –“Do not meddle in the affairs of wizards, for they are subtle and quick to anger” Split object into pieces –Each piece has own lock –Methods that work on disjoint pieces need not exclude each other

© Herlihy-Shavit Hand-over-Hand locking abc

© Herlihy-Shavit Hand-over-Hand locking abc

© Herlihy-Shavit Hand-over-Hand locking abc

© Herlihy-Shavit Hand-over-Hand locking abc

© Herlihy-Shavit Hand-over-Hand locking abc

© Herlihy-Shavit Removing a Node abcd remov e b

© Herlihy-Shavit Removing a Node abcd remov e b

© Herlihy-Shavit Removing a Node abcd remov e b

© Herlihy-Shavit Removing a Node abcd remov e b

© Herlihy-Shavit Removing a Node acd remov e b

© Herlihy-Shavit Removing a Node abcd remov e b remov e c

© Herlihy-Shavit Removing a Node abcd remov e b remov e c

© Herlihy-Shavit Removing a Node abcd remov e b remov e c

© Herlihy-Shavit Removing a Node abcd remov e b remov e c

© Herlihy-Shavit Removing a Node abcd remov e b remov e c

© Herlihy-Shavit Removing a Node abcd remov e b remov e c

© Herlihy-Shavit Removing a Node abcd remov e b remov e c

© Herlihy-Shavit Removing a Node abcd remov e b remov e c

© Herlihy-Shavit Removing a Node abcd remov e b remov e c

© Herlihy-Shavit Uh, Oh acd remov e b remov e c Still in list

© Herlihy-Shavit Problem To delete node b –Swing node a’s next field to c Problem is, –Someone could delete c concurrently b acbac

© Herlihy-Shavit Insight If a node is locked –No one can delete node’s successor If a thread locks –Node to be deleted –And its predecessor –Then it works

© Herlihy-Shavit Hand-Over-Hand Again abcd remov e b

© Herlihy-Shavit Hand-Over-Hand Again abcd remov e b

© Herlihy-Shavit Hand-Over-Hand Again abcd remov e b

© Herlihy-Shavit Hand-Over-Hand Again abcd remov e b Found it!

© Herlihy-Shavit Hand-Over-Hand Again abcd remov e b Found it!

© Herlihy-Shavit Hand-Over-Hand Again acd remov e b Found it!

© Herlihy-Shavit Removing a Node abcd remov e b remov e c

© Herlihy-Shavit Removing a Node abcd remov e b remov e c

© Herlihy-Shavit Removing a Node abcd remov e b remov e c

© Herlihy-Shavit Removing a Node abcd remov e b remov e c

© Herlihy-Shavit Removing a Node abcd remov e b remov e c

© Herlihy-Shavit Removing a Node abcd remov e b remov e c

© Herlihy-Shavit Removing a Node abcd remov e b remov e c

© Herlihy-Shavit Removing a Node abd remov e b remov e c

© Herlihy-Shavit Removing a Node abd remov e b remov e c

© Herlihy-Shavit Removing a Node ad remov e b remov e c

© Herlihy-Shavit Remove method public boolean remove(Object object) { int key = object.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }}

© Herlihy-Shavit Remove method public boolean remove(Object object) { int key = object.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }} Key used to order node

© Herlihy-Shavit Remove method public boolean remove(Object object) { int key = object.hashCode(); Node pred, curr; try { … } finally { currNode.unlock(); predNode.unlock(); }} Predecessor and current nodes

© Herlihy-Shavit Remove method public boolean remove(Object object) { int key = object.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }} Make sure locks released

© Herlihy-Shavit Remove method public boolean remove(Object object) { int key = object.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }} Everything else

© Herlihy-Shavit Remove method try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); … } finally { … }

© Herlihy-Shavit Remove method try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); … } finally { … } lock previous

© Herlihy-Shavit Remove method try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); … } finally { … } Lock current

© Herlihy-Shavit Remove method try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); … } finally { … } Traversing list

© Herlihy-Shavit Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

© Herlihy-Shavit Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Search key range

© Herlihy-Shavit Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; At start of each loop: curr and pred locked

© Herlihy-Shavit Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; If node found, remove it

© Herlihy-Shavit Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; If node found, remove it

© Herlihy-Shavit Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Unlock predecessor

© Herlihy-Shavit Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Only one node locked!

© Herlihy-Shavit Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; demote current

© Herlihy-Shavit Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = currNode; curr = curr.next; curr.lock(); } return false; Find and lock new current

© Herlihy-Shavit Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = currNode; curr = curr.next; curr.lock(); } return false; Lock invariant restored

© Herlihy-Shavit Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Otherwise, not present

© Herlihy-Shavit Why does this work? To remove node e –Must lock e –Must lock e’s predecessor Therefore, if you lock a node –It can’t be removed –And neither can its successor

© Herlihy-Shavit while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable head ~> pred A --> curr A so the object is in the set

© Herlihy-Shavit while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable Node locked, so no other thread can remove it ….

© Herlihy-Shavit while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable Linearization point

© Herlihy-Shavit while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable Object not present

© Herlihy-Shavit while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable pred A --> curr A pred A.key < key key < curr A.key

© Herlihy-Shavit while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable Linearization point: when curr A set to node with higher key

© Herlihy-Shavit Adding Nodes To add node e –Must lock predecessor –Must lock successor Neither can be deleted –(Is successor lock actually required?)

© Herlihy-Shavit Same Abstraction Map S(head) = –{ x | there exists a such that A reachable from head and a.object = x –}

© Herlihy-Shavit Rep Invariant Easy to check that –Tail always reachable from head –Nodes sorted, no duplicates

© Herlihy-Shavit Drawbacks Better than coarse-grained lock –Threads can traverse in parallel Still not ideal –Long chain of acquire/release –Inefficient

© Herlihy-Shavit Optimistic Synchronization Find nodes without locking Lock nodes Check that everything is OK

© Herlihy-Shavit Optimistic Fine Grained b d e To insert C: Optimistically traverse list to find b Lock b.curr then lock b.succ Re-Traverse list to find b and verify pred  curr Perform insert and release locks a c 4

© Herlihy-Shavit Same Abstraction Map S(head) = –{ x | there exists a such that A reachable from head and a.object = x –}

© Herlihy-Shavit Preserving Invariants Not same as before as we traverse deleted elements But we establish properties by –Validation –After we lock target nodes

© Herlihy-Shavit Scanning Deleted Items abde remov e e A Pred and curr could point to removed nodes

© Herlihy-Shavit Removing a Node abcd remov e c return true

© Herlihy-Shavit What Can Go Wrong? abcd remov e c remov e b

© Herlihy-Shavit Check that Node is Still Accessible abcd remov e c

© Herlihy-Shavit What Can Go Wrong? abcd remov e c Add b’ b’

© Herlihy-Shavit What Can Go Wrong? abcd remov e c b’

© Herlihy-Shavit Check that Nodes Still Adjacent abcd remov e c

© Herlihy-Shavit Correctness If –Nodes b and c both locked –Node b still accessible –Node c still successor to b Then –Neither will be deleted –OK to delete and return true

© Herlihy-Shavit Removing an Absent Node abde remov e c return false

© Herlihy-Shavit Correctness If –Nodes b and d both locked –Node b still accessible –Node d still successor to b Then –Neither will be deleted –No thread can add c after b –OK to return false

© Herlihy-Shavit Validation private boolean validate(Node pred, Node curry) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; }

© Herlihy-Shavit private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Predecessor & current nodes

© Herlihy-Shavit private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Start at the beginning

© Herlihy-Shavit private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Search range of keys

© Herlihy-Shavit private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Predecessor reachable

© Herlihy-Shavit private boolean validate(Node pred, Node curry) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Is current node next?

© Herlihy-Shavit private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Otherwise move on

© Herlihy-Shavit private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Predecessor not reachable

© Herlihy-Shavit Remove: searching public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } …

© Herlihy-Shavit public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } … Remove: searching Search key

© Herlihy-Shavit public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } … Remove: searching Retry on synchronization conflict

© Herlihy-Shavit public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } … Remove: searching Examine predecessor and current nodes

© Herlihy-Shavit public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } … Remove: searching Search by key

© Herlihy-Shavit public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } … Remove: searching Stop if we find object

© Herlihy-Shavit public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } … Remove: searching Move along

© Herlihy-Shavit On Exit from Loop If object is present –curr holds object –pred just before curr If object is absent –curr has first higher key –pred just before curr Assuming no synchronization problems

© Herlihy-Shavit Remove Method try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}}

© Herlihy-Shavit try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Remove Method Always unlock

© Herlihy-Shavit try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Remove Method Lock both nodes

© Herlihy-Shavit try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Remove Method Check for synchronization conflicts

© Herlihy-Shavit try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Remove Method Object found, remove node

© Herlihy-Shavit try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Remove Method Object not found

© Herlihy-Shavit Summary: Wait-free Traversals b c d a 1.Traverse removed nodes 2.Must have non-interference 3.Natural in GC languages like Java™ Reachable = in the set

© Herlihy-Shavit Summary Optimistic List b c e a 1.Limited Hotspots (Only at locked Add(), Remove(), Find() destination locations, not traversals) 2.But two traversals 3.Yet traversals are wait-free!

© Herlihy-Shavit So Far, So Good Much less lock acquisition/release –Performance –Concurrency Problems –Need to traverse list twice –contains() method acquires locks Most common method call

© Herlihy-Shavit Evaluation Optimistic works if –cost of scanning twice without locks < –cost of Scanning once with locks Drawback –Contains() acquires locks –90% of calls in many apps

© Herlihy-Shavit Lazy List Like optimistic, except –Scan once –Contains() never locks … Key insight –Removing nodes causes trouble –Do it “lazily”

© Herlihy-Shavit Lazy List Remove Method –Scans list (as before) –Locks predecessor & current (as before) Logical delete –Marks current node as removed (new!) Physical delete –Redirects predecessor’s next (as before)

© Herlihy-Shavit Lazy Removal a a b c 0 d 1 c Logical Removal = Set Mark Bit Physical Removal Add a separate mark bit field to each node: 1. mark node as deleted 2. redirect pointer

© Herlihy-Shavit Lazy Removal a a b c 0 d c 4 Logical Removal Physical Removal Release Locks 1.Optimistically traverse list to find b 2.Lock b.curr then lock b.succ 3.verify pred  curr 4.Perform logical then physical removal 5.Release locks

© Herlihy-Shavit Lazy List All Methods –Scan through locked and marked nodes –Removing a node doesn’t slow down other method calls … Must still lock pred and curr nodes.

© Herlihy-Shavit Validation No need to rescan list! Check that pred is not marked Check that curr is not marked Check that pred points to curr

© Herlihy-Shavit Business as Usual abcd

© Herlihy-Shavit Business as Usual abcd

© Herlihy-Shavit Business as Usual abcd

© Herlihy-Shavit Business as Usual abcd

© Herlihy-Shavit Business as Usual abcd

© Herlihy-Shavit Interference abcd

© Herlihy-Shavit Interference abcd

© Herlihy-Shavit Interference abcd

© Herlihy-Shavit Interference abcd Remove c

© Herlihy-Shavit Validation abcd b not marked

© Herlihy-Shavit Interference abcd c not marked

© Herlihy-Shavit Interference abcd b still points to c

© Herlihy-Shavit Logical Delete abcd mark

© Herlihy-Shavit Scan Through abcd So what? Remove c

© Herlihy-Shavit Physical Deletion abcd So what? Remove c

© Herlihy-Shavit New Abstraction Map S(head) = –{ x | there exists node a such that A reachable from head and a.object = x and a is unmarked –}

© Herlihy-Shavit Invariant If not marked then item in the set and reachable from head and if not yet traversed it is reachable from pred

© Herlihy-Shavit Validation private boolean validate(Node pred, Node curr) { return !pred.next.marked && !curr.next.marked && pred.next == curr); }

© Herlihy-Shavit private boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); } List Validate Method Predecessor not Logically removed

© Herlihy-Shavit private boolean validate(Node pred, Node curr) { return !pred.next.marked && !curr.next.marked && pred.next == curr); } List Validate Method Current not Logically removed

© Herlihy-Shavit private boolean validate(Node pred, Node curr) { return !pred.next.marked && !curr.next.marked && pred.next == curr); } List Validate Method Predecessor still Points to current

© Herlihy-Shavit Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}}

© Herlihy-Shavit Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Validate as before

© Herlihy-Shavit Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Key found

© Herlihy-Shavit Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Logical remove

© Herlihy-Shavit Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} physical remove

© Herlihy-Shavit Contains public boolean contains(Object object) { int key = object.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; }

© Herlihy-Shavit Contains public boolean contains(Object object) { int key = object.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Start at the head

© Herlihy-Shavit Contains public boolean contains(Object object) { int key = object.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Search key range

© Herlihy-Shavit Contains public boolean contains(Object object) { int key = object.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Traverse without locking (nodes may have been removed)

© Herlihy-Shavit Contains public boolean contains(Object object) { int key = object.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Present and undeleted?

© Herlihy-Shavit Summary: Wait-free Contains a a b c 0 e 1 d Use Mark bit + Fact that List is ordered 1.Not marked  in the set 2.Marked or missing  not in the set

© Herlihy-Shavit Lazy List a a b c 0 e 1 d Lazy Add() and Remove() + Wait-free Contains()

© Herlihy-Shavit Evaluation Good: –Contains method doesn’t need to lock –In fact, its wait-free! –Good because typically high % contains() –Uncontended calls don’t re-traverse Bad –Contended calls do re-traverse –Traffic jam if one thread delays

© Herlihy-Shavit Traffic Jam Any concurrent data structure based on mutual exclusion has a weakness If one thread –Enters critical section –And “eats the big muffin” (stops running) Cache miss, page fault, descheduled … –Everyone else using that lock is stuck!

© Herlihy-Shavit Reminder: Lock-Free Data Structures No matter what … –Some thread will complete method call –Even if others halt at malicious times –Weaker than wait-free, yet Implies that –You can’t use locks –Um, that’s why they call it lock-free

© Herlihy-Shavit Lock-free Lists Next logical step Eliminate locking entirely Contains() wait-free and Add() and Remove() lock-free Use only compareAndSet() What could go wrong?

© Herlihy-Shavit Adding a Node abc

© Herlihy-Shavit Adding a Node abcb

© Herlihy-Shavit Adding a Node abcb CAS

© Herlihy-Shavit Adding a Node abcb

© Herlihy-Shavit Adding a Node abcb

© Herlihy-Shavit Removing a Node abcd remov e b remov e c CAS

© Herlihy-Shavit Look Familiar? abcd remov e b remov e c Still in list

© Herlihy-Shavit Problem Method updates node’s next field After node has been removed

© Herlihy-Shavit Solution Use AtomicMarkableReference Atomically –Swing reference and –Update flag Remove in two steps –Set mark bit in next field –Redirect predecessor’s pointer

© Herlihy-Shavit Marking a Node AtomicMarkableReference class –Java.util.concurrent.atomic package address F mark bit Reference

© Herlihy-Shavit Extracting Reference & Mark Public T get(boolean[] marked);

© Herlihy-Shavit Extracting Reference & Mark Public T get(boolean[] marked); Returns reference to object of type T Returns mark at array index 0!

© Herlihy-Shavit Changing State Public boolean compareAndSet( T expectedRef, T updateRef, boolean expectedMark, boolean updateMark);

© Herlihy-Shavit Changing State Public boolean compareAndSet( T expectedRef, T updateRef, boolean expectedMark, boolean updateMark); If this is the current reference … And this is the current mark …

© Herlihy-Shavit Changing State Public boolean compareAndSet( T expectedRef, T updateRef, boolean expectedMark, boolean updateMark); …then change to this new reference … … and this new mark

© Herlihy-Shavit Changing State public boolean attemptMark( T expectedRef, boolean updateMark);

© Herlihy-Shavit Changing State public boolean attemptMark( T expectedRef, boolean updateMark); If this is the current reference …

© Herlihy-Shavit Changing State public boolean attemptMark( T expectedRef, boolean updateMark);.. then change to this new mark.

© Herlihy-Shavit Removing a Node abcd remov e c CAS

© Herlihy-Shavit Removing a Node abd remov e b remov e c c CAS failed

© Herlihy-Shavit Removing a Node abd remov e b remov e c c

© Herlihy-Shavit Removing a Node ad remov e b remov e c

© Herlihy-Shavit Traversing the List Q: what do you do when you find a “logically” deleted node in your path? A: finish the job. –CAS the predecessor’s next field –Proceed (repeat as needed)

© Herlihy-Shavit Lock-Free Traversal abcd CAS Uh-oh

© Herlihy-Shavit The Window Class class Window { public Node pred; public Node curr; Window(Node pred, Node curr) { this.pred = pred; this.curr = curr; }

© Herlihy-Shavit The Window Class class Window { public Node pred; public Node curr; Window(Node pred, Node curr) { this.pred = pred; this.curr = curr; } A container for pred and current values

© Herlihy-Shavit Using the Find Method Window window = find(head, key); Node pred = window.pred; curr = window.curr;

© Herlihy-Shavit Using the Find Method Window window = find(head, key); Node pred = window.pred; curr = window.curr; Find returns window

© Herlihy-Shavit Using the Find Method Window window = find(head, key); Node pred = window.pred; curr = window.curr; Extract pred and curr

© Herlihy-Shavit The Find Method Window window = find(head, key); At some instant, predcurrsucc object or …

© Herlihy-Shavit The Find Method Window window = find(head,key); At some instant, pred curr= succ succ object not in list

© Herlihy-Shavit Remove public boolean remove(T object) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}}

© Herlihy-Shavit Remove public boolean remove(T object) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} Keep trying

© Herlihy-Shavit Remove public boolean remove(T object) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} Find neighbors

© Herlihy-Shavit Remove public boolean remove(T object) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} She’s not there …

© Herlihy-Shavit Remove public boolean remove(T object) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} Try to mark node as deleted

© Herlihy-Shavit Remove public boolean remove(T object) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} If it doesn’t work, just retry, if it does, job essentially done

© Herlihy-Shavit Remove public boolean remove(T object) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} Try to advance reference (if we don’t succeed, someone else did or will). a

© Herlihy-Shavit Add public boolean add(T object) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(object); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true; }}}}

© Herlihy-Shavit Add public boolean add(T object) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(object); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true; }}}} Object already there.

© Herlihy-Shavit Add public boolean add(T object) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(object); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true; }}}} create new node

© Herlihy-Shavit Add public boolean add(T object) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(object); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true }}}} Install new node, else retry loop

© Herlihy-Shavit Wait-free Contains public boolean contains(T object) { boolean marked; int key = object.hashCode(); Node curr = this.head; while (curr.key < key) curr = curr.next; Node succ = curr.next.get(marked); return (curr.key == key && !marked[0]) }

© Herlihy-Shavit Wait-free Contains public boolean contains(T object) { boolean marked; int key = object.hashCode(); Node curr = this.head; while (curr.key < key) curr = curr.next; Node succ = curr.next.get(marked); return (curr.key == key && !marked[0]) } Only diff from lazy list is that we get and check marked

© Herlihy-Shavit Lock-free Find public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }}

© Herlihy-Shavit Lock-free Find public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} If list changes while traversed, start over Lock-Free because we start over only if someone else makes progress

© Herlihy-Shavit public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find Start looking from head

© Herlihy-Shavit public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find Move down the list

© Herlihy-Shavit public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find Get ref to successor and current deleted bit

© Herlihy-Shavit public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find Try to remove deleted nodes in path…code details soon

© Herlihy-Shavit public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find If found curr key that is greater or equal, return pred and curr

© Herlihy-Shavit public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find Otherwise advance “window” and loop again

© Herlihy-Shavit Lock-free Find retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); } …

© Herlihy-Shavit Lock-free Find retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); } … Try to snip out node

© Herlihy-Shavit Lock-free Find retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); } … if predecessor’s next field changed must retry whole traversal

© Herlihy-Shavit Lock-free Find retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); } … Otherwise move on to check if next node deleted

© Herlihy-Shavit Summary: Summary: Lock- free Removal a a b c 0 e 1 c Logical Removal = Set Mark Bit Physical Removal CAS pointer Use CAS to verify pointer is correct Not enough!

© Herlihy-Shavit Summary: Summary: Lock-free Removal a a b c 0 e 1 c Logical Removal = Set Mark Bit Physical Removal CAS 0 d Problem: d not added to list… Must Prevent manipulation of removed node’s pointer Node added Before Physical Removal CAS

© Herlihy-Shavit Our Solution: Combine Bit and Pointer a a b c 0 e 1 c Logical Removal = Set Mark Bit Physical Removal CAS 0 d Mark-Bit and Pointer are CASed together Fail CAS: Node not added after logical Removal

© Herlihy-Shavit A Lock-free Algorithm a a b c 0 e 1 c 1. Add() and Remove() physically clean up (remove) marked nodes 2. Wait-free find() traverses both marked and removed nodes

© Herlihy-Shavit Performance On 16 node shared memory machine Benchmark throughput of Java List-based Set algs. Vary % of Contains() method Calls.

© Herlihy-Shavit High Contains Ratio Lock-free Lazy list Course Grained Fine Lock-coupling

© Herlihy-Shavit Low Contains Ratio

© Herlihy-Shavit As Contains Ratio Increases % Contains()

© Herlihy-Shavit Summary Coarse-grained locking Fine-grained locking Optimistic synchronization Lock-free synchronization

© Herlihy-Shavit “To Lock or Not to Lock” Locking vs. Non-blocking: Extremist views on both sides The answer: nobler to compromise, combine locking and non-blocking –Example: Lazy list combines blocking add() and remove() and a wait-free contains() –Blocking/non-blocking is a property of a method