1. Linked Lists: Locking, Lock-Free, and Beyond … The Art of Multiprocessor Programming Spring 2007

2. © Herlihy-Shavit 20072 Last Lecture: Spin-Locks CS Resets lock upon exit spin lock critical section...

3. © Herlihy-Shavit 20073 Today: Concurrent Objects AddingThreads Should not lower throughput –Contention effects –Mostly fixed by Queue locks Should increase throughput –Not possible if inherently sequential –Surprising things are parallelizable

4. © Herlihy-Shavit 20074 Coarse-Grained Synchronization Each method locks the object –Avoid contention using queue locks –Easy to reason about In simple cases –Standard Java model Synchronized blocks and methods So, are we done?

5. © Herlihy-Shavit 20075 Coarse-Grained Synchronization Sequential bottleneck –All threads “stand in line” Adding more threads –Does not improve throughput –Struggle to keep it from getting worse So why even use a multiprocessor? –Well, some apps inherently parallel …

6. © Herlihy-Shavit 20076 This Lecture Introduce four “patterns” –Bag of tricks … –Methods that work more than once … For highly-concurrent objects Goal: –Concurrent access –More threads, more throughput

7. © Herlihy-Shavit 20077 First: Fine-Grained Synchronization Instead of using a single lock.. Split object into –Independently-synchronized components Methods conflict when they access –The same component … –At the same time

8. © Herlihy-Shavit 20078 Second: Optimistic Synchronization Object = linked set of components Search without locking … If you find it, lock and check … –OK, we are done –Oops, try again Evaluation –cheaper than locking –mistakes are expensive

9. © Herlihy-Shavit 20079 Third: Lazy Synchronization Postpone hard work Removing components is tricky –Logical removal Mark component to be deleted –Physical removal Do what needs to be done

10. © Herlihy-Shavit 200710 Fourth: Lock-Free Synchronization Don’t use locks at all –Use compareAndSet() & relatives … Advantages –Robust against asynchrony Disadvantages –Complex –Sometimes high overhead

11. © Herlihy-Shavit 200711 Linked List Illustrate these patterns … Using a list-based Set –Common application –Building block for other apps

12. © Herlihy-Shavit 200712 The Evolution of Concurrent List-based Sets Course Grained Fine Lock- Coupling Course Optimistic Fine Optim- istic Lock- free Lazy Fine Optim- istic

13. © Herlihy-Shavit 200713 Set Interface Unordered collection of objects No duplicates Methods –Add() a new object –Remove() an object –Test if set Contains() object

14. © Herlihy-Shavit 200714 List-Based Sets public interface Set { public boolean add(Object x); public boolean remove(Object x); public boolean contains(Object x); }

15. © Herlihy-Shavit 200715 List-Based Sets public interface Set { public boolean add(Object x); public boolean remove(Object x); public boolean contains(Object x); } Add object to set

16. © Herlihy-Shavit 200716 List-Based Sets public interface Set { public boolean add(Object x); public boolean remove(Object x); public boolean contains(Object x); } Remove object from set

17. © Herlihy-Shavit 200717 List-Based Sets public interface Set { public boolean add(Object x); public boolean remove(Object x); public boolean contains(Object x); } Is object in set?

18. © Herlihy-Shavit 200718 List Node public class Node { public Object object; public int key; public Node next; }

19. © Herlihy-Shavit 200719 List Node public class Node { public Object object; public int key; public Node next; } Object of interest

20. © Herlihy-Shavit 200720 List Node public class Node { public Object object; public int key; public Node next; } Usually hash code

21. © Herlihy-Shavit 200721 List Node public class Node { public Object object; public int key; public Node next; } Reference to next node

22. © Herlihy-Shavit 200722 The List-Based Set abc Ordered + Sentinel nodes (min & max possible keys)

23. © Herlihy-Shavit 200723 Reasoning about Concurrent Objects Invariant –Property that always holds Established by –True when object is created –Truth preserved by each method Each step of each method

24. © Herlihy-Shavit 200724 Specifically … Invariants preserved by –add() –remove() –contains() Most steps are trivial –Usually one step tricky –Often linearization point

25. © Herlihy-Shavit 200725 Interference Proof that invariants preserved works only if –methods considered –are the only modifiers Language encapsulation helps –List nodes not visible outside class

26. © Herlihy-Shavit 200726 Interference Freedom from interference neeeded even for removed nodes –Some algorithms traverse removed nodes –Careful with malloc() & free() ! Garbage-collection helps here

27. © Herlihy-Shavit 200727 Abstract Data Types Concrete representation Abstract Type –{a, b} ab

28. © Herlihy-Shavit 200728 Abstract Data Types Meaning of rep given by abstraction map –S( ) = {a,b} a b

29. © Herlihy-Shavit 200729 Rep Invariant Which concrete values are meaningful? –Sorted? Duplicates? Rep invariant –Characterizes legal concrete reps –Preserved by methods –Relied on by methods

30. © Herlihy-Shavit 200730 Blame Game Rep invariant is a contract Suppose –add() leaves behind 2 copies of x –remove() removes only 1 Which one is incorrect?

31. © Herlihy-Shavit 200731 Blame Game Suppose –add() leaves behind 2 copies of x –remove() removes only 1 Which one is incorrect? –If rep invariant says no duplicates add() is incorrect –Otherwise remove() is incorrect

32. © Herlihy-Shavit 200732 Rep Invariant (partly) Sentinel nodes –tail reachable from head Sorted, no duplicates

33. © Herlihy-Shavit 200733 Abstraction Map S(head) = –{ x | there exists a such that A reachable from head and a.object = x –}

34. © Herlihy-Shavit 200734 Sequential List Based Set a c d b a b c Add() Remove()

35. © Herlihy-Shavit 200735 Course Grained Locking a b d c Simple but hotspot + bottleneck

36. © Herlihy-Shavit 200736 Coarse-Grained Locking Easy, same as synchronized methods –“One lock to rule them all …” Simple, clearly correct –Deserves respect! Works poorly with contention –Queue locks help –But bottleneck still an issue

37. © Herlihy-Shavit 200737 Fine-grained Locking Requires careful thought –“Do not meddle in the affairs of wizards, for they are subtle and quick to anger” Split object into pieces –Each piece has own lock –Methods that work on disjoint pieces need not exclude each other

38. © Herlihy-Shavit 200738 Hand-over-Hand locking abc

39. © Herlihy-Shavit 200739 Hand-over-Hand locking abc

40. © Herlihy-Shavit 200740 Hand-over-Hand locking abc

41. © Herlihy-Shavit 200741 Hand-over-Hand locking abc

42. © Herlihy-Shavit 200742 Hand-over-Hand locking abc

43. © Herlihy-Shavit 200743 Removing a Node abcd remov e b

44. © Herlihy-Shavit 200744 Removing a Node abcd remov e b

45. © Herlihy-Shavit 200745 Removing a Node abcd remov e b

46. © Herlihy-Shavit 200746 Removing a Node abcd remov e b

47. © Herlihy-Shavit 200747 Removing a Node acd remov e b

48. © Herlihy-Shavit 200748 Removing a Node abcd remov e b remov e c

49. © Herlihy-Shavit 200749 Removing a Node abcd remov e b remov e c

50. © Herlihy-Shavit 200750 Removing a Node abcd remov e b remov e c

51. © Herlihy-Shavit 200751 Removing a Node abcd remov e b remov e c

52. © Herlihy-Shavit 200752 Removing a Node abcd remov e b remov e c

53. © Herlihy-Shavit 200753 Removing a Node abcd remov e b remov e c

54. © Herlihy-Shavit 200754 Removing a Node abcd remov e b remov e c

55. © Herlihy-Shavit 200755 Removing a Node abcd remov e b remov e c

56. © Herlihy-Shavit 200756 Removing a Node abcd remov e b remov e c

57. © Herlihy-Shavit 200757 Uh, Oh acd remov e b remov e c Still in list

58. © Herlihy-Shavit 200758 Problem To delete node b –Swing node a’s next field to c Problem is, –Someone could delete c concurrently b acbac

59. © Herlihy-Shavit 200759 Insight If a node is locked –No one can delete node’s successor If a thread locks –Node to be deleted –And its predecessor –Then it works

60. © Herlihy-Shavit 200760 Hand-Over-Hand Again abcd remov e b

61. © Herlihy-Shavit 200761 Hand-Over-Hand Again abcd remov e b

62. © Herlihy-Shavit 200762 Hand-Over-Hand Again abcd remov e b

63. © Herlihy-Shavit 200763 Hand-Over-Hand Again abcd remov e b Found it!

64. © Herlihy-Shavit 200764 Hand-Over-Hand Again abcd remov e b Found it!

65. © Herlihy-Shavit 200765 Hand-Over-Hand Again acd remov e b Found it!

66. © Herlihy-Shavit 200766 Removing a Node abcd remov e b remov e c

67. © Herlihy-Shavit 200767 Removing a Node abcd remov e b remov e c

68. © Herlihy-Shavit 200768 Removing a Node abcd remov e b remov e c

69. © Herlihy-Shavit 200769 Removing a Node abcd remov e b remov e c

70. © Herlihy-Shavit 200770 Removing a Node abcd remov e b remov e c

71. © Herlihy-Shavit 200771 Removing a Node abcd remov e b remov e c

72. © Herlihy-Shavit 200772 Removing a Node abcd remov e b remov e c

73. © Herlihy-Shavit 200773 Removing a Node abd remov e b remov e c

74. © Herlihy-Shavit 200774 Removing a Node abd remov e b remov e c

75. © Herlihy-Shavit 200775 Removing a Node ad remov e b remov e c

76. © Herlihy-Shavit 200776 Remove method public boolean remove(Object object) { int key = object.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }}

77. © Herlihy-Shavit 200777 Remove method public boolean remove(Object object) { int key = object.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }} Key used to order node

78. © Herlihy-Shavit 200778 Remove method public boolean remove(Object object) { int key = object.hashCode(); Node pred, curr; try { … } finally { currNode.unlock(); predNode.unlock(); }} Predecessor and current nodes

79. © Herlihy-Shavit 200779 Remove method public boolean remove(Object object) { int key = object.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }} Make sure locks released

80. © Herlihy-Shavit 200780 Remove method public boolean remove(Object object) { int key = object.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }} Everything else

81. © Herlihy-Shavit 200781 Remove method try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); … } finally { … }

82. © Herlihy-Shavit 200782 Remove method try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); … } finally { … } lock previous

83. © Herlihy-Shavit 200783 Remove method try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); … } finally { … } Lock current

84. © Herlihy-Shavit 200784 Remove method try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); … } finally { … } Traversing list

85. © Herlihy-Shavit 200785 Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

86. © Herlihy-Shavit 200786 Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Search key range

87. © Herlihy-Shavit 200787 Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; At start of each loop: curr and pred locked

88. © Herlihy-Shavit 200788 Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; If node found, remove it

89. © Herlihy-Shavit 200789 Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; If node found, remove it

90. © Herlihy-Shavit 200790 Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Unlock predecessor

91. © Herlihy-Shavit 200791 Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Only one node locked!

92. © Herlihy-Shavit 200792 Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; demote current

93. © Herlihy-Shavit 200793 Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = currNode; curr = curr.next; curr.lock(); } return false; Find and lock new current

94. © Herlihy-Shavit 200794 Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = currNode; curr = curr.next; curr.lock(); } return false; Lock invariant restored

95. © Herlihy-Shavit 200795 Remove: searching while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Otherwise, not present

96. © Herlihy-Shavit 200796 Why does this work? To remove node e –Must lock e –Must lock e’s predecessor Therefore, if you lock a node –It can’t be removed –And neither can its successor

97. © Herlihy-Shavit 200797 while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable head ~> pred A --> curr A so the object is in the set

98. © Herlihy-Shavit 200798 while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable Node locked, so no other thread can remove it ….

99. © Herlihy-Shavit 200799 while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable Linearization point

100. © Herlihy-Shavit 2007100 while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable Object not present

101. © Herlihy-Shavit 2007101 while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable pred A --> curr A pred A.key < key key < curr A.key

102. © Herlihy-Shavit 2007102 while (curr.key <= key) { if (object == curr.object) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable Linearization point: when curr A set to node with higher key

103. © Herlihy-Shavit 2007103 Adding Nodes To add node e –Must lock predecessor –Must lock successor Neither can be deleted –(Is successor lock actually required?)

104. © Herlihy-Shavit 2007104 Same Abstraction Map S(head) = –{ x | there exists a such that A reachable from head and a.object = x –}

105. © Herlihy-Shavit 2007105 Rep Invariant Easy to check that –Tail always reachable from head –Nodes sorted, no duplicates

106. © Herlihy-Shavit 2007106 Drawbacks Better than coarse-grained lock –Threads can traverse in parallel Still not ideal –Long chain of acquire/release –Inefficient

107. © Herlihy-Shavit 2007107 Optimistic Synchronization Find nodes without locking Lock nodes Check that everything is OK

108. © Herlihy-Shavit 2007108 Optimistic Fine Grained b d e To insert C: Optimistically traverse list to find b Lock b.curr then lock b.succ Re-Traverse list to find b and verify pred  curr Perform insert and release locks a 1 3 3 2 2 c 4

109. © Herlihy-Shavit 2007109 Same Abstraction Map S(head) = –{ x | there exists a such that A reachable from head and a.object = x –}

110. © Herlihy-Shavit 2007110 Preserving Invariants Not same as before as we traverse deleted elements But we establish properties by –Validation –After we lock target nodes

111. © Herlihy-Shavit 2007111 Scanning Deleted Items abde remov e e A Pred and curr could point to removed nodes

112. © Herlihy-Shavit 2007112 Removing a Node abcd remov e c return true

113. © Herlihy-Shavit 2007113 What Can Go Wrong? abcd remov e c remov e b

114. © Herlihy-Shavit 2007114 Check that Node is Still Accessible abcd remov e c

115. © Herlihy-Shavit 2007115 What Can Go Wrong? abcd remov e c Add b’ b’

116. © Herlihy-Shavit 2007116 What Can Go Wrong? abcd remov e c b’

117. © Herlihy-Shavit 2007117 Check that Nodes Still Adjacent abcd remov e c

118. © Herlihy-Shavit 2007118 Correctness If –Nodes b and c both locked –Node b still accessible –Node c still successor to b Then –Neither will be deleted –OK to delete and return true

119. © Herlihy-Shavit 2007119 Removing an Absent Node abde remov e c return false

120. © Herlihy-Shavit 2007120 Correctness If –Nodes b and d both locked –Node b still accessible –Node d still successor to b Then –Neither will be deleted –No thread can add c after b –OK to return false

121. © Herlihy-Shavit 2007121 Validation private boolean validate(Node pred, Node curry) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; }

122. © Herlihy-Shavit 2007122 private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Predecessor & current nodes

123. © Herlihy-Shavit 2007123 private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Start at the beginning

124. © Herlihy-Shavit 2007124 private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Search range of keys

125. © Herlihy-Shavit 2007125 private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Predecessor reachable

126. © Herlihy-Shavit 2007126 private boolean validate(Node pred, Node curry) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Is current node next?

127. © Herlihy-Shavit 2007127 private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Otherwise move on

128. © Herlihy-Shavit 2007128 private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Predecessor not reachable

129. © Herlihy-Shavit 2007129 Remove: searching public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } …

130. © Herlihy-Shavit 2007130 public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } … Remove: searching Search key

131. © Herlihy-Shavit 2007131 public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } … Remove: searching Retry on synchronization conflict

132. © Herlihy-Shavit 2007132 public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } … Remove: searching Examine predecessor and current nodes

133. © Herlihy-Shavit 2007133 public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } … Remove: searching Search by key

134. © Herlihy-Shavit 2007134 public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } … Remove: searching Stop if we find object

135. © Herlihy-Shavit 2007135 public boolean remove(Object object) { int key = object.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (object == curr.object) break; pred = curr; curr = curr.next; } … Remove: searching Move along

136. © Herlihy-Shavit 2007136 On Exit from Loop If object is present –curr holds object –pred just before curr If object is absent –curr has first higher key –pred just before curr Assuming no synchronization problems

137. © Herlihy-Shavit 2007137 Remove Method try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}}

138. © Herlihy-Shavit 2007138 try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Remove Method Always unlock

139. © Herlihy-Shavit 2007139 try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Remove Method Lock both nodes

140. © Herlihy-Shavit 2007140 try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Remove Method Check for synchronization conflicts

141. © Herlihy-Shavit 2007141 try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Remove Method Object found, remove node

142. © Herlihy-Shavit 2007142 try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.object == object) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Remove Method Object not found

143. © Herlihy-Shavit 2007143 Summary: Wait-free Traversals b c d a 1.Traverse removed nodes 2.Must have non-interference 3.Natural in GC languages like Java™ Reachable = in the set

144. © Herlihy-Shavit 2007144 Summary Optimistic List b c e a 1.Limited Hotspots (Only at locked Add(), Remove(), Find() destination locations, not traversals) 2.But two traversals 3.Yet traversals are wait-free!

145. © Herlihy-Shavit 2007145 So Far, So Good Much less lock acquisition/release –Performance –Concurrency Problems –Need to traverse list twice –contains() method acquires locks Most common method call

146. © Herlihy-Shavit 2007146 Evaluation Optimistic works if –cost of scanning twice without locks < –cost of Scanning once with locks Drawback –Contains() acquires locks –90% of calls in many apps

147. © Herlihy-Shavit 2007147 Lazy List Like optimistic, except –Scan once –Contains() never locks … Key insight –Removing nodes causes trouble –Do it “lazily”

148. © Herlihy-Shavit 2007148 Lazy List Remove Method –Scans list (as before) –Locks predecessor & current (as before) Logical delete –Marks current node as removed (new!) Physical delete –Redirects predecessor’s next (as before)

149. © Herlihy-Shavit 2007149 Lazy Removal a 0 0 0 a b c 0 d 1 c Logical Removal = Set Mark Bit Physical Removal Add a separate mark bit field to each node: 1. mark node as deleted 2. redirect pointer

150. © Herlihy-Shavit 2007150 Lazy Removal a 0 0 0 a b c 0 d 1 3 2 2 1 c 4 Logical Removal Physical Removal Release Locks 1.Optimistically traverse list to find b 2.Lock b.curr then lock b.succ 3.verify pred  curr 4.Perform logical then physical removal 5.Release locks

151. © Herlihy-Shavit 2007151 Lazy List All Methods –Scan through locked and marked nodes –Removing a node doesn’t slow down other method calls … Must still lock pred and curr nodes.

152. © Herlihy-Shavit 2007152 Validation No need to rescan list! Check that pred is not marked Check that curr is not marked Check that pred points to curr

153. © Herlihy-Shavit 2007153 Business as Usual abcd

154. © Herlihy-Shavit 2007154 Business as Usual abcd

155. © Herlihy-Shavit 2007155 Business as Usual abcd

156. © Herlihy-Shavit 2007156 Business as Usual abcd

157. © Herlihy-Shavit 2007157 Business as Usual abcd

158. © Herlihy-Shavit 2007158 Interference abcd

159. © Herlihy-Shavit 2007159 Interference abcd

160. © Herlihy-Shavit 2007160 Interference abcd

161. © Herlihy-Shavit 2007161 Interference abcd Remove c

162. © Herlihy-Shavit 2007162 Validation abcd b not marked

163. © Herlihy-Shavit 2007163 Interference abcd c not marked

164. © Herlihy-Shavit 2007164 Interference abcd b still points to c

165. © Herlihy-Shavit 2007165 Logical Delete abcd mark

166. © Herlihy-Shavit 2007166 Scan Through abcd So what? Remove c

167. © Herlihy-Shavit 2007167 Physical Deletion abcd So what? Remove c

168. © Herlihy-Shavit 2007168 New Abstraction Map S(head) = –{ x | there exists node a such that A reachable from head and a.object = x and a is unmarked –}

169. © Herlihy-Shavit 2007169 Invariant If not marked then item in the set and reachable from head and if not yet traversed it is reachable from pred

170. © Herlihy-Shavit 2007170 Validation private boolean validate(Node pred, Node curr) { return !pred.next.marked && !curr.next.marked && pred.next == curr); }

171. © Herlihy-Shavit 2007171 private boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); } List Validate Method Predecessor not Logically removed

172. © Herlihy-Shavit 2007172 private boolean validate(Node pred, Node curr) { return !pred.next.marked && !curr.next.marked && pred.next == curr); } List Validate Method Current not Logically removed

173. © Herlihy-Shavit 2007173 private boolean validate(Node pred, Node curr) { return !pred.next.marked && !curr.next.marked && pred.next == curr); } List Validate Method Predecessor still Points to current

174. © Herlihy-Shavit 2007174 Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}}

175. © Herlihy-Shavit 2007175 Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Validate as before

176. © Herlihy-Shavit 2007176 Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Key found

177. © Herlihy-Shavit 2007177 Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Logical remove

178. © Herlihy-Shavit 2007178 Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} physical remove

179. © Herlihy-Shavit 2007179 Contains public boolean contains(Object object) { int key = object.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; }

180. © Herlihy-Shavit 2007180 Contains public boolean contains(Object object) { int key = object.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Start at the head

181. © Herlihy-Shavit 2007181 Contains public boolean contains(Object object) { int key = object.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Search key range

182. © Herlihy-Shavit 2007182 Contains public boolean contains(Object object) { int key = object.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Traverse without locking (nodes may have been removed)

183. © Herlihy-Shavit 2007183 Contains public boolean contains(Object object) { int key = object.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Present and undeleted?

184. © Herlihy-Shavit 2007184 Summary: Wait-free Contains a 0 0 0 a b c 0 e 1 d Use Mark bit + Fact that List is ordered 1.Not marked  in the set 2.Marked or missing  not in the set

185. © Herlihy-Shavit 2007185 Lazy List a 0 0 0 a b c 0 e 1 d Lazy Add() and Remove() + Wait-free Contains()

186. © Herlihy-Shavit 2007186 Evaluation Good: –Contains method doesn’t need to lock –In fact, its wait-free! –Good because typically high % contains() –Uncontended calls don’t re-traverse Bad –Contended calls do re-traverse –Traffic jam if one thread delays

187. © Herlihy-Shavit 2007187 Traffic Jam Any concurrent data structure based on mutual exclusion has a weakness If one thread –Enters critical section –And “eats the big muffin” (stops running) Cache miss, page fault, descheduled … –Everyone else using that lock is stuck!

188. © Herlihy-Shavit 2007188 Reminder: Lock-Free Data Structures No matter what … –Some thread will complete method call –Even if others halt at malicious times –Weaker than wait-free, yet Implies that –You can’t use locks –Um, that’s why they call it lock-free

189. © Herlihy-Shavit 2007189 Lock-free Lists Next logical step Eliminate locking entirely Contains() wait-free and Add() and Remove() lock-free Use only compareAndSet() What could go wrong?

190. © Herlihy-Shavit 2007190 Adding a Node abc

191. © Herlihy-Shavit 2007191 Adding a Node abcb

192. © Herlihy-Shavit 2007192 Adding a Node abcb CAS

193. © Herlihy-Shavit 2007193 Adding a Node abcb

194. © Herlihy-Shavit 2007194 Adding a Node abcb

195. © Herlihy-Shavit 2007195 Removing a Node abcd remov e b remov e c CAS

196. © Herlihy-Shavit 2007196 Look Familiar? abcd remov e b remov e c Still in list

197. © Herlihy-Shavit 2007197 Problem Method updates node’s next field After node has been removed

198. © Herlihy-Shavit 2007198 Solution Use AtomicMarkableReference Atomically –Swing reference and –Update flag Remove in two steps –Set mark bit in next field –Redirect predecessor’s pointer

199. © Herlihy-Shavit 2007199 Marking a Node AtomicMarkableReference class –Java.util.concurrent.atomic package address F mark bit Reference

200. © Herlihy-Shavit 2007200 Extracting Reference & Mark Public T get(boolean[] marked);

201. © Herlihy-Shavit 2007201 Extracting Reference & Mark Public T get(boolean[] marked); Returns reference to object of type T Returns mark at array index 0!

202. © Herlihy-Shavit 2007202 Changing State Public boolean compareAndSet( T expectedRef, T updateRef, boolean expectedMark, boolean updateMark);

203. © Herlihy-Shavit 2007203 Changing State Public boolean compareAndSet( T expectedRef, T updateRef, boolean expectedMark, boolean updateMark); If this is the current reference … And this is the current mark …

204. © Herlihy-Shavit 2007204 Changing State Public boolean compareAndSet( T expectedRef, T updateRef, boolean expectedMark, boolean updateMark); …then change to this new reference … … and this new mark

205. © Herlihy-Shavit 2007205 Changing State public boolean attemptMark( T expectedRef, boolean updateMark);

206. © Herlihy-Shavit 2007206 Changing State public boolean attemptMark( T expectedRef, boolean updateMark); If this is the current reference …

207. © Herlihy-Shavit 2007207 Changing State public boolean attemptMark( T expectedRef, boolean updateMark);.. then change to this new mark.

208. © Herlihy-Shavit 2007208 Removing a Node abcd remov e c CAS

209. © Herlihy-Shavit 2007209 Removing a Node abd remov e b remov e c c CAS failed

210. © Herlihy-Shavit 2007210 Removing a Node abd remov e b remov e c c

211. © Herlihy-Shavit 2007211 Removing a Node ad remov e b remov e c

212. © Herlihy-Shavit 2007212 Traversing the List Q: what do you do when you find a “logically” deleted node in your path? A: finish the job. –CAS the predecessor’s next field –Proceed (repeat as needed)

213. © Herlihy-Shavit 2007213 Lock-Free Traversal abcd CAS Uh-oh

214. © Herlihy-Shavit 2007214 The Window Class class Window { public Node pred; public Node curr; Window(Node pred, Node curr) { this.pred = pred; this.curr = curr; }

215. © Herlihy-Shavit 2007215 The Window Class class Window { public Node pred; public Node curr; Window(Node pred, Node curr) { this.pred = pred; this.curr = curr; } A container for pred and current values

216. © Herlihy-Shavit 2007216 Using the Find Method Window window = find(head, key); Node pred = window.pred; curr = window.curr;

217. © Herlihy-Shavit 2007217 Using the Find Method Window window = find(head, key); Node pred = window.pred; curr = window.curr; Find returns window

218. © Herlihy-Shavit 2007218 Using the Find Method Window window = find(head, key); Node pred = window.pred; curr = window.curr; Extract pred and curr

219. © Herlihy-Shavit 2007219 The Find Method Window window = find(head, key); At some instant, predcurrsucc object or …

220. © Herlihy-Shavit 2007220 The Find Method Window window = find(head,key); At some instant, pred curr= succ succ object not in list

221. © Herlihy-Shavit 2007221 Remove public boolean remove(T object) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}}

222. © Herlihy-Shavit 2007222 Remove public boolean remove(T object) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} Keep trying

223. © Herlihy-Shavit 2007223 Remove public boolean remove(T object) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} Find neighbors

224. © Herlihy-Shavit 2007224 Remove public boolean remove(T object) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} She’s not there …

225. © Herlihy-Shavit 2007225 Remove public boolean remove(T object) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} Try to mark node as deleted

226. © Herlihy-Shavit 2007226 Remove public boolean remove(T object) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} If it doesn’t work, just retry, if it does, job essentially done

227. © Herlihy-Shavit 2007227 Remove public boolean remove(T object) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} Try to advance reference (if we don’t succeed, someone else did or will). a

228. © Herlihy-Shavit 2007228 Add public boolean add(T object) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(object); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true; }}}}

229. © Herlihy-Shavit 2007229 Add public boolean add(T object) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(object); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true; }}}} Object already there.

230. © Herlihy-Shavit 2007230 Add public boolean add(T object) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(object); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true; }}}} create new node

231. © Herlihy-Shavit 2007231 Add public boolean add(T object) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(object); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true }}}} Install new node, else retry loop

232. © Herlihy-Shavit 2007232 Wait-free Contains public boolean contains(T object) { boolean marked; int key = object.hashCode(); Node curr = this.head; while (curr.key < key) curr = curr.next; Node succ = curr.next.get(marked); return (curr.key == key && !marked[0]) }

233. © Herlihy-Shavit 2007233 Wait-free Contains public boolean contains(T object) { boolean marked; int key = object.hashCode(); Node curr = this.head; while (curr.key < key) curr = curr.next; Node succ = curr.next.get(marked); return (curr.key == key && !marked[0]) } Only diff from lazy list is that we get and check marked

234. © Herlihy-Shavit 2007234 Lock-free Find public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }}

235. © Herlihy-Shavit 2007235 Lock-free Find public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} If list changes while traversed, start over Lock-Free because we start over only if someone else makes progress

236. © Herlihy-Shavit 2007236 public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find Start looking from head

237. © Herlihy-Shavit 2007237 public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find Move down the list

238. © Herlihy-Shavit 2007238 public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find Get ref to successor and current deleted bit

239. © Herlihy-Shavit 2007239 public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find Try to remove deleted nodes in path…code details soon

240. © Herlihy-Shavit 2007240 public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find If found curr key that is greater or equal, return pred and curr

241. © Herlihy-Shavit 2007241 public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find Otherwise advance “window” and loop again

242. © Herlihy-Shavit 2007242 Lock-free Find retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); } …

243. © Herlihy-Shavit 2007243 Lock-free Find retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); } … Try to snip out node

244. © Herlihy-Shavit 2007244 Lock-free Find retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); } … if predecessor’s next field changed must retry whole traversal

245. © Herlihy-Shavit 2007245 Lock-free Find retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); } … Otherwise move on to check if next node deleted

246. © Herlihy-Shavit 2007246 Summary: Summary: Lock- free Removal a 0 0 0 a b c 0 e 1 c Logical Removal = Set Mark Bit Physical Removal CAS pointer Use CAS to verify pointer is correct Not enough!

247. © Herlihy-Shavit 2007247 Summary: Summary: Lock-free Removal a 0 0 0 a b c 0 e 1 c Logical Removal = Set Mark Bit Physical Removal CAS 0 d Problem: d not added to list… Must Prevent manipulation of removed node’s pointer Node added Before Physical Removal CAS

248. © Herlihy-Shavit 2007248 Our Solution: Combine Bit and Pointer a 0 0 0 a b c 0 e 1 c Logical Removal = Set Mark Bit Physical Removal CAS 0 d Mark-Bit and Pointer are CASed together Fail CAS: Node not added after logical Removal

249. © Herlihy-Shavit 2007249 A Lock-free Algorithm a 0 0 0 a b c 0 e 1 c 1. Add() and Remove() physically clean up (remove) marked nodes 2. Wait-free find() traverses both marked and removed nodes

250. © Herlihy-Shavit 2007250 Performance On 16 node shared memory machine Benchmark throughput of Java List-based Set algs. Vary % of Contains() method Calls.

251. © Herlihy-Shavit 2007251 High Contains Ratio Lock-free Lazy list Course Grained Fine Lock-coupling

252. © Herlihy-Shavit 2007252 Low Contains Ratio

253. © Herlihy-Shavit 2007253 As Contains Ratio Increases % Contains()

254. © Herlihy-Shavit 2007254 Summary Coarse-grained locking Fine-grained locking Optimistic synchronization Lock-free synchronization

255. © Herlihy-Shavit 2007255 “To Lock or Not to Lock” Locking vs. Non-blocking: Extremist views on both sides The answer: nobler to compromise, combine locking and non-blocking –Example: Lazy list combines blocking add() and remove() and a wait-free contains() –Blocking/non-blocking is a property of a method