1. Linked Lists: Locking, Lock- Free, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit Modified by Rajeev Alur for CIS640 University of Pennsylvania

2. Art of Multiprocessor Programming2 Today: Concurrent Objects Adding threads should not lower throughput –Contention effects –Mostly fixed by Queue locks Should increase throughput –Not possible if inherently sequential –Surprising things are parallelizable

3. Art of Multiprocessor Programming3 Coarse-Grained Synchronization Each method locks the object –Avoid contention using queue locks

4. Art of Multiprocessor Programming4 Coarse-Grained Synchronization Each method locks the object –Avoid contention using queue locks –Easy to reason about In simple cases

5. Art of Multiprocessor Programming5 Coarse-Grained Synchronization Each method locks the object –Avoid contention using queue locks –Easy to reason about In simple cases So, are we done?

6. Art of Multiprocessor Programming6 Coarse-Grained Synchronization Sequential bottleneck –Threads “stand in line”

7. Art of Multiprocessor Programming7 Coarse-Grained Synchronization Sequential bottleneck –Threads “stand in line” Adding more threads –Does not improve throughput –Struggle to keep it from getting worse

8. Art of Multiprocessor Programming8 Coarse-Grained Synchronization Sequential bottleneck –Threads “stand in line” Adding more threads –Does not improve throughput –Struggle to keep it from getting worse So why even use a multiprocessor? –Well, some apps inherently parallel …

9. Art of Multiprocessor Programming9 This Lecture Introduce four “patterns” –Bag of tricks … –Methods that work more than once …

10. Art of Multiprocessor Programming10 This Lecture Introduce four “patterns” –Bag of tricks … –Methods that work more than once … For highly-concurrent objects –Concurrent access –More threads, more throughput

11. Art of Multiprocessor Programming11 First: Fine-Grained Synchronization Instead of using a single lock … Split object into –Independently-synchronized components Methods conflict when they access –The same component … –At the same time

12. Art of Multiprocessor Programming12 Second: Optimistic Synchronization Search without locking … If you find it, lock and check … –OK: we are done –Oops: start over Evaluation –Usually cheaper than locking, but –Mistakes are expensive

13. Art of Multiprocessor Programming13 Third: Lazy Synchronization Postpone hard work Removing components is tricky –Logical removal Mark component to be deleted –Physical removal Do what needs to be done

14. Art of Multiprocessor Programming14 Fourth: Lock-Free Synchronization Don’t use locks at all –Use compareAndSet() & relatives … Advantages –No Scheduler Assumptions/Support Disadvantages –Complex –Sometimes high overhead

15. Art of Multiprocessor Programming15 Linked List Illustrate these patterns … Using a list-based Set –Common application –Building block for other apps

16. Art of Multiprocessor Programming16 Set Interface Unordered collection of items No duplicates Methods –add(x) put x in set –remove(x) take x out of set –contains(x) tests if x in set

17. Art of Multiprocessor Programming17 List-Based Sets public interface Set { public boolean add(T x); public boolean remove(T x); public boolean contains(T x); }

18. Art of Multiprocessor Programming18 List-Based Sets public interface Set { public boolean add(T x); public boolean remove(T x); public boolean contains(T x); } Add item to set

19. Art of Multiprocessor Programming19 List-Based Sets public interface Set { public boolean add(T x); public boolean remove(T x); public boolean contains(Tt x); } Remove item from set

20. Art of Multiprocessor Programming20 List-Based Sets public interface Set { public boolean add(T x); public boolean remove(T x); public boolean contains(T x); } Is item in set?

21. Art of Multiprocessor Programming21 List Node public class Node { public T item; public int key; public Node next; }

22. Art of Multiprocessor Programming22 List Node public class Node { public T item; public int key; public Node next; } item of interest

23. Art of Multiprocessor Programming23 List Node public class Node { public T item; public int key; public Node next; } Usually hash code

24. Art of Multiprocessor Programming24 List Node public class Node { public T item; public int key; public Node next; } Reference to next node

25. Art of Multiprocessor Programming25 The List-Based Set abc Sorted with Sentinel nodes (min & max possible keys) -∞ +∞

26. Art of Multiprocessor Programming26 Invariants Sentinel nodes –tail reachable from head Sorted No duplicates

27. Art of Multiprocessor Programming27 Sequential List Based Set a c d a b c Add() Remove()

28. Art of Multiprocessor Programming28 Sequential List Based Set a c d b a b c Add() Remove()

29. Art of Multiprocessor Programming29 Coarse Grained Locking a b d

30. Art of Multiprocessor Programming30 Coarse Grained Locking a b d c

31. Art of Multiprocessor Programming31 honk! Coarse Grained Locking a b d c Simple but hotspot + bottleneck honk!

32. Art of Multiprocessor Programming32 Coarse-Grained Locking Easy, same as synchronized methods –“One lock to rule them all …”

33. Art of Multiprocessor Programming33 Coarse-Grained Locking Easy, same as synchronized methods –“One lock to rule them all …” Simple, clearly correct –Deserves respect! Works poorly with contention –Queue locks help –But bottleneck still an issue

34. Art of Multiprocessor Programming34 Fine-grained Locking Requires careful thought Split object into pieces –Each piece has own lock –Methods that work on disjoint pieces need not exclude each other

35. Art of Multiprocessor Programming35 Hand-over-Hand locking abc

36. Art of Multiprocessor Programming36 Hand-over-Hand locking abc

37. Art of Multiprocessor Programming37 Hand-over-Hand locking abc

38. Art of Multiprocessor Programming38 Hand-over-Hand locking abc

39. Art of Multiprocessor Programming39 Hand-over-Hand locking abc

40. Art of Multiprocessor Programming40 Removing a Node abcd remove(b)

41. Art of Multiprocessor Programming41 Removing a Node abcd remove(b)

42. Art of Multiprocessor Programming42 Removing a Node abcd remove(b)

43. Art of Multiprocessor Programming43 Removing a Node abcd remove(b)

44. Art of Multiprocessor Programming44 Removing a Node abcd remove(b)

45. Art of Multiprocessor Programming45 Removing a Node acd remove(b) Why hold 2 locks?

46. Art of Multiprocessor Programming46 Concurrent Removes abcd remove(c) remove(b)

47. Art of Multiprocessor Programming47 Concurrent Removes abcd remove(b) remove(c)

48. Art of Multiprocessor Programming48 Concurrent Removes abcd remove(b) remove(c)

49. Art of Multiprocessor Programming49 Concurrent Removes abcd remove(b) remove(c)

50. Art of Multiprocessor Programming50 Concurrent Removes abcd remove(b) remove(c)

51. Art of Multiprocessor Programming51 Concurrent Removes abcd remove(b) remove(c)

52. Art of Multiprocessor Programming52 Concurrent Removes abcd remove(b) remove(c)

53. Art of Multiprocessor Programming53 Concurrent Removes abcd remove(b) remove(c)

54. Art of Multiprocessor Programming54 Concurrent Removes abcd remove(b) remove(c)

55. Art of Multiprocessor Programming55 Concurrent Removes abcd remove(b) remove(c)

56. Art of Multiprocessor Programming56 Uh, Oh acd remove(b) remove(c)

57. Art of Multiprocessor Programming57 Uh, Oh acd Bad news, c not removed remove(b) remove(c)

58. Art of Multiprocessor Programming58 Problem To delete node c –Swing node b’s next field to d Problem is, –Someone deleting b concurrently could direct a pointer to c ba cbac

59. Art of Multiprocessor Programming59 Insight If a node is locked –No one can delete node’s successor If a thread locks –Node to be deleted –And its predecessor –Then it works

60. Art of Multiprocessor Programming60 Hand-Over-Hand Again abcd remove(b)

61. Art of Multiprocessor Programming61 Hand-Over-Hand Again abcd remove(b)

62. Art of Multiprocessor Programming62 Hand-Over-Hand Again abcd remove(b)

63. Art of Multiprocessor Programming63 Hand-Over-Hand Again abcd remove(b) Found it!

64. Art of Multiprocessor Programming64 Hand-Over-Hand Again abcd remove(b) Found it!

65. Art of Multiprocessor Programming65 Hand-Over-Hand Again acd remove(b)

66. Art of Multiprocessor Programming66 Removing a Node abcd remove(b) remove(c)

67. Art of Multiprocessor Programming67 Removing a Node abcd remove(b) remove(c)

68. Art of Multiprocessor Programming68 Removing a Node abcd remove(b) remove(c)

69. Art of Multiprocessor Programming69 Removing a Node abcd remove(b) remove(c)

70. Art of Multiprocessor Programming70 Removing a Node abcd remove(b) remove(c)

71. Art of Multiprocessor Programming71 Removing a Node abcd remove(b) remove(c)

72. Art of Multiprocessor Programming72 Removing a Node abcd remove(b) remove(c)

73. Art of Multiprocessor Programming73 Removing a Node abcd remove(b) remove(c)

74. Art of Multiprocessor Programming74 Removing a Node abcd Must acquire Lock of b remove(c)

75. Art of Multiprocessor Programming75 Removing a Node abcd Cannot acquire lock of b remove(c)

76. Art of Multiprocessor Programming76 Removing a Node abcd Wait! remove(c)

77. Art of Multiprocessor Programming77 Removing a Node abd Proceed to remove(b)

78. Art of Multiprocessor Programming78 Removing a Node abd remove(b)

79. Art of Multiprocessor Programming79 Removing a Node abd remove(b)

80. Art of Multiprocessor Programming80 Removing a Node ad remove(b)

81. Art of Multiprocessor Programming81 Removing a Node ad

82. Art of Multiprocessor Programming82 Remove method public boolean remove(Item item) { int key = item.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }}

83. Art of Multiprocessor Programming83 Remove method public boolean remove(Item item) { int key = item.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }} Key used to order node

84. Art of Multiprocessor Programming84 Remove method public boolean remove(Item item) { int key = item.hashCode(); Node pred, curr; try { … } finally { currNode.unlock(); predNode.unlock(); }} Predecessor and current nodes

85. Art of Multiprocessor Programming85 Remove method public boolean remove(Item item) { int key = item.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }} Make sure locks released

86. Art of Multiprocessor Programming86 Remove method public boolean remove(Item item) { int key = item.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }} Everything else

87. Art of Multiprocessor Programming87 Remove method try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); … } finally { … }

88. Art of Multiprocessor Programming88 Remove method try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); … } finally { … } lock pred == head

89. Art of Multiprocessor Programming89 Remove method try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); … } finally { … } Lock current

90. Art of Multiprocessor Programming90 Remove method try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); … } finally { … } Traversing list

91. Art of Multiprocessor Programming91 Remove: searching while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false;

92. Art of Multiprocessor Programming92 Remove: searching while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Search key range

93. Art of Multiprocessor Programming93 Remove: searching while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; At start of each loop: curr and pred locked

94. Art of Multiprocessor Programming94 Remove: searching while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; If item found, remove node

95. Art of Multiprocessor Programming95 Remove: searching while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; If node found, remove it

96. Art of Multiprocessor Programming96 Remove: searching while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Unlock predecessor

97. Art of Multiprocessor Programming97 Remove: searching while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Only one node locked!

98. Art of Multiprocessor Programming98 Remove: searching while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; demote current

99. Art of Multiprocessor Programming99 Remove: searching while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = currNode; curr = curr.next; curr.lock(); } return false; Find and lock new current

100. Art of Multiprocessor Programming100 Remove: searching while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = currNode; curr = curr.next; curr.lock(); } return false; Lock invariant restored

101. Art of Multiprocessor Programming101 Remove: searching while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Otherwise, not present

102. Art of Multiprocessor Programming102 Why does this work? To remove node e –Must lock e –Must lock e’s predecessor Therefore, if you lock a node –It can’t be removed –And neither can its successor

103. Art of Multiprocessor Programming103 while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable pred reachable from head curr is pred.next So curr.item is in the set

104. Art of Multiprocessor Programming104 while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable Linearization point if item is present

105. Art of Multiprocessor Programming105 while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable Node locked, so no other thread can remove it ….

106. Art of Multiprocessor Programming106 while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable Item not present

107. Art of Multiprocessor Programming107 while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable pred reachable from head curr is pred.next pred.key < key key < curr.key

108. Art of Multiprocessor Programming108 while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Why remove() is linearizable Linearization point

109. Art of Multiprocessor Programming109 Adding Nodes To add node e –Must lock predecessor –Must lock successor Neither can be deleted –(Is successor lock actually required?)

110. Art of Multiprocessor Programming110 Drawbacks Better than coarse-grained lock –Threads can traverse in parallel Still not ideal –Long chain of acquire/release –Inefficient

111. Art of Multiprocessor Programming111 Optimistic Synchronization Find nodes without locking Lock nodes Check that everything is OK

112. Art of Multiprocessor Programming112 Optimistic: Traverse without Locking b d e a add(c) Aha!

113. Art of Multiprocessor Programming113 Optimistic: Lock and Load b d e a add(c)

114. Art of Multiprocessor Programming114 Optimistic: Lock and Load b d e a add(c) c

115. Art of Multiprocessor Programming115 What could go wrong? b d e a add(c) Aha!

116. Art of Multiprocessor Programming116 What could go wrong? b d e a add(c)

117. Art of Multiprocessor Programming117 What could go wrong? b d e a remove(b )

118. Art of Multiprocessor Programming118 What could go wrong? b d e a remove(b )

119. Art of Multiprocessor Programming119 What could go wrong? b d e a add(c)

120. Art of Multiprocessor Programming120 What could go wrong? b d e a add(c) c

121. Art of Multiprocessor Programming121 What could go wrong? d e a add(c) Uh-oh

122. Art of Multiprocessor Programming122 Validate – Part 1 b d e a add(c) Yes, b still reachable from head

123. Art of Multiprocessor Programming123 What Else Could Go Wrong? b d e a add(c) Aha!

124. Art of Multiprocessor Programming124 What Else Coould Go Wrong? b d e a add(c) add(b’)

125. Art of Multiprocessor Programming125 What Else Coould Go Wrong? b d e a add(c) add(b’) b’

126. Art of Multiprocessor Programming126 What Else Could Go Wrong? b d e a add(c) b’

127. Art of Multiprocessor Programming127 What Else Could Go Wrong? b d e a add(c) c

128. Art of Multiprocessor Programming128 Validate Part 2 (while holding locks) b d e a add(c) Yes, b still points to d

129. Art of Multiprocessor Programming129 Optimistic: Linearization Point b d e a add(c) c

130. Art of Multiprocessor Programming130 Correctness If –Nodes b and c both locked –Node b still accessible –Node c still successor to b Then –Neither will be deleted –OK to delete and return true

131. Art of Multiprocessor Programming131 Unsuccessful Remove abde remove(c) Aha!

132. Art of Multiprocessor Programming132 Validate (1) abde Yes, b still reachable from head remove(c)

133. Art of Multiprocessor Programming133 Validate (2) abde remove(c) Yes, b still points to d

134. Art of Multiprocessor Programming134 OK Computer abde remove(c) return false

135. Art of Multiprocessor Programming135 Correctness If –Nodes b and d both locked –Node b still accessible –Node d still successor to b Then –Neither will be deleted –No thread can add c after b –OK to return false

136. Art of Multiprocessor Programming136 Validation private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; }

137. Art of Multiprocessor Programming137 private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Predecessor & current nodes

138. Art of Multiprocessor Programming138 private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Begin at the beginning

139. Art of Multiprocessor Programming139 private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Search range of keys

140. Art of Multiprocessor Programming140 private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Predecessor reachable

141. Art of Multiprocessor Programming141 private boolean validate(Node pred, Node curry) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Is current node next?

142. Art of Multiprocessor Programming142 private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Otherwise move on

143. Art of Multiprocessor Programming143 private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Validation Predecessor not reachable

144. Art of Multiprocessor Programming144 Remove: searching public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } …

145. Art of Multiprocessor Programming145 public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } … Remove: searching Search key

146. Art of Multiprocessor Programming146 public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } … Remove: searching Retry on synchronization conflict

147. Art of Multiprocessor Programming147 public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } … Remove: searching Examine predecessor and current nodes

148. Art of Multiprocessor Programming148 public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } … Remove: searching Search by key

149. Art of Multiprocessor Programming149 public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } … Remove: searching Stop if we find item

150. Art of Multiprocessor Programming150 public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } … Remove: searching Move along

151. Art of Multiprocessor Programming151 On Exit from Loop If item is present –curr holds item –pred just before curr If item is absent –curr has first higher key –pred just before curr Assuming no synchronization problems

152. Art of Multiprocessor Programming152 Remove Method try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}}

153. Art of Multiprocessor Programming153 try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Remove Method Always unlock

154. Art of Multiprocessor Programming154 try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Remove Method Lock both nodes

155. Art of Multiprocessor Programming155 try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Remove Method Check for synchronization conflicts

156. Art of Multiprocessor Programming156 try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Remove Method target found, remove node

157. Art of Multiprocessor Programming157 try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Remove Method target not found

158. Art of Multiprocessor Programming158 Optimistic List Limited hot-spots –Targets of add(), remove(), contains() –No contention on traversals Moreover –Traversals are wait-free –Food for thought …

159. Art of Multiprocessor Programming159 So Far, So Good Much less lock acquisition/release –Performance –Concurrency Problems –Need to traverse list twice –contains() method acquires locks

160. Art of Multiprocessor Programming160 Evaluation Optimistic is effective if –cost of scanning twice without locks is less than –cost of scanning once with locks Drawback –contains() acquires locks –90% of calls in many apps

161. Art of Multiprocessor Programming161 Lazy List Like optimistic, except –Scan once –contains(x) never locks … Key insight –Removing nodes causes trouble –Do it “lazily”

162. Art of Multiprocessor Programming162 Lazy List remove() –Scans list (as before) –Locks predecessor & current (as before) Logical delete –Marks current node as removed (new!) Physical delete –Redirects predecessor’s next (as before)

163. Art of Multiprocessor Programming163 Lazy Removal aa b c d

164. c Art of Multiprocessor Programming164 Lazy Removal aa b d Present in list

165. c Art of Multiprocessor Programming165 Lazy Removal aa b d Logically deleted

166. Art of Multiprocessor Programming166 Lazy Removal aa b c d Physically deleted

167. Art of Multiprocessor Programming167 Lazy Removal aa b d Physically deleted

168. Art of Multiprocessor Programming168 Lazy List All Methods –Scan through locked and marked nodes –Removing a node doesn’t slow down other method calls … Must still lock pred and curr nodes.

169. Art of Multiprocessor Programming169 Validation No need to rescan list! Check that pred is not marked Check that curr is not marked Check that pred points to curr

170. Art of Multiprocessor Programming170 Business as Usual abc

171. Art of Multiprocessor Programming171 Business as Usual abc

172. Art of Multiprocessor Programming172 Business as Usual abc

173. Art of Multiprocessor Programming173 Business as Usual abc remove(b)

174. Art of Multiprocessor Programming174 Business as Usual abc a not marked

175. Art of Multiprocessor Programming175 Business as Usual abc a still points to b

176. Art of Multiprocessor Programming176 Business as Usual a bc Logical delete

177. Art of Multiprocessor Programming177 Business as Usual a bc physical delete

178. Art of Multiprocessor Programming178 Business as Usual a bc

179. Art of Multiprocessor Programming179 Validation private boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); }

180. Art of Multiprocessor Programming180 private boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); } List Validate Method Predecessor not Logically removed

181. Art of Multiprocessor Programming181 private boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); } List Validate Method Current not Logically removed

182. Art of Multiprocessor Programming182 private boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); } List Validate Method Predecessor still Points to current

183. Art of Multiprocessor Programming183 Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}}

184. Art of Multiprocessor Programming184 Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Validate as before

185. Art of Multiprocessor Programming185 Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Key found

186. Art of Multiprocessor Programming186 Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Logical remove

187. Art of Multiprocessor Programming187 Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} physical remove

188. Art of Multiprocessor Programming188 Contains public boolean contains(Item item) { int key = item.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; }

189. Art of Multiprocessor Programming189 Contains public boolean contains(Item item) { int key = item.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Start at the head

190. Art of Multiprocessor Programming190 Contains public boolean contains(Item item) { int key = item.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Search key range

191. Art of Multiprocessor Programming191 Contains public boolean contains(Item item) { int key = item.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Traverse without locking (nodes may have been removed)

192. Art of Multiprocessor Programming192 Contains public boolean contains(Item item) { int key = item.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Present and undeleted?

193. Art of Multiprocessor Programming193 Summary: Wait-free Contains a 0 0 0 a b c 0 e 1 d Use Mark bit + list ordering 1.Not marked  in the set 2.Marked or missing  not in the set

194. Art of Multiprocessor Programming194 Lazy List a 0 0 0 a b c 0 e 1 d Lazy add() and remove() + Wait-free contains()

195. Art of Multiprocessor Programming195 Evaluation Good: –contains() doesn’t lock –In fact, its wait-free! –Good because typically high % contains() –Uncontended calls don’t re-traverse Bad –Contended add() and remove() calls do re-traverse –Traffic jam if one thread delays

196. Art of Multiprocessor Programming196 Traffic Jam Any concurrent data structure based on mutual exclusion has a weakness If one thread –Enters critical section –And “eats the big muffin” Cache miss, page fault, descheduled … –Everyone else using that lock is stuck! –Need to trust the scheduler….

197. Art of Multiprocessor Programming197 Reminder: Lock-Free Data Structures No matter what … –Guarantees minimal progress in any execution –i.e. Some thread will always complete a method call –Even if others halt at malicious times –Implies that implementation can’t use locks

198. Art of Multiprocessor Programming198 Lock-free Lists Next logical step –Wait-free contains() –lock-free add() and remove() Use only compareAndSet() –What could go wrong?

199. Art of Multiprocessor Programming199 Remove Using CAS a 0 0 0 a b c 0 e 1 c Logical Removal = Set Mark Bit Physical Removal CAS pointer Use CAS to verify pointer is correct Not enough!

200. Art of Multiprocessor Programming200 Problem… a 0 0 0 a b c 0 e 1 c Logical Removal = Set Mark Bit Physical Removal CAS 0 d Problem: d not added to list… Must Prevent manipulation of removed node’s pointer Node added Before Physical Removal CAS

201. Art of Multiprocessor Programming201 The Solution: Combine Bit and Pointer a 0 0 0 a b c 0 e 1 c Logical Removal = Set Mark Bit Physical Removal CAS 0 d Mark-Bit and Pointer are CASed together (AtomicMarkableReference) Fail CAS: Node not added after logical Removal

202. Art of Multiprocessor Programming202 Solution Use AtomicMarkableReference Atomically –Swing reference and –Update flag Remove in two steps –Set mark bit in next field –Redirect predecessor’s pointer

203. Art of Multiprocessor Programming203 Marking a Node AtomicMarkableReference class –Java.util.concurrent.atomic package address F mark bit Reference

204. Art of Multiprocessor Programming204 Extracting Reference & Mark Public Object get(boolean[] marked);

205. Art of Multiprocessor Programming205 Extracting Reference & Mark Public Object get(boolean[] marked); Returns reference Returns mark at array index 0!

206. Art of Multiprocessor Programming206 Extracting Mark Only public boolean isMarked(); Value of mark

207. Art of Multiprocessor Programming207 Changing State Public boolean compareAndSet( Object expectedRef, Object updateRef, boolean expectedMark, boolean updateMark);

208. Art of Multiprocessor Programming208 Changing State Public boolean compareAndSet( Object expectedRef, Object updateRef, boolean expectedMark, boolean updateMark); If this is the current reference … And this is the current mark …

209. Art of Multiprocessor Programming209 Changing State Public boolean compareAndSet( Object expectedRef, Object updateRef, boolean expectedMark, boolean updateMark); …then change to this new reference … … and this new mark

210. Art of Multiprocessor Programming210 Changing State public boolean attemptMark( Object expectedRef, boolean updateMark);

211. Art of Multiprocessor Programming211 Changing State public boolean attemptMark( Object expectedRef, boolean updateMark); If this is the current reference …

212. Art of Multiprocessor Programming212 Changing State public boolean attemptMark( Object expectedRef, boolean updateMark);.. then change to this new mark.

213. Art of Multiprocessor Programming213 Removing a Node abcd remov e c CAS

214. Art of Multiprocessor Programming214 Removing a Node abd remov e b remov e c c CAS failed

215. Art of Multiprocessor Programming215 Removing a Node abd remov e b remov e c c

216. Art of Multiprocessor Programming216 Removing a Node ad remov e b remov e c

217. Art of Multiprocessor Programming217 Traversing the List Q: what do you do when you find a “logically” deleted node in your path? A: finish the job. –CAS the predecessor’s next field –Proceed (repeat as needed)

218. Art of Multiprocessor Programming218 Lock-Free Traversal (only Add and Remove) abcd CAS Uh-oh pred curr pred curr

219. Art of Multiprocessor Programming219 The Window Class class Window { public Node pred; public Node curr; Window(Node pred, Node curr) { this.pred = pred; this.curr = curr; }

220. Art of Multiprocessor Programming220 The Window Class class Window { public Node pred; public Node curr; Window(Node pred, Node curr) { this.pred = pred; this.curr = curr; } A container for pred and current values

221. Art of Multiprocessor Programming221 Using the Find Method Window window = find(head, key); Node pred = window.pred; curr = window.curr;

222. Art of Multiprocessor Programming222 Using the Find Method Window window = find(head, key); Node pred = window.pred; curr = window.curr; Find returns window

223. Art of Multiprocessor Programming223 Using the Find Method Window window = find(head, key); Node pred = window.pred; curr = window.curr; Extract pred and curr

224. Art of Multiprocessor Programming© Herlihy-Shavit 2007 224 The Find Method Window window = find(item); At some instant, predcurrsucc item or …

225. Art of Multiprocessor Programming© Herlihy-Shavit 2007 225 The Find Method Window window = find(item); At some instant, pred curr= null succ item not in list

226. Art of Multiprocessor Programming226 Remove public boolean remove(T item) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}}

227. Art of Multiprocessor Programming227 Remove public boolean remove(T item) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} Keep trying

228. Art of Multiprocessor Programming228 Remove public boolean remove(T item) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} Find neighbors

229. Art of Multiprocessor Programming229 Remove public boolean remove(T item) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} She’s not there …

230. Art of Multiprocessor Programming230 Remove public boolean remove(T item) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} Try to mark node as deleted

231. Art of Multiprocessor Programming231 Remove public boolean remove(T item) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} If it doesn’t work, just retry, if it does, job essentially done

232. Art of Multiprocessor Programming232 Remove public boolean remove(T item) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.attemptMark(succ, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} Try to advance reference (if we don’t succeed, someone else did or will). a

233. Art of Multiprocessor Programming233 Add public boolean add(T item) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(item); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;} }}}

234. Art of Multiprocessor Programming234 Add public boolean add(T item) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(item); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;} }}} Item already there.

235. Art of Multiprocessor Programming235 Add public boolean add(T item) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(item); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;} }}} create new node

236. Art of Multiprocessor Programming236 Add public boolean add(T item) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(item); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;} }}} Install new node, else retry loop

237. Art of Multiprocessor Programming237 Wait-free Contains public boolean contains(Tt item) { boolean marked; int key = item.hashCode(); Node curr = this.head; while (curr.key < key) curr = curr.next; Node succ = curr.next.get(marked); return (curr.key == key && !marked[0]) }

238. Art of Multiprocessor Programming238 Wait-free Contains public boolean contains(T item) { boolean marked; int key = item.hashCode(); Node curr = this.head; while (curr.key < key) curr = curr.next; Node succ = curr.next.get(marked); return (curr.key == key && !marked[0]) } Only diff is that we get and check marked

239. Art of Multiprocessor Programming239 Lock-free Find public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }}

240. Art of Multiprocessor Programming240 Lock-free Find public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} If list changes while traversed, start over Lock-Free because we start over only if someone else makes progress

241. Art of Multiprocessor Programming241 public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find Start looking from head

242. Art of Multiprocessor Programming242 public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find Move down the list

243. Art of Multiprocessor Programming243 public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find Get ref to successor and current deleted bit

244. Art of Multiprocessor Programming244 public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find Try to remove deleted nodes in path…code details soon

245. Art of Multiprocessor Programming245 public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find If curr key that is greater or equal, return pred and curr

246. Art of Multiprocessor Programming246 public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Lock-free Find Otherwise advance window and loop again

247. Art of Multiprocessor Programming247 Lock-free Find retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); } …

248. Art of Multiprocessor Programming248 Lock-free Find retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); } … Try to snip out node

249. Art of Multiprocessor Programming249 Lock-free Find retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); } … if predecessor’s next field changed must retry whole traversal

250. Art of Multiprocessor Programming250 Lock-free Find retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); } … Otherwise move on to check if next node deleted

251. Art of Multiprocessor Programming251 Performance On 16 node shared memory machine Benchmark throughput of Java List-based Set algs. Vary % of Contains() method Calls.

252. Art of Multiprocessor Programming252 High Contains Ratio Lock-free Lazy list Course Grained Fine Lock-coupling

253. Art of Multiprocessor Programming253 Low Contains Ratio

254. Art of Multiprocessor Programming254 As Contains Ratio Increases % Contains()

255. Art of Multiprocessor Programming255 Summary Coarse-grained locking Fine-grained locking Optimistic synchronization Lock-free synchronization