1. Art of Multiprocessor Programming1 Concurrent Queues Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit Modified by Rajeev Alur For CIS 640 University of Pennsylvania

2. Art of Multiprocessor Programming2 2 Filter class Filter implements Lock { int[] level; // level[i] for thread i int[] victim; // victim[L] for level L public Filter(int n) { level = new int[n]; victim = new int[n]; for (int i = 1; i < n; i++) { level[i] = 0; }} … } level victim n -1 0 1 0000004 2 2 Thread 2 at level 4 0 4

3. Art of Multiprocessor Programming3 3 Filter class Filter implements Lock { … public void lock(){ for (int L = 1; L < n; L++) { level[i] = L; victim[L] = i; while ((   k != i level[k] >= L) && victim[L] == i ); }} public void unlock() { level[i] = 0; }}

4. Art of Multiprocessor Programming44 Queues & Stacks pool of items

5. Art of Multiprocessor Programming55 Queues deq()/ enq( ) Total order First in First out

6. Art of Multiprocessor Programming66 Stacls pop()/ push( ) Total order Last in First out

7. Art of Multiprocessor Programming77 Bounded Fixed capacity Good when resources an issue

8. Art of Multiprocessor Programming88 Unbounded Unlimited capacity Often more convenient …

9. Art of Multiprocessor Programming9 Blocking zzz … Block on attempt to remove from empty stack or queue

10. Art of Multiprocessor Programming10 Blocking zzz … Block on attempt to add to full bounded stack or queue

11. Art of Multiprocessor Programming11 Non-Blocking Ouch! Throw exception on attempt to remove from empty stack or queue

12. Art of Multiprocessor Programming12 Queue: Concurrency enq(x) y=deq() enq() and deq() work at different ends of the object tailhead

13. Art of Multiprocessor Programming13 Concurrency enq(x) Challenge: what if the queue is empty or full? y=deq() tail head

14. Art of Multiprocessor Programming14 Bounded Queue Sentinel head tail

15. Art of Multiprocessor Programming15 Bounded Queue head tail First actual item

16. Art of Multiprocessor Programming16 Bounded Queue head tail Lock out other deq() calls deqLock

17. Art of Multiprocessor Programming17 Bounded Queue head tail Lock out other enq() calls deqLock enqLock

18. Art of Multiprocessor Programming18Art of Multiprocessor Programming 18 Not Done Yet head tail deqLock enqLock Need to tell whether queue is full or empty

19. Art of Multiprocessor Programming19 Not Done Yet head tail deqLock enqLock Permission to enqueue 8 items permits 8

20. Art of Multiprocessor Programming20 Not Done Yet head tail deqLock enqLock Incremented by deq() Decremented by enq() permits 8

21. Art of Multiprocessor Programming21 Enqueuer head tail deqLock enqLock permits 8 Lock enqLock

22. Art of Multiprocessor Programming22 Enqueuer head tail deqLock enqLock permits 8 Read permits OK

23. Art of Multiprocessor Programming23 Enqueuer head tail deqLock enqLock permits 8 No need to lock tail

24. Art of Multiprocessor Programming24 Enqueuer head tail deqLock enqLock permits 8 Enqueue Node

25. Art of Multiprocessor Programming25 Enqueuer head tail deqLock enqLock permits 8 7 getAndDecrement()

26. Art of Multiprocessor Programming26 Enqueuer head tail deqLock enqLock permits 8 Release lock 7

27. Art of Multiprocessor Programming27 Enqueuer head tail deqLock enqLock permits 7 If queue was empty, notify waiting dequeuers

28. Art of Multiprocessor Programming28 Unsuccesful Enqueuer head tail deqLock enqLock permits 0 Uh-oh Read permits

29. Art of Multiprocessor Programming29 Dequeuer head tail deqLock enqLock permits 8 Lock deqLock

30. Art of Multiprocessor Programming30 Dequeuer head tail deqLock enqLock permits 7 Read sentinel’s next field OK

31. Art of Multiprocessor Programming31 Dequeuer head tail deqLock enqLock permits 7 Read value

32. Art of Multiprocessor Programming32 Dequeuer head tail deqLock enqLock permits 7 Make first Node new sentinel

33. Art of Multiprocessor Programming33 Dequeuer head tail deqLock enqLock permits 8 Increment permits

34. Art of Multiprocessor Programming34 Dequeuer head tail deqLock enqLock permits 7 Release deqLock

35. Art of Multiprocessor Programming35 Unsuccesful Dequeuer head tail deqLock enqLock permits 8 Read sentinel’s next field uh-oh

36. Art of Multiprocessor Programming36 Bounded Queue public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger permits; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); }

37. Art of Multiprocessor Programming37 Bounded Queue public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger permits; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } Enq & deq locks

38. Art of Multiprocessor Programming38 Digression: Monitor Locks Java Synchronized objects and Java ReentrantLocks are monitors Allow blocking on a condition rather than spinning Threads: –acquire and release lock –wait on a condition

39. Art of Multiprocessor Programming39 public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } The Java Lock Interface Acquire lock

40. Art of Multiprocessor Programming40 public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } The Java Lock Interface Release lock

41. Art of Multiprocessor Programming41 public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } The Java Lock Interface Try for lock, but not too hard

42. Art of Multiprocessor Programming42 public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } The Java Lock Interface Create condition to wait on

43. Art of Multiprocessor Programming43 The Java Lock Interface public interface Lock { void lock(); void lockInterruptibly() throws InterruptedException; boolean tryLock(); boolean tryLock(long time, TimeUnit unit); Condition newCondition(); void unlock; } Guess what this method does?

44. Art of Multiprocessor Programming44 Lock Conditions public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); }

45. Art of Multiprocessor Programming45 public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); } Lock Conditions Release lock and wait on condition

46. Art of Multiprocessor Programming46 public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); } Lock Conditions Wake up one waiting thread

47. Art of Multiprocessor Programming47 public interface Condition { void await(); boolean await(long time, TimeUnit unit); … void signal(); void signalAll(); } Lock Conditions Wake up all waiting threads

48. Art of Multiprocessor Programming48 Await Releases lock associated with q Sleeps (gives up processor) Awakens (resumes running) Reacquires lock & returns q.await()

49. Art of Multiprocessor Programming49 Signal Awakens one waiting thread –Which will reacquire lock q.signal();

50. Art of Multiprocessor Programming50 Signal All Awakens all waiting threads –Which will each reacquire lock q.signalAll();

51. Art of Multiprocessor Programming51 A Monitor Lock Critical Section waiting room Lock() unLock()

52. Art of Multiprocessor Programming52 Unsuccessful Deq Critical Section waiting room Lock() await() Deq() Oh no, Empty!

53. Art of Multiprocessor Programming53 Another One Critical Section waiting room Lock() await() Deq() Oh no, Empty!

54. Art of Multiprocessor Programming54 Enqueur to the Rescue Critical Section waiting room Lock() signalAll() Enq( ) unLock() Yawn!

55. Art of Multiprocessor Programming55 Yawn! Monitor Signalling Critical Section waiting room Yawn! Awakend thread might still lose lock to outside contender…

56. Art of Multiprocessor Programming56 Dequeurs Signalled Critical Section waiting room Found it Yawn!

57. Art of Multiprocessor Programming57 The Bounded Queue public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger permits; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); }

58. Art of Multiprocessor Programming58 Bounded Queue Fields public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger permits; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } Enq & deq locks

59. Art of Multiprocessor Programming59 Bounded Queue Fields public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger permits; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } Enq lock’s associated condition

60. Art of Multiprocessor Programming60 Bounded Queue Fields public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger permits; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } Num permits: 0 to capacity

61. Art of Multiprocessor Programming61 Bounded Queue Fields public class BoundedQueue { ReentrantLock enqLock, deqLock; Condition notEmptyCondition, notFullCondition; AtomicInteger permits; Node head; Node tail; int capacity; enqLock = new ReentrantLock(); notFullCondition = enqLock.newCondition(); deqLock = new ReentrantLock(); notEmptyCondition = deqLock.newCondition(); } Head and Tail

62. Art of Multiprocessor Programming62 Enq Method Part One public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (permits.get() == 0) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = e; if (permits.getAndDecrement() == capacity) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … }

63. Art of Multiprocessor Programming63 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (permits.get() == 0) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = e; if (permits.getAndDecrement() == capacity) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One Lock and unlock enq lock

64. Art of Multiprocessor Programming64 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (permits.get() == 0) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = e; if (permits.getAndDecrement() == capacity) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One If queue is full, patiently await further instructions …

65. Art of Multiprocessor Programming65 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (permits.get() == 0) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = e; if (permits.getAndDecrement() == capacity) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Be Afraid How do we know the permits field won’t change?

66. Art of Multiprocessor Programming66 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (permits.get() == 0) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = e; if (permits.getAndDecrement() == capacity) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One Add new node

67. Art of Multiprocessor Programming67 public void enq(T x) { boolean mustWakeDequeuers = false; enqLock.lock(); try { while (permits.get() == 0) notFullCondition.await(); Node e = new Node(x); tail.next = e; tail = e; if (permits.getAndDecrement() == capacity) mustWakeDequeuers = true; } finally { enqLock.unlock(); } … } Enq Method Part One If queue was empty, wake frustrated dequeuers

68. Art of Multiprocessor Programming68 Enq Method Part Deux public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); }

69. Art of Multiprocessor Programming69 Enq Method Part Deux public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); } Are there dequeuers to be signaled?

70. Art of Multiprocessor Programming70 public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); } Enq Method Part Deux Lock and unlock deq lock

71. Art of Multiprocessor Programming71 public void enq(T x) { … if (mustWakeDequeuers) { deqLock.lock(); try { notEmptyCondition.signalAll(); } finally { deqLock.unlock(); } Enq Method Part Deux Signal dequeuers that queue no longer empty

72. Art of Multiprocessor Programming72 The Enq() & Deq() Methods Share no locks –That’s good But do share an atomic counter –Accessed on every method call –That’s not so good Can we alleviate this bottleneck? –By splitting the counter

73. Art of Multiprocessor Programming73 A Lock-Free Queue (Michael&Scott) Sentinel head tail

74. Art of Multiprocessor Programming74 Compare and Set CAS

75. Art of Multiprocessor Programming75 Enqueue head tail Enq( )

76. Art of Multiprocessor Programming76 Enqueue head tail

77. Art of Multiprocessor Programming77 Logical Enqueue head tail CAS

78. Art of Multiprocessor Programming78 Physical Enqueue head tail Enqueue Node CAS

79. Art of Multiprocessor Programming79 Enqueue These two steps are not atomic The tail field refers to either –Actual last Node (good) –Penultimate Node (not so good) Be prepared!

80. Art of Multiprocessor Programming80 Enqueue What do you do if you find –A trailing tail? Stop and help fix it –If tail node has non-null next field –CAS the queue’s tail field to tail.next

81. Art of Multiprocessor Programming81 When CASs Fail During logical enqueue –Abandon hope, restart –Still lock-free (why?) During physical enqueue –Ignore it (why?)

82. Art of Multiprocessor Programming82 Dequeuer head tail Read value

83. Art of Multiprocessor Programming83 Dequeuer head tail Make first Node new sentinel CAS

84. Art of Multiprocessor Programming84 Enq Method public void enq(T x) { Node node = new Node(x); while (true) { Node last = tail.get(); Node next = last.next.get(); if (last == tail.get()) { if (next == null) { if (last.next.compareAndSet(next,node)) { tail.compareAndSet(last, node); return; } } else { tail.compareAndSet(last, next); } }

85. Art of Multiprocessor Programming85 Memory Reuse? What do we do with nodes after we dequeue them? Java: let garbage collector deal? Suppose there is no GC, or we prefer not to use it?

86. Art of Multiprocessor Programming86 Dequeuer head tail CAS Can recycle

87. Art of Multiprocessor Programming87 Simple Solution Each thread has a free list of unused queue nodes Allocate node: pop from list Free node: push onto list Deal with underflow somehow …

88. Art of Multiprocessor Programming88 Why Recycling is Hard Free pool headtail Want to redirect tail from grey to red zzz…

89. Art of Multiprocessor Programming89 Both Nodes Reclaimed Free pool zzz headtail

90. Art of Multiprocessor Programming90 One Node Recycled Free pool Yawn! headtail

91. Art of Multiprocessor Programming91 Why Recycling is Hard Free pool CAS headtail OK, here I go!

92. Art of Multiprocessor Programming92 Epic FAIL Free pool zOMG what went wrong? headtail Bad news

93. Art of Multiprocessor Programming93 The Dreaded ABA Problem Head pointer has value A Thread reads value A headtail

94. Art of Multiprocessor Programming94 Dreaded ABA continued zzz Head pointer has value B Node A freed headtail

95. Art of Multiprocessor Programming95 Dreaded ABA continued Yawn! Head pointer has value A again Node A recycled & reinitialized headtail

96. Art of Multiprocessor Programming96 Dreaded ABA continued CAS succeeds because pointer matches even though pointer’s meaning has changed CAS headtail

97. Art of Multiprocessor Programming97 The Dreaded ABA Problem Is a result of CAS() semantics –blame Sun, Intel, AMD, … Not with Load-Locked/Store- Conditional –Good for IBM?

98. Art of Multiprocessor Programming98 Dreaded ABA – A Solution Tag each pointer with a counter Unique over lifetime of node Pointer size vs word size issues Overflow? –Don’t worry be happy? –Bounded tags? AtomicStampedReference class

99. Art of Multiprocessor Programming99 Atomic Stamped Reference AtomicStampedReference class –Java.util.concurrent.atomic package address S Stamp Reference Can get reference and stamp atomically