Lecture 9 : Concurrent Data Structures Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

Concurrent Data Structures We assume –shared-memory multiprocessors environment –concurrently execute multiple threads which communicate and synchronize through data structures in shared memory

Concurrent Data Structures : Issues Far more difficult to design than sequential ones –Correctness Primary source of difficulty is concurrency The steps of different threads can be interleaved arbitrarily –Scalability (high performance) 3

Lock Granularity Coarse grained locking –Simple & Easy –Sequential bottleneck Problem at most one thread does useful work Fine grained locking –Improve the problem of sequential bottleneck (i.e. multiple threades wait for a working thread) 4

Example Concurrent linked list –Coarse grained locking –Fine grained locking Scalable counter 5

6 List-Based Sets public interface Set { public boolean add(T x); public boolean remove(T x); public boolean contains(T x); } We assume data are sorted in list.

7 List Node public class Node { public T item; // item of interest public int key; // usually hash code public Node next; // reference to next node }

8 The List-Based Set abc Sorted with Sentinel nodes (min & max possible keys) -∞ +∞

9 Sequential List Based Set a c d a b c Add() Remove()

10 Sequential List Based Set a c d b a b c Add() Remove()

11 Course Grained Locking a b d

12 Course Grained Locking a b d c

13 honk! Course Grained Locking a b d c Simple but hotspot + bottleneck honk!

14 Coarse-Grained Synchronization Sequential bottleneck –Threads “stand in line” Adding more threads –Does not improve throughput –Struggle to keep it from getting worse So why even use a multiprocessor? –Well, some apps inherently parallel …

public class CoarseList { private Node head; private Node tail; private Lock lock = new ReentrantLock(); public CoarseList() { // Add sentinels to start and end head = new Node(Integer.MIN_VALUE); tail = new Node(Integer.MAX_VALUE); head.next = this.tail; } 15 public boolean add(T item) { Node pred, curr; int key = item.hashCode(); lock.lock(); try { pred = head; curr = pred.next; while (curr.key < key) { pred = curr; curr = curr.next; } if (key == curr.key) { return false; } else { Node node = new Node(item); node.next = curr; pred.next = node; return true; } } finally { lock.unlock(); } Coarse-Grained Synchronization (Linked List)

16 public boolean contains(T item) { Node pred, curr; int key = item.hashCode(); lock.lock(); try { pred = head; curr = pred.next; while (curr.key < key) { pred = curr; curr = curr.next; } return (key == curr.key); } finally { lock.unlock(); } public boolean remove(T item) { Node pred, curr; int key = item.hashCode(); lock.lock(); try { pred = this.head; curr = pred.next; while (curr.key < key) { pred = curr; curr = curr.next; } if (key == curr.key) pred.next = curr.next; return true; } else { return false; } } finally { lock.unlock(); }

17 Coarse-Grained Locking Easy, same as synchronized methods –“One lock to rule them all …” Simple, clearly correct –Deserves respect! Works poorly with contention

18 Performance Improvement For highly-concurrent objects Goal: –Concurrent access –More threads, more throughput

19 First: Fine-Grained Synchronization Instead of using a single lock.. Split object into –Independently-synchronized components Methods conflict when they access –The same component … –At the same time

20 Second: Optimistic Synchronization Search without locking … If you find it, lock and check … –OK: we are done –Oops: start over Evaluation –Usually cheaper than locking –Mistakes are expensive

21 Third: Lazy Synchronization Postpone hard work Removing components is tricky –Logical removal Mark component to be deleted –Physical removal Do what needs to be done

22 Fine-grained Locking Requires careful thought –“Do not meddle in the affairs of wizards, for they are subtle and quick to anger” Split object into pieces –Each piece has own lock –Methods that work on disjoint pieces need not exclude each other

23 Fine-grained Locking Use multiple locks of small granularity to protect different parts of the data structure Goal –To allow concurrent operations to proceed in parallel when they do not access the same parts of the data structure

24 Hand-over-Hand locking abc

25 Hand-over-Hand locking abc

26 Hand-over-Hand locking abc

27 Hand-over-Hand locking abc

28 Hand-over-Hand locking abc

29 Removing a Node abcd remove(b)

30 Removing a Node abcd remove(b)

31 Removing a Node abcd remove(b)

32 Removing a Node abcd remove(b)

33 Removing a Node abcd remove(b)

34 Removing a Node acd remove(b) Why do we need to always hold 2 locks?

35 Concurrent Removes abcd remove(c) remove(b)

36 Concurrent Removes abcd remove(b) remove(c)

37 Concurrent Removes abcd remove(b) remove(c)

38 Concurrent Removes abcd remove(b) remove(c)

39 Concurrent Removes abcd remove(b) remove(c)

40 Concurrent Removes abcd remove(b) remove(c)

41 Concurrent Removes abcd remove(b) remove(c)

42 Concurrent Removes abcd remove(b) remove(c)

Art of Multiprocessor Programming43 Uh, Oh acd remove(b) remove(c)

44 Uh, Oh acd Bad news, C not removed remove(b) remove(c)

45 Problem To delete node c –Swing node b’s next field to d Problem is, –Someone deleting b concurrently could direct a pointer to c ba cbac

46 Insight If a node is locked –No one can delete node’s successor If a thread locks –Node to be deleted –And its predecessor –Then it works

47 Hand-Over-Hand Again abcd remove(b)

48 Hand-Over-Hand Again abcd remove(b)

49 Hand-Over-Hand Again abcd remove(b)

50 Hand-Over-Hand Again abcd remove(b) Found it!

51 Hand-Over-Hand Again abcd remove(b) Found it!

52 Hand-Over-Hand Again acd remove(b)

53 Removing a Node abcd remove(b) remove(c)

54 Removing a Node abcd remove(b) remove(c)

55 Removing a Node abcd remove(b) remove(c)

56 Removing a Node abcd remove(b) remove(c)

57 Removing a Node abcd remove(b) remove(c)

58 Removing a Node abcd remove(b) remove(c)

59 Removing a Node abcd remove(b) remove(c)

60 Removing a Node abcd remove(b) remove(c)

61 Removing a Node abcd Must acquire Lock of b remove(c)

62 Removing a Node abcd Cannot acquire lock of b remove(c)

63 Removing a Node abcd Wait! remove(c)

64 Removing a Node abd Proceed to remove(b)

65 Removing a Node abd remove(b)

66 Removing a Node abd remove(b)

67 Removing a Node ad remove(b)

68 Removing a Node ad

69 Fine-Grained Synchronization: hand-over-hand locking Linked List public boolean add(T item) { int key = item.hashCode(); head.lock(); Node pred = head; try { Node curr = pred.next; curr.lock(); try { while (curr.key < key) { pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } if (curr.key == key) return false; Node newNode = new Node(item); newNode.next = curr; pred.next = newNode; return true; } finally { curr.unlock(); } } finally { pred.unlock(); }

70 public boolean contains(T item) { Node last = null, pred = null, curr = null; int key = item.hashCode(); head.lock(); try { pred = head; curr = pred.next; curr.lock(); try { while (curr.key < key) { pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return (curr.key == key); } finally { curr.unlock(); } } finally { pred.unlock(); } public boolean remove(T item) { Node pred = null, curr = null; int key = item.hashCode(); head.lock(); try { pred = head; curr = pred.next; curr.lock(); try { while (curr.key < key) { pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } if (curr.key == key) { pred.next = curr.next; return true; } return false; } finally { curr.unlock(); } } finally { pred.unlock(); }

71 Why does this work? To remove node e –Must lock e –Must lock e’s predecessor Therefore, if you lock a node –It can’t be removed –And neither can its successor

72 Adding Nodes To add node e –Must lock predecessor –Must lock successor Neither can be deleted –(Is successor lock actually required?)

73 Drawbacks Better than coarse-grained lock –Threads can traverse in parallel Still not ideal –Long chain of acquire/release –Inefficient

74 Improved method for concurrent linked list? Find nodes without locking Lock nodes Check that everything is OK Think about it…

Concurrent Hash Table Coarse grained locking Fine grained locking 75

Concurrent Counter Problem Concurrent Counter –Commonly used data structure –Has simple interface –Sequential bottleneck problem Design Scalable Concurrent Counter 76

Counter without lock 77

Counter with lock 78

Scalable concurrent lock? Perfermance vs Accuracy tradeoff 79