1. Introduction to Multiprocessor Synchronization Maurice Herlihy TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAA

2. Art of Multiprocessor Programming 2 Moore's Law Clock speed flattening sharply Transistor count still rising

3. Moore's Law (in practice) Art of Multiprocessor Programming3

4. 4 Once roamed the Earth: the Uniprocesor memory cpu

5. Art of Multiprocessor Programming 5 Endangered: The Shared Memory Multiprocessor (SMP) cache Bus shared memory cache

6. Art of Multiprocessor Programming 6 Meet he New Boss: The Multicore Processor (CMP) cache Bus shared memory cache All on the same chip Oracle Niagara Chip

7. Art of Multiprocessor Programming 7 From the 2008 press… …Intel has announced a press conference in San Francisco on November 17th, where it will officially launch the Core i7 Nehalem processor… …Sun's next generation Enterprise T5140 and T5240 servers, based on the 3rd Generation UltraSPARC T2 Plus processor, were released two days ago…

8. Art of Multiprocessor Programming 8 Why is Kunle Smiling? Niagara 1

9. Art of Multiprocessor Programming 9 Why do we care? Time no longer cures software bloat –The “free ride” is over When you double your program's path length –You can't just wait 6 months –Your software must somehow exploit twice as much concurrency

10. Art of Multiprocessor Programming 10 Traditional Scaling Process User code Traditional Uniprocessor Speedup 1.8x 7x 3.6x Time: Moore's law

11. Ideal Multicore Scaling Process Art of Multiprocessor Programming 11 User code Multicore Speedup 1.8x7x3.6x Unfortunately, not so simple…

12. Actual Multicore Scaling Process Art of Multiprocessor Programming 12 1.8x 2x 2.9x User code Multicore Speedup Parallelization and Synchronization require great care…

13. Art of Multiprocessor Programming 13 Multicore Programming: Course Overview Fundamentals –Models, algorithms, impossibility Real-World programming –Architectures –Techniques

14. Art of Multiprocessor Programming 14 Sequential Computation memory object thread

15. Art of Multiprocessor Programming 15 Concurrent Computation memory object threads

16. Art of Multiprocessor Programming 16 Asynchrony Sudden unpredictable delays –Cache misses (short) –Page faults (long) –Scheduling quantum used up (really long)

17. Art of Multiprocessor Programming 17 Model Summary Multiple threads Single shared memory Objects live in memory Unpredictable asynchronous delays

18. 18 Road Map We are going to focus on principles first, then practice –Start with idealized models –Look at simplistic problems –Emphasize correctness over pragmatism –“Correctness may be theoretical, but incorrectness has practical impact” Art of Multiprocessor Programming

19. 19 Concurrency Jargon Hardware –Processors Software –Threads, processes Sometimes OK to confuse them, sometimes not. Art of Multiprocessor Programming

20. 20 Parallel Primality Testing Challenge –Print primes from 1 to 10 10 Given –Ten-processor multiprocessor –One thread per processor Goal –Get ten-fold speedup (or close) Art of Multiprocessor Programming

21. 21 Load Balancing Split the work evenly Each thread tests range of 10 9 … …10 910 2·10 9 1 P0P0 P1P1 P9P9

22. 22 Procedure for Thread i void primePrint { int i = ThreadID.get(); // IDs in {0..9} for (j = i*10 9 +1, j<(i+1)*10 9 ; j++) { if (isPrime(j)) print(j); } Art of Multiprocessor Programming

23. 23 Issues Higher ranges have fewer primes Yet larger numbers harder to test Thread workloads –Uneven –Hard to predict Art of Multiprocessor Programming

24. 24 Issues Higher ranges have fewer primes Yet larger numbers harder to test Thread workloads –Uneven –Hard to predict Need dynamic load balancing rejected

25. Art of Multiprocessor Programming 25 17 18 19 Shared Counter each thread takes a number

26. 26 Procedure for Thread i int counter = new Counter(1); void primePrint { long j = 0; while (j < 10 10 ) { j = counter.getAndIncrement(); if (isPrime(j)) print(j); } Art of Multiprocessor Programming

27. 27 Counter counter = new Counter(1); void primePrint { long j = 0; while (j < 10 10 ) { j = counter.getAndIncrement(); if (isPrime(j)) print(j); } Procedure for Thread i Shared counter object

28. Art of Multiprocessor Programming 28 Where Things Reside cache Bus cache 1 shared counter shared memory void primePrint { int i = ThreadID.get(); // IDs in {0..9} for (j = i*10 9 +1, j<(i+1)*10 9 ; j++) { if (isPrime(j)) print(j); } code Local variables

29. Art of Multiprocessor Programming 29 Procedure for Thread i Counter counter = new Counter(1); void primePrint { long j = 0; while (j < 10 10 ) { j = counter.getAndIncrement(); if (isPrime(j)) print(j); } Stop when every value taken

30. Art of Multiprocessor Programming 30 Counter counter = new Counter(1); void primePrint { long j = 0; while (j < 10 10 ) { j = counter.getAndIncrement(); if (isPrime(j)) print(j); } Procedure for Thread i Increment & return each new value

31. 31 Counter Implementation public class Counter { private long value; public long getAndIncrement() { return value++; } Art of Multiprocessor Programming

32. 32 Counter Implementation public class Counter { private long value; public long getAndIncrement() { return value++; } OK for single thread, not for concurrent threads

33. Art of Multiprocessor Programming 33 What It Means public class Counter { private long value; public long getAndIncrement() { return value++; }

34. Art of Multiprocessor Programming 34 What It Means public class Counter { private long value; public long getAndIncrement() { return value++; } temp = value; value = temp + 1; return temp;

35. Art of Multiprocessor Programming 35 time Not so good… Value… 1 read 1 read 1 write 2 read 2 write 3 write 2 232

36. Art of Multiprocessor Programming 36 Is this problem inherent? If we could only glue reads and writes together… read write read write !!

37. 37 Challenge public class Counter { private long value; public long getAndIncrement() { temp = value; value = temp + 1; return temp; } Art of Multiprocessor Programming

38. 38 Challenge public class Counter { private long value; public long getAndIncrement() { temp = value; value = temp + 1; return temp; } Make these steps atomic (indivisible)

39. Art of Multiprocessor Programming 39 Hardware Solution public class Counter { private long value; public long getAndIncrement() { temp = value; value = temp + 1; return temp; } ReadModifyWrite() instruction

40. Art of Multiprocessor Programming 40 An Aside: Java™ public class Counter { private long value; public long getAndIncrement() { synchronized { temp = value; value = temp + 1; } return temp; }

41. Art of Multiprocessor Programming 41 An Aside: Java™ public class Counter { private long value; public long getAndIncrement() { synchronized { temp = value; value = temp + 1; } return temp; } Synchronized block

42. Art of Multiprocessor Programming 42 An Aside: Java™ public class Counter { private long value; public long getAndIncrement() { synchronized { temp = value; value = temp + 1; } return temp; } Mutual Exclusion

43. 43 Mutual Exclusion, or “Alice & Bob share a pond” AB Art of Multiprocessor Programming

44. 44 Alice has a pet AB Art of Multiprocessor Programming

45. 45 Bob has a pet AB Art of Multiprocessor Programming

46. 46 The Problem AB The pets don't get along Art of Multiprocessor Programming

47. 47 Formalizing the Problem Two types of formal properties in asynchronous computation: Safety Properties –Nothing bad happens ever Liveness Properties –Something good happens eventually Art of Multiprocessor Programming

48. 48 Formalizing our Problem Mutual Exclusion –Both pets never in pond simultaneously –This is a safety property No Deadlock –if only one wants in, it gets in –if both want in, one gets in –This is a liveness property Art of Multiprocessor Programming

49. 49 Simple Protocol Idea –Just look at the pond Gotcha –Not atomic –Trees obscure the view Art of Multiprocessor Programming

50. 50 Interpretation Threads can't “see” what other threads are doing Explicit communication required for coordination Art of Multiprocessor Programming

51. 51 Cell Phone Protocol Idea –Bob calls Alice (or vice-versa) Gotcha –Bob takes shower –Alice recharges battery –Bob out shopping for pet food … Art of Multiprocessor Programming

52. 52 Interpretation Message-passing doesn't work Recipient might not be –Listening –There at all Communication must be –Persistent (like writing) –Not transient (like speaking) Art of Multiprocessor Programming

53. 53 Can Protocol cola Art of Multiprocessor Programming

54. 54 Bob conveys a bit AB cola Art of Multiprocessor Programming

55. 55 Bob conveys a bit AB cola Art of Multiprocessor Programming

56. 56 Can Protocol Idea –Cans on Alice's windowsill –Strings lead to Bob's house –Bob pulls strings, knocks over cans Gotcha –Cans cannot be reused –Bob runs out of cans Art of Multiprocessor Programming

57. 57 Interpretation Cannot solve mutual exclusion with interrupts –Sender sets fixed bit in receiver's space –Receiver resets bit when ready –Requires unbounded number of interrupt bits Art of Multiprocessor Programming

58. 58 Flag Protocol AB Art of Multiprocessor Programming

59. 59 Alice's Protocol (sort of) AB Art of Multiprocessor Programming

60. 60 Bob's Protocol (sort of) AB Art of Multiprocessor Programming

61. 61 Alice's Protocol Raise flag Wait until Bob's flag is down Unleash pet Lower flag when pet returns Art of Multiprocessor Programming

62. 62 Bob's Protocol Raise flag Wait until Alice's flag is down Unleash pet Lower flag when pet returns danger!

63. 63 Bob's Protocol (2 nd try) Raise flag While Alice's flag is up –Lower flag –Wait for Alice's flag to go down –Raise flag Unleash pet Lower flag when pet returns Art of Multiprocessor Programming

64. 64 Bob's Protocol Raise flag While Alice's flag is up –Lower flag –Wait for Alice's flag to go down –Raise flag Unleash pet Lower flag when pet returns Bob defers to Alice

65. 65 The Flag Principle Raise the flag Look at other's flag Flag Principle: –If each raises and looks, then –Last to look must see both flags up Art of Multiprocessor Programming

66. 66 Proof of Mutual Exclusion Assume both pets in pond –Derive a contradiction –By reasoning backwards Consider the last time Alice and Bob each looked before letting the pets in Without loss of generality assume Alice was the last to look… Art of Multiprocessor Programming

67. 67 Proof time Alice's last look Alice last raised her flag Bob's last look QED Alice must have seen Bob's Flag. A Contradiction Bob last raised flag

68. 68 Proof of No Deadlock If only one pet wants in, it gets in. Art of Multiprocessor Programming

69. 69 Proof of No Deadlock If only one pet wants in, it gets in. Deadlock requires both continually trying to get in. Art of Multiprocessor Programming

70. 70 Proof of No Deadlock If only one pet wants in, it gets in. Deadlock requires both continually trying to get in. If Bob sees Alice's flag, he gives her priority (a gentleman…) QED

71. 71 Remarks Protocol is unfair –Bob's pet might never get in Protocol uses waiting –If Bob is eaten by his pet, Alice's pet might never get in Art of Multiprocessor Programming

72. 72 Moral of Story Mutual Exclusion cannot be solved by –transient communication (cell phones) –interrupts (cans) It can be solved by – one-bit shared variables – that can be read or written Art of Multiprocessor Programming

73. 73 The Arbiter Problem (an aside) Pick a point

74. 74 The Fable Continues Alice and Bob fall in love & marry Art of Multiprocessor Programming

75. 75 The Fable Continues Alice and Bob fall in love & marry Then they fall out of love & divorce –She gets the pets –He has to feed them Art of Multiprocessor Programming

76. 76 The Fable Continues Alice and Bob fall in love & marry Then they fall out of love & divorce –She gets the pets –He has to feed them Leading to a new coordination problem: Producer-Consumer Art of Multiprocessor Programming

77. 77 Bob Puts Food in the Pond A Art of Multiprocessor Programming

78. 78 mmm… Alice releases her pets to Feed B mmm… Art of Multiprocessor Programming

79. 79 Producer/Consumer Alice and Bob can't meet –Each has restraining order on other –So he puts food in the pond –And later, she releases the pets Avoid –Releasing pets when there's no food –Putting out food if uneaten food remains Art of Multiprocessor Programming

80. 80 Producer/Consumer Need a mechanism so that –Bob lets Alice know when food has been put out –Alice lets Bob know when to put out more food Art of Multiprocessor Programming

81. 81 Surprise Solution AB cola Art of Multiprocessor Programming

82. 82 Bob puts food in Pond AB cola Art of Multiprocessor Programming

83. 83 Bob knocks over Can AB cola Art of Multiprocessor Programming

84. 84 Alice Releases Pets AB cola yum… B Art of Multiprocessor Programming

85. 85 Alice Resets Can when Pets are Fed AB cola Art of Multiprocessor Programming

86. 86 Pseudocode while (true) { while (can.isUp()){}; pet.release(); pet.recapture(); can.reset(); } Alice's code

87. Art of Multiprocessor Programming 87 Pseudocode while (true) { while (can.isUp()){}; pet.release(); pet.recapture(); can.reset(); } Alice's code while (true) { while (can.isDown()){}; pond.stockWithFood(); can.knockOver(); } Bob's code

88. 88 Correctness Mutual Exclusion –Pets and Bob never together in pond Art of Multiprocessor Programming

89. 89 Correctness Mutual Exclusion –Pets and Bob never together in pond No Starvation if Bob always willing to feed, and pets always famished, then pets eat infinitely often. Art of Multiprocessor Programming

90. 90 Correctness Mutual Exclusion –Pets and Bob never together in pond No Starvation if Bob always willing to feed, and pets always famished, then pets eat infinitely often. Producer/Consumer The pets never enter pond unless there is food, and Bob never provides food if there is unconsumed food. safety liveness safety

91. 91 Could Also Solve Using Flags AB Art of Multiprocessor Programming

92. 92 Waiting Both solutions use waiting –while(mumble){} In some cases waiting is problematic –If one participant is delayed –So is everyone else –But delays are common & unpredictable Art of Multiprocessor Programming

93. 93 The Fable drags on … Bob and Alice still have issues Art of Multiprocessor Programming

94. 94 The Fable drags on … Bob and Alice still have issues So they need to communicate Art of Multiprocessor Programming

95. 95 The Fable drags on … Bob and Alice still have issues So they need to communicate They agree to use billboards … Art of Multiprocessor Programming

96. 96 E 1 D 2 C 3 Billboards are Large B 3 A 1 Letter Tiles From Scrabble™ box Art of Multiprocessor Programming

97. 97 E 1 D 2 C 3 Write One Letter at a Time … B 3 A 1 W 4 A 1 S 1 H 4 Art of Multiprocessor Programming

98. 98 To post a message W 4 A 1 S 1 H 4 A 1 C 3 R 1 T 1 H 4 E 1 whew Art of Multiprocessor Programming

99. 99 S 1 Let's send another message S 1 E 1 L 1 L 1 L 1 V 4 L 1 A 1 M 3 A 1 A 1 P 3 Art of Multiprocessor Programming

100. 100 Uh-Oh A 1 C 3 R 1 T 1 H 4 E 1 S 1 E 1 L 1 L 1 L 1 OK Art of Multiprocessor Programming

101. 101 Readers/Writers Devise a protocol so that –Writer writes one letter at a time –Reader reads one letter at a time –Reader sees “snapshot” Old message or new message No mixed messages Art of Multiprocessor Programming

102. 102 Readers/Writers (continued) Easy with mutual exclusion But mutual exclusion requires waiting –One waits for the other –Everyone executes sequentially Remarkably –We can solve R/W without mutual exclusion Art of Multiprocessor Programming

103. 103 Esoteric? Java container size() method Single shared counter? –incremented with each add() and –decremented with each remove() Threads wait to exclusively access counter performance bottleneck

104. 104 Readers/Writers Solution Each thread i has size[i] counter –only it increments or decrements. To get object's size, a thread reads a “snapshot” of all counters without mutex This eliminates the bottleneck Art of Multiprocessor Programming 4 3001 4 2007 size

105. 105 Why do we care About Sequential Bottlenecks? We want as much of the code as possible to execute in parallel A larger sequential part implies reduced performance Amdahl's law: this relation is not linear… Art of Multiprocessor Programming Eugene Amdahl

106. Art of Multiprocessor Programming 106 Amdahl's Law Speedup = 1-thread execution time N-thread execution time

107. Art of Multiprocessor Programming 107 Amdahl's Law Speedup =

108. Art of Multiprocessor Programming 108 Amdahl's Law Speedup = Parallel fraction

109. Art of Multiprocessor Programming 109 Amdahl's Law Speedup = Parallel fraction Sequential fraction

110. Art of Multiprocessor Programming 110 Amdahl's Law Speedup = Parallel fraction Sequential fraction Number of threads

111. Amdahl's Law (in practice) Art of Multiprocessor Programming111

112. 112 Example Ten processors 60% concurrent, 40% sequential How close to 10-fold speedup? Art of Multiprocessor Programming

113. 113 Example Ten processors 60% concurrent, 40% sequential How close to 10-fold speedup? Speedup = 2.17 = Art of Multiprocessor Programming

114. 114 Example Ten processors 80% concurrent, 20% sequential How close to 10-fold speedup? Art of Multiprocessor Programming

115. 115 Example Ten processors 80% concurrent, 20% sequential How close to 10-fold speedup? Speedup = 3.57 = Art of Multiprocessor Programming

116. 116 Example Ten processors 90% concurrent, 10% sequential How close to 10-fold speedup? Art of Multiprocessor Programming

117. 117 Example Ten processors 90% concurrent, 10% sequential How close to 10-fold speedup? Speedup = 5.26 =

118. 118 Example Ten processors 99% concurrent, 01% sequential How close to 10-fold speedup? Art of Multiprocessor Programming

119. 119 Example Ten processors 99% concurrent, 01% sequential How close to 10-fold speedup? Speedup = 9.17 =

120. Back to Real-World Multicore Scaling Art of Multiprocessor Programming 120 1.8x 2x 2.9x User code Multicore Speedup Not reducing sequential % of code

121. Shared Data Structures 75% Unshared 25% Shared Coarse Grained Fine Grained 75% Unshared 25% Shared

122. Shared Data Structures 75% Unshared 25% Shared Coarse Grained Fine Grained Why only 2.9 speedup 75% Unshared 25% Shared Honk!

123. Shared Data Structures 75% Unshared 25% Shared Coarse Grained Fine Grained Why fine-grained parallelism maters 75% Unshared 25% Shared Honk!

124. Art of Multiprocessor Programming 124 This Course Learn to minimize parallelization overhead of the fraction P that is easy Learn how to introduce parallelism into the 1-P that are hard

125. Art of Multiprocessor Programming Diminishing Returns This course is about the parts that are hard to make concurrent … but still have a big influence on speedup!

126. Grading 10 Homeworks 50% 2 In-class midterms 50% Art of Multiprocessor Programming

127. Collaboration Permitted –talking about the homework problems with other students; using other textbooks; using the Internet. Not Permitted –obtaining the answer directly from anyone else in any form. Art of Multiprocessor Programming

128. Crowdsourcing! You can annotate the textbook online –See something interesting? –Have a question? –Answer a question? –Like or dislike a note? http://nb.mit.edu –Play the video Details to follow … Art of Multiprocessor Programming

129. Capstone Yes, you can take this course as a capstone course There is a fixed project (concurrent packet filter) Requires reading ahead in the course See web page for details Art of Multiprocessor Programming

130. 130 This work is licensed under a Creative Commons Attribution- ShareAlike 2.5 License.Creative Commons Attribution- ShareAlike 2.5 License You are free: –to Share — to copy, distribute and transmit the work –to Remix — to adapt the work Under the following conditions: –Attribution. You must attribute the work to “The Art of Multiprocessor Programming” (but not in any way that suggests that the authors endorse you or your use of the work). –Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license. For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to –http://creativecommons.org/licenses/by-sa/3.0/. Any of the above conditions can be waived if you get permission from the copyright holder. Nothing in this license impairs or restricts the author's moral rights.