1. Distributed Computing Group The Consensus Problem Roger Wattenhofer a lot of kudos to Maurice Herlihy and Costas Busch for some of their slides

2. Distributed Computing Group Roger Wattenhofer2 Sequential Computation memory object thread

3. Distributed Computing Group Roger Wattenhofer3 Concurrent Computation memory object threads

4. Distributed Computing Group Roger Wattenhofer4 Asynchrony Sudden unpredictable delays –Cache misses (short) –Page faults (long) –Scheduling quantum used up (really long)

5. Distributed Computing Group Roger Wattenhofer5 Model Summary Multiple threads –Sometimes called processes Single shared memory Objects live in memory Unpredictable asynchronous delays

6. Distributed Computing Group Roger Wattenhofer6 Road Map We are going to focus on principles –Start with idealized models –Look at a simplistic problem –Emphasize correctness over pragmatism –“Correctness may be theoretical, but incorrectness has practical impact”

7. Distributed Computing Group Roger Wattenhofer7 You may ask yourself … I’m no theory weenie - why all the theorems and proofs?

8. Distributed Computing Group Roger Wattenhofer8 Fundamentalism Distributed & concurrent systems are hard –Failures –Concurrency Easier to go from theory to practice than vice-versa

9. Distributed Computing Group Roger Wattenhofer9 The Two Generals Red army wins If both sides attack together

10. Distributed Computing Group Roger Wattenhofer10 Communications Red armies send messengers across valley

11. Distributed Computing Group Roger Wattenhofer11 Communications Messengers don’t always make it

12. Distributed Computing Group Roger Wattenhofer12 Your Mission Design a protocol to ensure that red armies attack simultaneously

13. Distributed Computing Group Roger Wattenhofer13 Real World Generals Date: Wed, 11 Dec 2002 12:33:58 +0100 From: Friedemann Mattern To: Roger Wattenhofer Subject: Vorlesung Sie machen jetzt am Freitag, 08:15 die Vorlesung Verteilte Systeme, wie vereinbart. OK? (Ich bin jedenfalls am Freitag auch gar nicht da.) Ich uebernehme das dann wieder nach den Weihnachtsferien.

14. Distributed Computing Group Roger Wattenhofer14 Real World Generals Date: Mi 11.12.2002 12:34 From: Roger Wattenhofer To: Friedemann Mattern Subject: Re: Vorlesung OK. Aber ich gehe nur, wenn sie diese Email nochmals bestaetigen... :-) Gruesse -- Roger Wattenhofer

15. Distributed Computing Group Roger Wattenhofer15 Real World Generals Date: Wed, 11 Dec 2002 12:53:37 +0100 From: Friedemann Mattern To: Roger Wattenhofer Subject: Naechste Runde: Re: Vorlesung... Das dachte ich mir fast. Ich bin Praktiker und mache es schlauer: Ich gehe nicht, unabhaengig davon, ob Sie diese email bestaetigen (beziehungsweise rechtzeitig erhalten). (:-)

16. Distributed Computing Group Roger Wattenhofer16 Real World Generals Date: Mi 11.12.2002 13:01 From: Roger Wattenhofer To: Friedemann Mattern Subject: Re: Naechste Runde: Re: Vorlesung... Ich glaube, jetzt sind wir so weit, dass ich diese Emails in der Vorlesung auflegen werde...

17. Distributed Computing Group Roger Wattenhofer17 Real World Generals Date: Wed, 11 Dec 2002 18:55:08 +0100 From: Friedemann Mattern To: Roger Wattenhofer Subject: Re: Naechste Runde: Re: Vorlesung... Kein Problem. (Hauptsache es kommt raus, dass der Prakiker am Ende der schlauere ist... Und der Theoretiker entweder heute noch auf das allerletzte Ack wartet oder wissend das das ja gar nicht gehen kann alles gleich von vornherein bleiben laesst... (:-))

18. Distributed Computing Group Roger Wattenhofer18 Theorem There is no non-trivial protocol that ensures the red armies attacks simultaneously

19. Distributed Computing Group Roger Wattenhofer19 Proof Strategy Assume a protocol exists Reason about its properties Derive a contradiction

20. Distributed Computing Group Roger Wattenhofer20 Proof 1.Consider the protocol that sends fewest messages 2.It still works if last message lost 3.So just don’t send it –Messengers’ union happy 4.But now we have a shorter protocol! 5.Contradicting #1

21. Distributed Computing Group Roger Wattenhofer21 Fundamental Limitation Need an unbounded number of messages Or possible that no attack takes place

22. Distributed Computing Group Roger Wattenhofer22 You May Find Yourself … I want a real-time YAFA compliant Two Generals protocol using UDP datagrams running on our enterprise-level fiber tachyion network...

23. Distributed Computing Group Roger Wattenhofer23 You might say I want a real-time YAFA compliant Two Generals protocol using UDP datagrams running on our enterprise-level fiber tachyion network... Yes, Ma’am, right away!

24. Distributed Computing Group Roger Wattenhofer24 You might say I want a real-time dot-net compliant Two Generals protocol using UDP datagrams running on our enterprise-level fiber tachyion network... Yes, Ma’am, right away! Advantage: Buys time to find another job No one expects software to work anyway Advantage: Buys time to find another job No one expects software to work anyway

25. Distributed Computing Group Roger Wattenhofer25 You might say I want a real-time dot-net compliant Two Generals protocol using UDP datagrams running on our enterprise-level fiber tachyion network... Yes, Ma’am, right away! Advantage: Buys time to find another job No one expects software to work anyway Disadvantage: You’re doomed Without this course, you may not even know you’re doomed Disadvantage: You’re doomed Without this course, you may not even know you’re doomed

26. Distributed Computing Group Roger Wattenhofer26 You might say I want a real-time YAFA compliant Two Generals protocol using UDP datagrams running on our enterprise-level fiber tachyion network... I can’t find a fault-tolerant algorithm, I guess I’m just a pathetic loser.

27. Distributed Computing Group Roger Wattenhofer27 You might say I want a real-time YAFA compliant Two Generals protocol using UDP datagrams running on our enterprise-level fiber tachyion network... I can’t find a fault-tolerant algorithm, I guess I’m just a pathetic loser. Advantage: No need to take course Advantage: No need to take course

28. Distributed Computing Group Roger Wattenhofer28 You might say I want a real-time YAFA compliant Two Generals protocol using UDP datagrams running on our enterprise-level fiber tachyion network... I can’t find a fault-tolerant algorithm, I guess I’m just a pathetic loser Advantage: No need to take course Advantage: No need to take course Disadvantage: Boss fires you, hires University St. Gallen graduate Disadvantage: Boss fires you, hires University St. Gallen graduate

29. Distributed Computing Group Roger Wattenhofer29 You might say I want a real-time YAFA compliant Two Generals protocol using UDP datagrams running on our enterprise-level fiber tachyion network... Using skills honed in course, I can avert certain disaster! Rethink problem spec, or Weaken requirements, or Build on different platform Using skills honed in course, I can avert certain disaster! Rethink problem spec, or Weaken requirements, or Build on different platform

30. Distributed Computing Group Roger Wattenhofer30 Consensus: Each Thread has a Private Input 32 19 21

31. Distributed Computing Group Roger Wattenhofer31 They Communicate

32. Distributed Computing Group Roger Wattenhofer32 They Agree on Some Thread’s Input 19

33. Distributed Computing Group Roger Wattenhofer33 Consensus is important With consensus, you can implement anything you can imagine… Examples: with consensus you can decide on a leader, implement mutual exclusion, or solve the two generals problem

34. Distributed Computing Group Roger Wattenhofer34 You gonna learn In some models, consensus is possible In some other models, it is not Goal of this and next lecture: to learn whether for a given model consensus is possible or not … and prove it!

35. Distributed Computing Group Roger Wattenhofer35 Consensus #1 shared memory n processors, with n > 1 Processors can atomically read or write (not both) a shared memory cell

36. Distributed Computing Group Roger Wattenhofer36 Protocol (Algorithm?) There is a designated memory cell c. Initially c is in a special state “?” Processor 1 writes its value v 1 into c, then decides on v 1. A processor j (j not 1) reads c until j reads something else than “?”, and then decides on that.

37. Distributed Computing Group Roger Wattenhofer37 Unexpected Delay Swapped out back at ???

38. Distributed Computing Group Roger Wattenhofer38 Heterogeneous Architectures ??? Pentium 286 yawn (1)

39. Distributed Computing Group Roger Wattenhofer39 Fault-Tolerance ???

40. Distributed Computing Group Roger Wattenhofer40 Consensus #2 wait-free shared memory n processors, with n > 1 Processors can atomically read or write (not both) a shared memory cell Processors might crash (halt) Wait-free implementation… huh?

41. Distributed Computing Group Roger Wattenhofer41 Wait-Free Implementation Every process (method call) completes in a finite number of steps Implies no mutual exclusion We assume that we have wait-free atomic registers (that is, reads and writes to same register do not overlap)

42. Distributed Computing Group Roger Wattenhofer42 A wait-free algorithm… There is a cell c, initially c=“?” Every processor i does the following r = Read(c); if (r == “?”) then Write(c, v i ); decide v i ; else decide r;

43. Distributed Computing Group Roger Wattenhofer43 Is the algorithm correct? time cell c 32 17 ? ? ? 32 17 32! 17!

44. Distributed Computing Group Roger Wattenhofer44 Theorem: No wait-free consensus ???

45. Distributed Computing Group Roger Wattenhofer45 Proof Strategy Make it simple –n = 2, binary input Assume that there is a protocol Reason about the properties of any such protocol Derive a contradiction

46. Distributed Computing Group Roger Wattenhofer46 Wait-Free Computation Either A or B “moves” Moving means –Register read –Register write A movesB moves

47. Distributed Computing Group Roger Wattenhofer47 The Two-Move Tree Initial state Final states

48. Distributed Computing Group Roger Wattenhofer48 Decision Values 10 0111

49. Distributed Computing Group Roger Wattenhofer49 Bivalent: Both Possible 111 bivalent 10 0

50. Distributed Computing Group Roger Wattenhofer50 Univalent: Single Value Possible 111 univalent 10 0

51. Distributed Computing Group Roger Wattenhofer51 1-valent: Only 1 Possible 0 111 1-valent 0 1

52. Distributed Computing Group Roger Wattenhofer52 0-valent: Only 0 possible 111 0-valent 10 0

53. Distributed Computing Group Roger Wattenhofer53 Summary Wait-free computation is a tree Bivalent system states –Outcome not fixed Univalent states –Outcome is fixed –Maybe not “known” yet –1-Valent and 0-Valent states

54. Distributed Computing Group Roger Wattenhofer54 Claim Some initial system state is bivalent (The outcome is not always fixed from the start.)

55. Distributed Computing Group Roger Wattenhofer55 A 0-Valent Initial State All executions lead to decision of 0 0 0

56. Distributed Computing Group Roger Wattenhofer56 A 0-Valent Initial State Solo execution by A also decides 0 0

57. Distributed Computing Group Roger Wattenhofer57 A 1-Valent Initial State All executions lead to decision of 1 1 1

58. Distributed Computing Group Roger Wattenhofer58 A 1-Valent Initial State Solo execution by B also decides 1 1

59. Distributed Computing Group Roger Wattenhofer59 A Univalent Initial State? Can all executions lead to the same decision? 0 1

60. Distributed Computing Group Roger Wattenhofer60 State is Bivalent Solo execution by A must decide 0 Solo execution by B must decide 1 0 1

61. Distributed Computing Group Roger Wattenhofer61 0-valent Critical States 1-valent critical

62. Distributed Computing Group Roger Wattenhofer62 Critical States Starting from a bivalent initial state The protocol can reach a critical state –Otherwise we could stay bivalent forever –And the protocol is not wait-free

63. Distributed Computing Group Roger Wattenhofer63 From a Critical State c If A goes first, protocol decides 0 If B goes first, protocol decides 1 0-valent 1-valent

64. Distributed Computing Group Roger Wattenhofer64 Model Dependency So far, memory-independent! True for –Registers –Message-passing –Carrier pigeons –Any kind of asynchronous computation

65. Distributed Computing Group Roger Wattenhofer65 What are the Threads Doing? Reads and/or writes To same/different registers

66. Distributed Computing Group Roger Wattenhofer66 Possible Interactions x.read()y.read()x.write()y.write() x.read() ???? y.read() ???? x.write() ???? y.write() ????

67. Distributed Computing Group Roger Wattenhofer67 Reading Registers A runs solo, decides 0 B reads x 1 0 A runs solo, decides 1 c States look the same to A

68. Distributed Computing Group Roger Wattenhofer68 Possible Interactions x.read()y.read()x.write()y.write() x.read() no y.read() no x.write() no ?? y.write() no ??

69. Distributed Computing Group Roger Wattenhofer69 Writing Distinct Registers A writes yB writes x 10 c The song remains the same A writes y B writes x

70. Distributed Computing Group Roger Wattenhofer70 Possible Interactions x.read()y.read()x.write()y.write() x.read() no y.read() no x.write() no ? y.write() no ?

71. Distributed Computing Group Roger Wattenhofer71 Writing Same Registers States look the same to A A writes x B writes x 1 A runs solo, decides 1 c 0 A runs solo, decides 0 A writes x

72. Distributed Computing Group Roger Wattenhofer72 That’s All, Folks! x.read()y.read()x.write()y.write() x.read() no y.read() no x.write() no y.write() no

73. Distributed Computing Group Roger Wattenhofer73 Theorem It is impossible to solve consensus using read/write atomic registers –Assume protocol exists –It has a bivalent initial state –Must be able to reach a critical state –Case analysis of interactions Reads vs others Writes vs writes

74. Distributed Computing Group Roger Wattenhofer74 What Does Consensus have to do with Distributed Systems?

75. Distributed Computing Group Roger Wattenhofer75 We want to build a Concurrent FIFO Queue

76. Distributed Computing Group Roger Wattenhofer76 With Multiple Dequeuers!

77. Distributed Computing Group Roger Wattenhofer77 A Consensus Protocol 2-element array FIFO Queue with red and black balls 8 Coveted red ballDreaded black ball

78. Distributed Computing Group Roger Wattenhofer78 Protocol: Write Value to Array 01 0

79. Distributed Computing Group Roger Wattenhofer79 0 Protocol: Take Next Item from Queue 01 8

80. Distributed Computing Group Roger Wattenhofer80 01 Protocol: Take Next Item from Queue I got the coveted red ball, so I will decide my value I got the dreaded black ball, so I will decide the other’s value from the array 8

81. Distributed Computing Group Roger Wattenhofer81 Why does this Work? If one thread gets the red ball Then the other gets the black ball Winner can take her own value Loser can find winner’s value in array –Because threads write array before dequeuing from queue

82. Distributed Computing Group Roger Wattenhofer82 Implication We can solve 2-thread consensus using only –A two-dequeuer queue –Atomic registers

83. Distributed Computing Group Roger Wattenhofer83 Implications Assume there exists –A queue implementation from atomic registers Given –A consensus protocol from queue and registers Substitution yields –A wait-free consensus protocol from atomic registers contradiction

84. Distributed Computing Group Roger Wattenhofer84 Corollary It is impossible to implement a two- dequeuer wait-free FIFO queue with read/write shared memory. This was a proof by reduction; important beyond NP-completeness…