1. Sergio Rajsbaum 2006 Lecture 4 Introduction to Principles of Distributed Computing Sergio Rajsbaum Math Institute UNAM, Mexico

2. Sergio Rajsbaum 2006 Lecture 4 Consensus in partially synchronous systems, and failure detectors Part I: Realistic timing model and metric Part II: Failure detectors, algorithms Part III: this is the best possible Part IV: New directions and extensions

3. Sergio Rajsbaum 2006 CONSENSUS A fundamental Abstraction Each process has an input, should decide an output s.t. Agreement: correct processes’ decisions are the same Validity: decision is input of one process Termination: eventually all correct processes decide There are at least two possible input values 0 and 1. all possible vectors over the input values V

4. Sergio Rajsbaum 2006 The lecture in a nutshell Consensus solvability depends on how long connectivity preserved by a particular model In synchronous it is solvable, in asynchronous not. What about intermediate, more realistic models? X0X0 L(X 0 ) L 2 (X 0 ) Initial states states after one round states after 2 rounds Connectivity preserved Connectivity destroyed

5. Sergio Rajsbaum 2006 Basic Model Message passing (essentially equivalent to read/write shared memory model) Channels between every pair of processes Crash failures t 1 processes No message loss among correct processes

6. Sergio Rajsbaum 2006 Is consensus solvable? If so, how long does it take to solve it? It depends on what exactly the model is But what is a realistic model? And what are the common scenarios within the model? The nature of a distributed system is to include complex combinations of failures and delays

7. Sergio Rajsbaum 2006 How Fast Can We Solve Consensus? Depends on the timing model: Message delays Processing times Clocks And on the metric used: Worst case Average etc

8. Sergio Rajsbaum 2006 The Rest of This Lecture Part I: Realistic timing model and metric Part II: Upper bounds Part III: this is the best possible Part IV: New directions and extensions

9. Sergio Rajsbaum 2006 Part I: Realistic Timing Model

10. Sergio Rajsbaum 2006 First two simple models

11. Sergio Rajsbaum 2006 Asynchronous Model Unbounded message delay, processor speed Consensus impossible even for t=1 [FLP85]

12. Sergio Rajsbaum 2006 Round Synchronous Model Algorithm runs in synchronous rounds: –send messages to any set of processes, –receive messages from previous round, –do local processing (possibly decide, halt) If process i crashes in a round, then any subset of the messages i sends in this round can be lost

13. Sergio Rajsbaum 2006 Synchronous Consensus In a run with f failures (f<t) –Processes can decide in f+1 rounds [Lamport Fischer 82; Dolev, Reischuk, Strong 90] (early-deciding) 1 round with no failures In this talk deciding – halting takes min(f+2,t+1) [Dolev, Reischuk, Strong 90]

14. Sergio Rajsbaum 2006 The Middle Ground Many real networks are neither synchronous nor asynchronous During long stable periods, delays and processing times are bounded –Like synchronous model Some unstable periods –Like asynchronous model

15. Sergio Rajsbaum 2006 Partial Synchrony Model [Dwork, Lynch, Stockmeyer 88] Processes have clocks (with bounded drift)  upper bound on message delay , upper bound on processing time GST, global stabilization time –Until GST, unstable: bounds do not hold –After GST, stable: bounds hold –GST unknown

16. Sergio Rajsbaum 2006 Partial Synchrony in Practice For , , choose bounds that hold with high probability Stability forever? –We assume that once stable remains stable –In practice, has to last “long enough” for given algorithm to terminate –A commonly used model that alternates between stable and unstable times: Timed Asynchronous Model [Cristian, Fetzer 98]

17. Sergio Rajsbaum 2006 Consensus with Partial Synchrony Solvable requires t < n/2 [DLS88] Unbounded running time by [FLP85], because model can be asynchronous for unbounded time

18. Sergio Rajsbaum 2006 Exercise Prove that consensus is not solvable in the partially synchronous model, if t ≥ n/2 Prove that if t<n/2, it takes unbounded running time to be solved

19. Sergio Rajsbaum 2006 In a Practical System Can we say more than: consensus will be solved eventually ?

20. Sergio Rajsbaum 2006 Performance Metric Number of rounds in well-behaved runs Well-behaved: –No failures –Stable from the beginning Motivation: common case

21. Sergio Rajsbaum 2006 The Rest of This Lecture Part II: best known algorithms decide in 2 rounds in well-behaved runs –2  time (with delay bound , 0 processing time) Part III: this is the best possible Part IV: new directions and extensions

22. Sergio Rajsbaum 2006 Part II: Algorithms, and the Failure Detector Abstraction II.a Failure Detectors and Partial Synchrony II.b Algorithms -=

23. Sergio Rajsbaum 2006 Time-Free Algorithms Goal: abstract away time, get simpler algorithms We describe the algorithms using failure detector abstraction [Chandra, Toueg 96]

24. Sergio Rajsbaum 2006 Unreliable Failure Detectors [Chandra, Toueg 96] Each process has local failure detector oracle –Typically outputs list of processes suspected to have crashed at any given time Unreliable: failure detector output can be arbitrary for unbounded (finite) prefix of run

25. Sergio Rajsbaum 2006 Performance of Failure Detector Based Consensus Algorithms Implement a failure detector in the partial synchrony model Design an algorithm for the failure detector Analyze the performance in well-behaved runs of the combined algorithm

26. Sergio Rajsbaum 2006 A Natural Failure Detector Implementation in Partial Synchrony Model Implement failure detector using timeouts: –When expecting a message from a process i, wait  clock skew before suspecting i In well-behaved runs,  always hold, hence no false suspicions

27. Sergio Rajsbaum 2006 The resulting failure detector is <>P - Eventually Perfect Strong Completeness: From some point on, every faulty process is suspected by every correct process Eventual Strong Accuracy: From some point on, every correct process is not suspected

28. Sergio Rajsbaum 2006 Weakest Failure Detectors for Consensus <>S - Eventually Strong –Strong Completeness –Eventual Weak Accuracy: From some point on, some correct process is not suspected  - Leader –Outputs one trusted process –From some point, all correct processes trust the same correct process

29. Sergio Rajsbaum 2006 A Simple  Implementation Use <>P implementation Output lowest id non-suspected process In well-behaved runs: process 1 always trusted

30. Sergio Rajsbaum 2006 Exercise Write the algorithm code for this failure detector  and prove it is correct

31. Sergio Rajsbaum 2006 Relationships among Failure Detector Classes <>S is a subset of <>P <>S is strictly weaker than <>P <>S ~  [Chandra, Hadzilacos, Toueg 96] Food for thought: What is the weakest timing model where <>S and/or  are implementable but <>P is not?

32. Sergio Rajsbaum 2006 Relationships among Failure Detector Classes- Recent Results Partial Answer: In PODC’03 Aguilera et al present a system with synchronous processes S : –any number of them may crash, and –only the output links of an unknown correct process are eventually timely (all other links can be asynchronous and/or lossy) <>P is not implementable in S,  yes New proof that: <>S is strictly weaker than <>P

33. Sergio Rajsbaum 2006 Note on the Power of Consensus Consensus cannot implement <>P, interactive consistency, atomic commit, … So its “universality”, in the sense of –wait-free objects in shared memory [Herlihy 93] –state machine replication [Lamport 78; Schneider 90] does not cover sensitivity to failures, timing, etc.

34. Sergio Rajsbaum 2006 Other Failure Detector Implementations Food for thought: When is building <>P more costly than <>S or  ? Partial answer: Aguilera at al PODC’03 observe –any implementation of <>P (even in a perfectly synchronous system) requires all alive processes to send messages forever, while  can be implemented such that eventually only the leader sends messages

35. Sergio Rajsbaum 2006 Other Failure Detector Implementations Message efficient <>S implementation [Larrea, Fernández, Arévalo 00] QoS tradeoffs between accuracy and completeness [Chen, Toueg, Aguilera 00] Leader Election [Aguilera, Delporte, Fauconnier, Toueg 01] Adaptive <>P [Fetzer, Raynal, Tronel 01]

36. Sergio Rajsbaum 2006 Part II: Algorithms, and the Failure Detector Abstraction II.a Failure Detectors and Partial Synchrony II.b Algorithms

37. Sergio Rajsbaum 2006 Algorithms that Take 2 Rounds in Well-Behaved Runs <>S-based [ Schiper 97; Hurfin, Raynal 99 ; Mostefaoui, Raynal 99 ]  -based for t < n/3 [ Mostefaoui, Raynal 00]  -based for t < n/2 [ Dutta, Guerraoui 01] Paxos (optimized version) [Lamport 89; 96] –Leader-based (  ) –Also tolerates omissions, crash recoveries COReL - Atomic Broadcast [Keidar, Dolev 96] –Group membership based (<>P)

38. Sergio Rajsbaum 2006 Of This Laundry List, We Present Two Algorithms 1<>S-based [MR99] 2Paxos

39. Sergio Rajsbaum 2006 <>S-based Consensus [MR99] val  input v; est  null for r =1, 2, … do coord  (r mod n)+1 if I am coord, then send (r,val) to all wait for ( (r, val) from coord OR suspect coord (by <>S)) if receive val from coord then est  val else est  null send (r, est) to all wait for (r,est) from n-t processes if any non-null est received then val  est if all ests have same v then send (“decide”, v) to all; return(v) od Upon receive (“decide”, v), forward to all, return(v) 1 2

40. Sergio Rajsbaum 2006 In Well-Behaved Runs 11 2 n...... (1, v 1 ) 1 2 n...... est = v 1 decide v 1

41. Sergio Rajsbaum 2006 In Case of Omissions The algorithm can block in case of transient message omissions, waiting for a specific round message that will not arrive

42. Sergio Rajsbaum 2006 Paxos [Lamport 88; 96; 01] Uses  failure detector Phase 1: prepare –A process who trusts itself tries to become leader –Chooses largest unique (using ids) ballot number –Learns outcome of all smaller ballots Phase 2: accept –Leader proposes a value with his ballot number. –Leader gets majority to accept his proposal. –A value accepted by a majority can be decided

43. Sergio Rajsbaum 2006 Paxos - Variables Type Rank –totally ordered set with minimum element r 0 Variables: Rank BallotNum, initially r 0 Rank AcceptNum, initially r 0 Value  {  } AcceptVal, initially 

44. Sergio Rajsbaum 2006 Paxos Phase I: Prepare Periodically, until decision is reached do: if leader (by  ) then BallotNum  (unique rank > BallotNum) send (“prepare”, rank) to all Upon receive (“prepare”, rank) from i if rank > BallotNum then BallotNum  rank send (“ack”, rank, AcceptNum, AcceptVal) to i

45. Sergio Rajsbaum 2006 Paxos Phase II: Accept Upon receive (“ack”, BallotNum, b, val) from n-t if all vals =  then myVal = initial value else myVal = received val with highest b send (“accept”, BallotNum, myVal) to all /* proposal */ Upon receive (“accept”, b, v) with b  BallotNum AcceptNum  b; AcceptVal  v /* accept proposal */ send (“accept”, b, v) to all (first time only)

46. Sergio Rajsbaum 2006 Paxos – Deciding Upon receive (“accept”, b, v) from n-t decide v periodically send (“decide”, v) to all Upon receive (“decide”, v) decide v

47. Sergio Rajsbaum 2006 In Well-Behaved Runs 11 2 n...... (“accept”,1,v 1 ) 1 2 n...... 11 2 n...... (“prepare”,1) (“ack”,1,r 0,  ) decide v 1 (“accept”,1,v 1 ) Our  implementation always trusts process 1

48. Sergio Rajsbaum 2006 Optimization Allow process 1 (only!) to skip Phase 1 –use rank r 0 –propose its own initial value Takes 2 rounds in well-behaved runs Takes 2 rounds for repeated invocations with the same leader

49. Sergio Rajsbaum 2006 What About Message Loss? Does not block in case of a lost message –Phase I can start with new rank even if previous attempts never ended But constant omissions can violate liveness Specify conditional liveness: If n-t correct processes including the leader can communicate with each other then they eventually decide

50. Sergio Rajsbaum 2006 Synchronous Consensus In a run with f failures (f<t) –Processes can decide in f+1 rounds –And no less ! [Lamport Fischer 82; Dolev, Reischuk, Strong 90] (early-deciding) 1 round with no failures In this talk deciding – halting takes min(f+2,t+1) [Dolev, Reischuk, Strong 90]

51. Sergio Rajsbaum 2006 Uniform Consensus Uniform agreement: decision of every two processes is the same Recall: with consensus, only correct processes have to agree (disagreement with the dead is OK) This version of consensus will be useful to extend the lower bound argument to asynchronous models

52. Sergio Rajsbaum 2006 Synchronous Uniform Consensus Every algorithm has a run with f failures (f<t-1), that takes at least f+2 rounds to decide [Charron-Bost, Schiper 00; KR 01] –as opposed to f+1 for consensus

53. Sergio Rajsbaum 2006 A Simple Proof of the Uniform Consensus Synchronous Lower Bound [Keidar, Rajsbaum IPL 02]

54. Sergio Rajsbaum 2006 Theorem: f+2 Lower Bound Assume n>t, and f < t-1 L f (X 0 ) - final states of runs with  f failures –connected –in any state in L f (X 0 ) exist at least 3 non-failed processes and 2 can fail Take z, z’  X 0 s.t. val(z)  val(z’), –let x, x’ be failure-free extensions of z, z’: x=z.(i,[0]) f  L f (X 0 )

55. Sergio Rajsbaum 2006 Exercise 1.Consider Modify the theorem and the proof of this talk for the consensus problem (instead of the uniform consensus problem)

56. Sergio Rajsbaum 2006 Upper Bounds From Part II We saw that there are algorithms that take 2 rounds to decide in well-behaved runs <>S-based,  -based, Paxos, COReL Presented two of them.

57. Sergio Rajsbaum 2006 Why are there no 1-Round Algorithms? There is a lower bound of 2 rounds in well- behaved executions –Similar bounds shown in [Dwork, Skeen 83; Lamport 00] We will show that the bound follows from a similar bound on Uniform Consensus in the synchronous model

58. Sergio Rajsbaum 2006 Uniform Consensus Uniform agreement: decision of every two processes is the same Recall: with consensus, only correct processes have to agree

59. Sergio Rajsbaum 2006 From Consensus to Uniform Consensus In partial synchrony model, any algorithm A for consensus solves uniform consensus [Guerraoui 95] Proof: Assume by contradiction that A does not solve uniform consensus –in some run, p,q decide differently, p fails –p may be non-faulty, and may wake up after q decides

60. Sergio Rajsbaum 2006 Synchronous Uniform Consensus Every algorithm has a well-behaved run that takes 2 rounds to decide More generally, it has a run with f failures (f<t-1), that takes at least f+2 rounds to decide [Charron-Bost, Schiper 00; KR 01] –as opposed to f+1 for consensus

61. Sergio Rajsbaum 2006 Bibliography Keidar and Rajsbaum, “A Simple Proof of the Uniform Consensus Synchronous Lower Bound,” in IPL, Vol. 85, pp. 47-52, 2003. Keidar and Rajsbaum, “On the Cost of Fault-Tolerant Consensus When There Are No Faults” in Keidar’s page, including slides and papers. Moses, Rajsbaum, “A Layered Analysis of Consensus,” SIAM J. Comput. 31(4): 989-1021, 2002. Mostéfaoui, Rajsbaum, Raynal: Conditions on input vectors for consensus solvability in asynchronous distributed systems. J. ACM, 2003

62. Sergio Rajsbaum 2006