1. Distributed Algorithms – 2g1513 Lecture 10 – by Ali Ghodsi Fault-Tolerance in Asynchronous Networks

2. 2 Consensus Problems Consensus problems very important in DS  Distributed Databases All processes must agree whether to commit or abort a transaction If any process says abort, all processes should abort  Atomic Broadcast All processes receive the same set of messages coming from correct processes only Can be used to implement consensus, vice versa

3. 3 Fischer, Lynch, Paterson 1983/85 Consensus cannot be solved in asynchronous model  With possibility of one process crashing http://www.sics.se/~ali/flp85.pdf Most influential paper award PODC 2001

4. 4 Modified Model To proof the result, we will modify our model of a distributed system slightly Processes execute local algorithms, modeled by a STS But, given any state, a correct process can always execute a “dummy” instruction  For any state  in a process, there exists a transition     There exists always an applicable event on every process A crashed process, cannot make any transitions

5. 5 Definition: T-crash fair executions A t-crash-robust algorithm is a consensus algorithm if it satisfies:  Termination All correct processes eventually decides  Agreement In every configuration, the decided processes should have decided for the same value (0 or 1)  Non-triviality There exists at least one possible input configuration where the decision is 0 There exists at least one possible input configuration where the decision is 1  Example, maybe input “0,0,1”->0 while “0,1,1”->1

6. 6 Definitions 0-decided configuration  A configuration with decide ”0” on some process 1-decided configuration  A configuration with decide ”1” on some process 0-valent configuration  A configuration in which every reachable decided configuration is a 0-decide 1-valent configuration  A configuration in which every reachable decided configuration is a 1-decide Bivalent configuration  A configuration which can reach a 0-decided and 1-decided configuration

7. 7 Definitions Illustrated 1(4) 0-decided configuration  A configuration with decide ”0” on some process 0-decided configuration { STATE2, STATE,5 DECIDE-0, STATE7 {msg1, msg2} } At least of them is in state DECIDE-0 msg1 msg2 P1 state2 P2 state5 P4 state7 P3 decide0

8. 8 Definitions Illustrated 2(4) 0-valent configuration  No 1-decided configurations are reachable  Future determined, means ”everyone will decide 0” 0- valent configuration {P1_state, P2_state, P3_state, P4_state, {msg1} } 0-valent configuration {P1_state, P2_state2, P3_state, P4_state, {msg1} } 0-valent configuration {decide-0, P2_state, P3_state, P4_state, {msg1, msg2} } 0-valent configuration {decide-0, P2_state2, P3_state2, P4_state, {msg1, msg2} } 0-valent configuration {decide-0, P2_state, P3_state, decide-0, { msg2} } 0-valent configuration {decide-0, P2_state2, P3_state2, decide-0, { msg2} } 0-valent configuration {decide-0, P2_state, decide-0, P4_state, {msg1, msg2} } 0-valent configuration {decide-0, P2_state3, P3_state, decide-0, {} }

9. 9 Definitions Illustrated 3(4) 1-valent configuration  No 0-decided configurations are reachable  Future determined, means ”everyone will decide 1” 0- valent configuration {P1_state, P2_state, P3_state, P4_state, {msg1} } 0-valent configuration {P1_state, P2_state2, P3_state, P4_state, {msg1} } 0-valent configuration {decide-1, P2_state, P3_state, P4_state, {msg1, msg2} } 0-valent configuration {decide-1, P2_state2, P3_state2, P4_state, {msg1, msg2} } 0-valent configuration {decide-1, P2_state, P3_state, decide-1, { msg2} } 0-valent configuration {decide-1, P2_state2, P3_state2, decide-1, { msg2} } 0-valent configuration {decide-1, P2_state, decide-1, P4_state, {msg1, msg2} } 0-valent configuration {decide-1, P2_state3, P3_state, decide-1, {} }

10. 10 Definitions Illustrated 4(4) Bivalent configuration  Both 0 and 1-decided configurations are reachable  Future undetermined, could go either way… bivalent configuration {P1_state, P2_state, P3_state, P4_state, {msg1} } 0-valent configuration {P1_state, P2_state2, P3_state, P4_state, {msg1} } 1-valent configuration {decide-1, P2_state5, P3_state6, P4_state5, {msg1, msg3} } 0-valent configuration {decide-0, P2_state2, P3_state2, P4_state, {msg1, msg2} } 1-valent configuration {decide-1, P2_state5, P3_state6, decide-1, { msg2} } 0-valent configuration {decide-0, P2_state2, P3_state2, decide-0, { msg2} } 0-valent configuration {decide-0, P2_state, decide-0, P4_state, {msg1, msg2} } 1-valent configuration {decide-1, P2_state9, P3_state6, decide-1, {} }

11. 11 Bivalent Initial Configuration Theorem  For any algorithm that solves the 1-crash consensus problem there exists an initial bivalent configuration

12. 12 Proof 1/(10) We know that the algorithm must be non-trivial  There should be some initial configuration that will lead to a 0-decide  There should be some initial configuration that will lead to a 1-decide Take two such configuration i 1 and i 2  E.g. 4 processes initial values (0,1,0,1,1) lead to 1 Initial values (0,0,1,0,0) lead to 0

13. 13 Proof 2/(10) We know there exists inputs p 1, p 2, p 3, p 4, p 5  (0,1,0,1,1) leading to 1  (0,0,1,0,0) leading to 0 Lets look at other initial configurations by flipping the inputs transforming the upper input to the lower input

14. 14 Proof 3/(10) We know there exists inputs p 1, p 2, p 3, p 4, p 5  (0,1,0,1,1) leading to 1  (0,0,0,1,1) leading to ?  (0,0,1,0,0) leading to 0 Lets look at other initial configurations by flipping the inputs transforming the upper input to the lower input

15. 15 Proof 4/(10) We know there exists inputs p 1, p 2, p 3, p 4, p 5  (0,1,0,1,1) leading to 1  (0,0,0,1,1) leading to ?  (0,0,1,1,1) leading to ?  (0,0,1,0,0) leading to 0 Lets look at other initial configurations by flipping the inputs transforming the upper input to the lower input

16. 16 Proof 5/(10) We know there exists inputs p 1, p 2, p 3, p 4, p 5  (0,1,0,1,1) leading to 1  (0,0,0,1,1) leading to ?  (0,0,1,1,1) leading to ?  (0,0,1,0,1) leading to ?  (0,0,1,0,0) leading to 0 Lets look at other initial configurations by flipping the inputs transforming the upper input to the lower input

17. 17 Proof 6/(10) We know there exists inputs p 1, p 2, p 3, p 4, p 5  (0,1,0,1,1) leading to 1  (0,0,0,1,1) leading to ?  (0,0,1,1,1) leading to ?  (0,0,1,0,1) leading to ?  (0,0,1,0,0) leading to 0 There must exist two neighboring configurations here, with two different outcomes Lets look at other initial configurations by flipping the inputs transforming the upper input to the lower input

18. 18 Proof 7/(10) We know there exists inputs p 1, p 2, p 3, p 4, p 5  (0,1,0,1,1) leading to 1  (0,0,0,1,1) leading to 1  (0,0,1,1,1) leading to 1  (0,0,1,0,1) leading to 0  (0,0,1,0,0) leading to 0 Assume the following two Lets look at other initial configurations by flipping the inputs

19. 19 Proof 8/(10) We know there exists inputs p 1, p 2, p 3, p 4, p 5  (0,1,0,1,1) leading to 1  (0,0,0,1,1) leading to 1  (0,0,1,1,1) leading to 1  (0,0,1,0,1) leading to 0  (0,0,1,0,0) leading to 0 Assume the following two Identical configurations except for process p 4

20. 20 Proof 9/(10) We know there exists inputs p 1, p 2, p 3, p 4, p 5  (0,0,1,1,1) leading to 1  (0,0,1,0,1) leading to 0 The consensus algorithm should tolerate if p 4 crashes!  (0,0,1,X,1), leads to ? (either 0 or 1) Assume the following two

21. 21 Proof 10/(10) We know there exists inputs p 1, p 2, p 3, p 4, p 5  (0,0,1,1,1) leading to 1  (0,0,1,0,1) leading to 0 The consensus algorithm should tolerate if p 4 crashes!  (0,0,1,X,1), leads to ? (either 0 or 1)  If it leads to 1, then depending on whether p 4 crashes or not (0,0,1,0,1) either leads to 0 or 1 (bivalent)  If it leads to 0, then depending on whether p 4 crashes or not (0,0,1,1,1) either leads to 0 or 1 (bivalent) Assume the following two

22. 22 Initial Bivalence Intuition  Given any algorithm, we can find some start state, that depending on the failure of one process, will either lead to a 0-decide or a 1-decide Bivalent Initial Config {P1_state, P2_state, P3_state, P4_state, {msg1} } 1-valent configuration {P1_state, P2_state2, P3_state, P4_state, {msg1} } 0-valent configuration {P1_state, P2_state, P3_state, P4_state, {msg1, msg2} } 1-valent configuration {decide-1, P2_state2, P3_state2, P4_state, {msg1, msg2} } 0-valent configuration {decide-0, P2_state, P3_state, P4_state, { msg2} } 1-valent configuration {P1_state, P2_state, decide-1, P4_state, {msg1, msg2} } 0-valent configuration {decide-0, decide-0, P3_state, decide-0, {} }

23. 23 Coarse-grained Model of Distributed Systems In our model, we will now let each event be the receipt of a message  After the receipt of a message m, a process deterministically makes all internal and send events it can do  In other words, we make our course-grained model a bit more fine-grained An event represents the receipt of a message, some internal transitions and the sending of some messages  A receipt of message m at process p is always applicable if a message m with destination p is in the network

24. 24 Intuition behind model receive from q for x:=1 to 3 do begin y:=y+1; send neigh p [x]; end receive from q; print z+y Receipt event e Initial state of p State of p after receipt of e Deterministic transitions Receipt event f Deterministic transitions State of p after receipt of f

25. 25 Order of events Intuition  The order in which two applicable events are executed is not important! Order Theorem  Let e p and e q be two events on two different processors p and q which are both applicable in configuration . Then e p can be applied to e q (  ), and e q can be applied to e p (  ).  Moreover, e p ( e q (  )) = e q ( e p (  ) ).

26. 26 Definitions A sequence of events  =( e 1, e 2,…,e k ) is applicable in configuration  if  e 1 is applicable in ,  e 2 applicable in e 1 (  ) ... If the resulting configuration is  we write  (  )=  or     If  only contains events of a subset of the processes P, we write  P 

27. 27 Order of sequences Diamond Theorem  Let sequences  1 and  2 be applicable in configuration , and let no process participate in both  1 and  2. Then  2 is applicable in  1 (  ),  2 is applicable in  2 (  ), and  1 (  2 (  ))=  2 (  1 (  )) Proof  By induction using the order theorem

28. 28 Illustration of the Diamond Theorem  11 22 1()1() 2()2()  22 11  =  2 (  1 (  ) )=  1 (  2 (  ))

29. 29 Bivalent Configuration Any configuration of the 1-robust consensus algorithm is exactly one of these three  Bivalent  0-valent  1-valent Why?  Any configuration leads to a decide because of termination  We know bivalent configurations exist  If it is not bivalent, it must lead to either 0-decide or 1- decide, so it is either 0-valent or 1-valent

30. 30 Bivalent Configurations In any bivalent config , either  one applicable event goes to a bivalent config, or  there exists two applicable events, leading to a 0- valent and 1-valent configurations (respectively) Bivalent 1-valent 0-valent Case 1Case 2

31. 31 Staying Bivalent Theorem  Given any bivalent config  and an event e applicable in  There exists another reachable config  where e is applicable, and e (  ) is bivalent Bivalent … Theorem Illustration e Bivalent … e … … e  

32. 32 Proof definitions Assume e involves process p Call the set of all possible configs reachable from  without applying e the set C Apply event e to all configs in C and call the resulting configs D Bivalent … e Theorem Illustration … … … … … … … e e … … e … e C D … e

33. 33 Proof intuition We will proof that D contains a bivalent config by contradiction I.e., assume there exists no bivalent config in D, show that this will lead to a contradiction or absurdity Bivalent … e Theorem Illustration … … … … … … … … e e e … … e … e C D

34. 34 Proof Assume D contains no bivalent configs  I.e. all configs in D are either 0-valent or 1-valent Then it follows that there exists a 0-valent and a 1-valent config in D (next slides)

35. 35 Proof We know we can reach a 0-valent and 1-valent config from , c all them  1 and  2 (non-triviality) Either  1 and  2 are in C or they are not in C If inside C, then e (  1 ) and e (  2 ) is in D and they are 0-valent/1-valent Bivalent … e  1 and  2 are in C  1 and  2 are not in C 11 … … … 22 … … … e e e … … e … e C Bivalent … e … … … 22 11 … e e e … … e … e C

36. 36 Proof If not inside C, then some  1 and  2 exists on the path to  1 and  2, such that e (  1 ) and e (  2 ) are in D and they are 0-valent/1-valent [Remember we assumed no bivalent config available in D] Bivalent … e  1 and  2 are in C  1 and  2 are not in C 11 … … … 22 … … … e e e … … e … e C Bivalent … e  1 … … …  2 22 11 … e e e … … e … e C

37. 37 Reflection We now know that D must always contain a 0-valent and 1-valent config, assuming no bivalent config exists in D Lets call the two 0-valent and 1-valent configs in D, d 0 and d 1 We will now show that this situation is a contradiction itself. Hence, D must contain a bivalent config

38. 38 f Deriving the contradiction There must exist two configs c 0 and c 1 in C such that c 1 = f ( c 0 ), and d 0 = e ( c 0 ) and d 1 = e ( c 1 ) c0c0 c1c1 d0d0 d1d1 e e C D Lets see why!

39. 39 Proofing two neighbors exist 1(4) We know  is bivalent, and e (  ) is in D and is either 0-valent or 1- valent, assume 0-valent  0-valent e C D

40. 40 Proofing two neighbors exist 2(4) We know  is bivalent, and e (  ) is in D and is either 0-valent or 1- valent, assume 0-valent There is a reachable 1-valent config in D f0f0  11 0-valent e e C 22 … mm 1-valent D

41. 41 Proofing two neighbors exist 3(4) We know  is bivalent, and e (  ) is in D and is either 0-valent or 1- valent, assume 0-valent There is a reachable 1-valent config in D e is applicable in each  i, and must be 0-valent or 1-valent  11 0-valent 1-valent e e C 22 … mm x-valent y-valent z-valent D eee f0f0

42. 42 There exists two neighbors, one 1- valent and one 0- valent Proofing two neighbors exist 4(4)  11 0-valent 1-valent e e C 22 … mm 0-valent 1-valent z-valent D eee f0f0 f1f1 f2f2 f3f3 We know  is bivalent, and e (  ) is in D and is either 0-valent or 1- valent, assume 0-valent There is a reachable 1-valent config in D e is applicable in each  i, and must be 0-valent or 1-valent

43. 43 There exists two neighbors, one 1- valent and one 0- valent Proofing two neighbors exist 4(4) We know  is bivalent, and e (  ) is in D and is either 0-valent or 1- valent, assume 0-valent There is a reachable 1-valent config in D e is applicable in each  i, and is 0/1-valent f 11 C 22 0-valent 1-valent D ee

44. 44 There exists two neighbors, one 1- valent and one 0- valent Neighbors lead to contradiction 1(3) We now know there exist two configs c 0 and c 1 in C such that c 1 = f ( c 0 ), and d 0 = e ( c 0 ) and d 1 = e ( c 1 ) Either the events e and f happen on the same processor or on different processors, both cases will lead to contradictions f 11 C 22 0-valent 1-valent D ee

45. 45 Neighbors lead to contradiction 2(3) We now know there exist two configs c 0 and c 1 in C such that c 1 = f ( c 0 ), and d 0 = e ( c 0 ) and d 1 = e ( c 1 ) Assume e and f happen on two different processes p and q  Then, the order of their execution can be exchanged f c0c0 c1c1 d1d1 e e C D 0-valent1-valent f d0d0 Contradiction as d 0 is 0-valent, but it can lead to a 1- valent config, hence d 0 must be bivalent, but we assumed no bivalent configs exist in D

46. 46 Neighbors lead to contradiction 3(3) We now know there exist two configs c 0 and c 1 in C such that c 1 = f ( c 0 ), and d 0 = e ( c 0 ) and d 1 = e ( c 1 ) Assume e and f happen on the same process p, the algorithm should still work if p is silent f c0c0 c1c1 d1d1 e e C 0-valent1-valent d0d0 Contradiction as A should be a 0/1-valent configuration, but we have shown that A can lead to both  0 and  1 f 22 e e A If p is silent, the algorithm should continue and terminate with a decision in some config A 00 If p is silent, some execution leading to  0 should exist 11 If p is silent, some execution leading to  1 should exist

47. 47 Proof Map Assume there is no bivalent config in D  We know all configs in D are 0-valent or 1-valent  Show that we can find a 0-valent and 1-valent config in D Show that two neighboring configs c 0 ─ e → c 1 exist, where c 0 ─ f →”0-valent config”, c 1 ─ f →”1-valent config” Show this is a contradiction Assumption must be incorrect D must contain a bivalent configuration

48. 48 Final Theorem No deterministic 1-crash-robust consensus algorithm exists for the asynchronous model Proof 1.Start in a initial bivalent config 2.Given the bivalent config, pick the event e that has been applicable longest Pick the execution taking us to another config where e is applicable Apply e, and get a bivalent config 3.Repeat 2.

49. 49 Consensus not Impossible! Lets do deterministic consensus algorithm for the a different failure model  Initially dead processes Assume t failures can happen initially  Where t=4 for N=10, t=5 for N=11  Let L denote L=6 for N=10, L=6 for N=11 N=t+LN=t+L

50. 50 Intuition Assume N processes are connected in a underlying graph, and at most t fail We know L processes are alive after the start  Broadcast your identity, and receive/collect L identities For any two correct processes, their set of collected identities will overlap  Quorom concept  There are N nodes, any two processes have L identities each, i.e. total  N+1 identities, total N nodes, at least two must be same (PHP)

51. 51 Initially Dead Consensus Receive L messages Initial state of p Any two processes have overlapping Succ Keep identity of senders in Rcvd Wait until you’ve received a message from every process that is transitively in each Succ Every process has the same set Alive

52. 52 Summary We have proved that a 1-crash resilient deterministic consensus algorithm does not exist  Hence, there exists always an execution which stays in bivalent configurations and still keeps applying all applicable events!  All correct processes execute infinite number of events, and still leads to no decision! We have shown an algorithm for consensus which is for the initially dead processes model