 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Principles of Reliable Distributed Systems Lecture 7: Failure Detectors Spring 2006 Dr. Idit Keidar

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Material Chandra and Toueg, Unreliable Failure Detectors for Reliable Distributed Systems. Mostefaoui and Raynal, Solving Consensus using Chandra-Toueg’s Unreliable Failure Detectors: A General Approach. Keidar and Rajsbaum, On the Cost of Fault- Tolerant Consensus When There are no Faults: A Tutorial.

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Fault-Tolerant Asynchronous Consensus is Impossible Every asynchronous fault-tolerant consensus algorithm has a fair execution in which no process decides [ FLP85 ] It is possible to design asynchronous consensus algorithms that don’t always terminate

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Should We Give Up? We can always model a system as synchronous with the right timeout –messages never take more than 2 days But modeling systems as synchronous requires conservative timeouts –problem?

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Motivation: Choosing a Model Example network: –99% of packets arrive within 10 µsec –upper bound of 1000 µsec on message latency What would we choose the round duration for a round-based synchronous system? –implication? We would like to choose a timeout of 10 µsec, but without violating safety…

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring The Middle Ground We can choose timeouts that usually hold –during long stable periods, delays and processing times are bounded like synchronous model –some unstable periods like asynchronous model We can design algorithms that always ensure safety, but ensure liveness only at stable times

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring How Do We Model This? Option 1: Change the problem statement to require conditional liveness –e.g., replace termination with: “if there is a time after which the system is stable (synchronous) then all correct processes eventually decide” Option 2: Change the model –assume there is a Global Stabilization Time (GST) after which the system is stable

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring These Options Are Equivalent If a safety property holds for a given algorithm in a partial synchrony model, the same property also holds for runs of that algorithm in an asynchronous model –because safety properties are prefix-closed The liveness properties are conditional on the existence of GST

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Eventual Synchrony Model [Dwork, Lynch, Stockmeyer 88] Processes have clocks with bounded drift There are upper bounds –  on message delay, and –  on processing time GST, global stabilization time –Until GST, unstable: bounds do not hold –After GST, stable: bounds hold –GST unbounded, unknown

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Eventual Synchrony in Practice For , , choose bounds that hold with high probability Stability forever? –we assume that once stable remains stable –in practice, has to last “long enough” for given algorithm to terminate

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Time-Free Algorithms Describe algorithms using a failure detector abstraction [Chandra, Toueg 96] Goal: abstract away time, get simpler algorithms What makes a good abstraction?

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring The Failure Detector Abstraction [Chandra, Toueg 96] Each process has local failure detector oracle –typically outputs list of processes suspected to have crashed at any given time In each execution step, a process –receives a message (if there is one ready) –queries its failure detector oracle –makes a transition to a new state –may send messages to other processes

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring A Natural Failure Detector Implementation in Eventual Synchrony Model Implement failure detector using timeouts: –When expecting a message from a process i, wait  clock skew before suspecting i In stable periods,  always hold, hence no false suspicions

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring The resulting failure detector is ◊P - Eventually Perfect Strong Completeness: From some point on, every faulty process is suspected by every correct process Eventual Strong Accuracy: From some point on, no correct process is suspected Is it implementable in asynchronous systems?

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring t 0 q does not suspect p 00 t 0 p crashes '0'0 t 1 q suspects p t 0 p’s msgs delayed 11 t 1 q suspects pt 2 q does not suspect p '1'1 Are we done? Now,  1 is fair Build a fair run without failures s.t. there is no time after which q does not suspect p t0t0 t 1 q suspects p t 2 p crahses t 3 q suspects p 22 t0t0 t 1 q suspects p t 2 p’s msgs delayed t 3 q suspects p Continue by induction to build an infinite run in which q is correct, suspected at t 1,t 3,t 5, …

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Unreliable Failure Detectors Failure detector’s output can be wrong (even arbitrary) for an unbounded (finite) prefix of a run Captures eventual synchrony An algorithm that tolerates unbounded periods of asynchrony is called indulgent [ Guerraoui 98 ]

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Observations on Indulgent Consensus Algorithms Every indulgent consensus algorithm also solves uniform consensus [ Guerraoui 98 ] It is impossible to solve t-resilient indulgent consensus when t ≥ n/2 [ Chandra, Toueg 96; Guerraoui 98 ] –henceforward, assume t < n/2

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Weaker Failure Detector ◊S – Eventually Strong –Strong Completeness –Eventual Weak Accuracy: From some point on, some correct process is not suspected by any correct process ◊P is a subset of ◊S –Every failure detector of class ◊P is also of class ◊S Strictly weaker than ◊P –homework question Equivalent to the weakest for consensus

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Model n process 1,…n t<n/2 of them can crash Reliable links between correct processes Asynchronous with ◊S

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring ◊S-based Consensus [ Mostefaoui, Raynal 99 ] val  input; est   || for r =1, 2, … do coord  (r-1 mod n)+1 if I am coord, then send (r,val) to all wait for ( (r, v) from coord OR suspect coord (by ◊S)) if receive v from coord then est  v else est   send (r, est) to all wait for (r,e) from n-t processes if any non-  value e received then val  e if all received e’s have same non-  value v then send (“decide”, v) to all return(v) || Upon receive (“decide”, v), forward to all; return(v) 1 2

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Failure-Free Suspicion-Free Run 11 2 n (1, v 1 ) 1 2 n all have est = v 1 all decide v 1 (decide, v 1 )

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Validity Proof For every i, val i and est i always store the initial value of some process or  By induction on the length of the execution Initially, for every process i, val i stores i’s initial value, est i is  Subsequently, they can only change to store a val j or est i value sent by some process j

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Lemmas Lemma 1: If in some round r, two messages (r,v) and (r,v’) are sent such that v ≠  and v’ ≠ , then v=v’. Lemma 2: If in some round r, n-t processes send (r,v), then for every round r’>r, if a message (r’,v’) with v’ ≠  is sent, then v=v’. –Hint: n-t > n/2.

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Agreement Proof Assume by contradiction that two different decisions, v ≠ v’ are made. Let r (r’) be the first round in which some process i (i’) decides v (v’) when it receives n-t (r,v) ((r’,v’)) messages. By Lemma 1, r ≠ r’, and by Lemma 2, neither r > r’ nor r’>r. A contradiction.

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Termination Proof Steps Progress: until some process decides, no process is ever “stuck” in a round forever First decision: some correct process eventually decides Subsequent decisions: if some correct process decides, then all correct processes eventually decide

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring How Long Does It Take? The algorithm can take unbounded time –what if no failures occur? Is this inevitable? Can we say more than “decision is reached eventually” ?

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Performance Metric Number of communication steps in well-behaved runs Well-behaved: –No failures –Stable (synchronous) from the beginning –No false suspicions Motivation: common case

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring The Algorithm’s Running Time in Well-Behaved Runs In round 1, the coordinator is correct, not suspected by any process All processes decide at the end of phase two of round 1 –decision in two communication steps –halting (stopping) takes three steps

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Alternative Weak Failure Detector  – Leader –Outputs one trusted process –From some point, all correct processes trust the same correct process Can easily implement ◊S Is the weakest for consensus [Chandra, Hadzilacos, Toueg 96]

 Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring A Natural  Implementation Use ◊P implementation Output lowest id non-suspected process