1. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 1 Principles of Reliable Distributed Systems Recitation 9: Paxos (  ) Made Simple Spring 2009 Alex Shraer

2. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 2 Abstract “The Paxos algorithm, when presented in plain English, is very simple.”

3. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 3 The Model Asynchronous Benign failures (non-Byzantine) Processes may fail by stopping, and restart. –No limitation on number of failures. Messages can be duplicated and lost, but not corrupted.

4. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 4 Agents Three classes: –proposers proposing the values –acceptors accepting the values –learners learn the outcome from others, after a value was chosen A single process may act as more than one agent.

5. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 5 Our Goal – Understand Paxos Safety requirements: –A single value is chosen, –Only a value that has been proposed may be chosen, –A process never learns that a value has been chosen unless it actually has been. Liveness requirements (conditional): –Some proposed value is eventually chosen, and –If a value has been chosen, then a process can eventually learn the value.

6. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 6 Choosing a value Take I: –Have a single acceptor agent. –All proposers send proposals to the acceptor. –Acceptor chooses first proposed value it receives. Where’s the problem? –Acceptor may fail (single point of failure). No further progress.

7. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 7 Choosing a value (cont’d) Take II: –Multiple acceptors –A proposer sends a proposal to a set of acceptors –An acceptor may accept the proposed value. –The value is chosen when a large enough set of acceptors have accepted it. How large is large enough? –Majority: any 2 majorities have a least one acceptor in common The same value is chosen by all acceptors Works only if an acceptor can accept only one value –Otherwise, an acceptor cannot be sure that every other acceptor will choose the same value

8. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 8 When to choose? If there are no failures or message loss, we want a value to be chosen (liveness) even if there is only one proposer: P1: An acceptor must accept the first proposal that it receives. What happens if several values are proposed by different proposers at about the same time?

9. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 9 Two proposals v1 v2 v1 v2 What value was accepted by the middle process? No way to learn which value was chosen –Unless some acceptor is allowed to change its mind!

10. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 10 Numbered Proposals An acceptor must be allowed to accept >1 proposal. Keeping track of proposals: unique numbers –In fact, pairs: A value is chosen when: –A single proposal with that value has been accepted by a majority In this case, we also say that the proposal was chosen

11. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 11 Maintaining Agreement We can allow multiple proposals to be chosen, but Must guarantee that all chosen proposals have the same value (agreement), so P2: If a proposal with value v is chosen, then every higher-numbered proposal that is chosen has value v.

12. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 12 Maintaining Agreement (cont’d) To be chosen, a proposal must be accepted by at least one acceptor, so we can satisfy P2 by satisfying a stronger requirement: P2a: If a proposal with value v is chosen, then every higher-numbered proposal accepted by any acceptor has value v.

13. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 13 Worst Case Scenario 1, v1 2,v2 v 1 was chosen 2, v2 propose Contradicts P2a! Does not hear any message (yet) must accept according to P1

14. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 14 Maintaining Agreement (cont’d) To guarantee P2a, we need to strengthen it: P2b: If a proposal with value v is chosen, then every higher-numbered proposal issued by any proposer has value v –Note: P2b  P2a  P2 How can P2b be enforced? –before a value is proposed, the process must check whether a value has already been chosen –how can this be done?

15. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 15 Issuing a proposal P2c: For any v and n, if a proposal with value v and number n is issued, then there is a set S consisting of a majority of acceptors such that either a)no acceptor in S accepts any proposal numbered < n or b)v is the value of the highest-numbered proposal among all proposals numbered < n accepted by the acceptors in S Lets show that if we preserve invariant P2c, then P2b is also satisfied either before or after the issue event either before or after the issue event

16. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 16 Proof that P2c  P2b By induction on n –Suppose v was chosen and its number is m  n this means there is a set C with a majority of acceptors that accepted v and m –And suppose that a proposal is issued according to P2c with number n –Need to show the proposed value is v –Base (n=m): trivial, since only v is proposed with m –Induction assumption: all proposals numbered m..n-1 have value v this means that v is the only possible value accepted with m..n-1  v is the highest numbered value accepted by every process  C. v's number at these processes is at least m –According to P2c, there is a majority set S such that either (a) or (b) *S intersects C, therefore at least one process in S accepts (in past or future) the value v with number  [m.. n-1] Since no other value has a higher timestamp <n (by induction assumption) v must be the value with highest number < n (that has been or will be) accepted by an acceptor in S  the proposed value is v

17. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 17 Maintaining P2c (Prepare) A proposer needs to contact majority S of acceptors –and then chose the higher numbered value But what if after they answer they accept a different value with a higher number? –Need to prevent this –Lets make them promise that they will not do that Choose proposal number n Request from all acceptors 1.A promise to not accept any proposal numbered < n 2.The proposal with the highest number < n that has been accepted (if any) Await replies from majority of acceptors

18. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 18 Issuing Proposals Issue a proposal (v,n) such that: –Either v is the value of the highest-numbered proposal, –Or v is any value (e.g., your input) if no proposals were reported Is it enough to get the response that no proposal < n was accepted from (only) a majority of acceptors? –yes, because even if others have accepted such proposals, their values can never be chosen

19. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 19 Acceptor’s Algorithm Can ignore any request without compromising safety. Should be allowed to respond (for progress) –Can always respond to prepare. –Can respond to accept only if it not promised not to. P1a: An acceptor can accept a proposal numbered n iff it has not responded to a prepare request having a number greater than n. –Notice that P1a  P1

20. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 20 Optimizations If an acceptor receives a prepare request numbered n, but already responded to a prepare request numbered > n… –No need to reply! (The value will not be accepted anyway) A proposer can abandon a request if someone else is issue a higher-numbered one Instead of ignoring accept/prepare messages, an acceptor can reply with a negative response and send the higher number it has heard of

21. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 21 Acceptor State An acceptor must remember: –The value and number of the highest-numbered proposal that it has ever accepted AcceptVal and AcceptNum –The number of the highest-numbered prepare request to which it has responded (promising not to accept anything smaller) BallotNum Need stable storage –Why? Homework question

22. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 22 Putting it all together…Phase 1 Proposer: selects a proposal number n and sends a prepare request with n to a majority of acceptors. Acceptor: if a prepare request is received with number n > that of any prepare request to which it has already responded, it responds to the request with a promise not to accept any more proposals numbered less than n + with the highest-numbered proposal (if any) that it has accepted.

23. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 23 Phase 2 Proposer: if receives a response to prepare(n) from a majority of acceptors  sends an accept(n,v) request to each of those acceptors –v is the value of the highest-numbered proposal among the responses, or input value if the responses reported no proposals. Acceptor: if receives an accept request for a proposal numbered n, it accepts the proposal unless it has already responded to a prepare request having a number greater than n.

24. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 24 Progress........................ Prepare(1) Ack(1,  ) Prepare(3) Ack(3,  )...... Prepare(4)

25. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 25 Progress (cont’d) A a distinguished proposer is required to guarantee progress –A leader –The only one allowed to issue proposals in stable state

26. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 26 Learning a Chosen Value Multiple options (communication complexity vs fault-tolerance): –Accepted value sent to all learners number of messages = #acceptors * #learners –Acceptors send to a distinguished learner, which then sends to all other learners number of messages = #acceptors + #learners not very fault tolerant –Possible compromise: acceptors send to a subset of learners

27. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 27 Putting it all Together 11 2 n...... (“accept”,  1,1 ,v 1 ) 1 2 n...... 11 2 n...... (“prepare”,1) (“ack”,  1,1 ,r 0,  ) decide v 1 (“accept”,  1,1 ,v 1 ) 1.Failure-free run 2.Our  implementation always trusts process 1

28. Eddie Bortnikov & Aran Bergman, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 28 Leslie Lamport “At the PODC 2001 conference, I got tired of everyone saying how difficult it was to understand the Paxos algorithm … The current version is 13 pages long, and contains no formula more complicated than n1 > n2 “