Download presentation
Presentation is loading. Please wait.
1
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 1 Principles of Reliable Distributed Systems Lecture 9: SMR with Paxos and Authenticated Byzantine Paxos Spring 2008 Prof. Idit Keidar
2
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 2 Material The Part-Time Parliament Lamport, TOCS 1998 Practical Byzantine Fault-Tolerance Castro and Liskov, OSDI 1999 The ABCDs of Paxos Lampson 2001
3
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 3 SMR (Atomic Broadcast) by Running a Sequence of Consensus Instances
4
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 4 Reminder: State-Machine Replication (SMR) Data is replicated at n servers Operations are initiated by clients Operations need to be performed at all correct servers in the same order Servers need to agree upon the sequence of operations
5
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 5 Reminder: Paxos Algorithm for state machine replication with eventual synchrony (ES) –Uses failure detector (leader election) Overcomes transient crashes & recoveries and message loss Main component: (one-shot) consensus protocol (aka Synod) –Last week
6
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 6 The 2 Phases of Paxos (Synod) 11 2 n...... (“accept”, b, v) 1 2 n...... 11 2 n...... (“prepare”, b) (“ack”, b, n’, v’) (“accept”, b, v) Phase 1: Learn about smaller ballotsPhase 2: Majority accepts v
7
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 7 SMR: Client-Server Interaction Leader-based: each process (client/server) has an estimate of who is the current leader A client sends a request to its current leader –E.g., “store X 100” The leader runs the Paxos (Synod) consensus algorithm to agree on the place of the request in the sequence –Input value: request + proposed sequence number –E.g., “store X 100” is the 7 th operation in the sequence The leader sends the response to the client –After invoking the operation on its copy
8
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 8 Consensus per Request Number Many consensus instances are running at the same time, each for some request number, ReqNum Each node has unbounded arrays –AcceptNum[r], AcceptVal[r], r = 1,2, … –AcceptVal holds the client’s requested operation E.g., “store X 100” Invoke operations on the state machine –In order: AcceptVal[1], then AcceptVal[2], etc. –After the respective consensus algorithm decides –Then send outcome as response to client (leader only)
9
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 9 Failure-Free Message Flow S1 S2 Sn...... C S1 S2 Sn...... S1 S2 Sn...... (“accept”, b, r, v) (“prepare”, b) (“ack”, b, n’, v’) C Phase 1Phase 2 request response (“accept”, b, r, v) Client’s request ReqNum
10
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 10 Observation In Phase 1, no new consensus values sent: –Leader chooses largest unique ballot number –Gets a majority to join this ballot number –Learns the outcome of all smaller ballots from this majority In Phase 2, leader proposes either its initial value (request from client) or latest value it learned in Phase 1
11
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 11 Failure-Free Message Flow S1 S2 Sn...... C S1 S2 Sn...... S1 S2 Sn...... (“accept”, b, r, v) (“prepare”, b) (“ack”, b, n’, v’) C Phase 1Phase 2 request response (“accept”, b, r, v)
12
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 12 Message Flow: Take 2 S1 S2 Sn...... C S1 S2 Sn...... S1 S2 Sn...... C Phase 1 Phase 2 request response S1 (“accept”, b, r, v) (“prepare”, b) (“ack”, b, n’, v’) (“accept”, b, r, v)
13
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 13 Optimization Run Phase 1 only when the leader changes –Phase 1 is called “view change” or “recovery mode” –Phase 2 is the “normal mode” Each message includes BallotNum (from the last Phase 1) and ReqNum –E.g., ReqNum = 7 when we’re trying to agree what the 7 th operation to invoke on the state machine should be Respond only to messages with the “right” BallotNum
14
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 14 Paxos Atomic Broadcast: Normal Mode Upon receive (“request”, v) from client if (I am not the leader) then forward to leader else /* propose v as request number n */ ReqNum ReqNum +1; send (“accept”, BallotNum, ReqNum, v) to all Upon receive (“accept”, b, r, v) with b = BallotNum /* accept proposal for request number n */ AcceptNum[r] b; AcceptVal[r] v send (“accept”, b, r, v) to all (first time only)
15
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 15 Recovery Mode Run once per ballot, for all request numbers –No ReqNum in “prepare” and “ack” messages The new leader must learn the outcome of all the pending requests that have smaller BallotNums –The “ack” messages include AcceptNum[r] and AcceptVal[r] for all pending requests For each of the pending requests, the leader sends an “accept” message What if there are holes? –E.g., leader learns of request number 13 and not of 12 –Fill in the gaps with dummy “do nothing” requests
16
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 16 Practical Byzantine Fault-Tolerance Aka Byzantine Paxos
17
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 17 Reminder: Byzantine Faults Faulty process can behave arbitrarily, i.e., they don’t have to follow the protocol. E.g., –Can suffer benign failures – crash, timing –Can send bogus values in messages –Can send messages at the wrong time –Can send different messages to different processes, etc. Captures software bugs, hacker intrusions
18
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 18 Reminder: Authenticated (Byzantine) Model Authentication: The receiver of a message can ascertain its origin –An intruder cannot masquerade as someone else Integrity: The receiver of a message can verify that it has not been modified in transit; –An intruder cannot substitute a false message for a legitimate one Nonrepudiation: A sender cannot falsely deny later that he sent a message
19
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 19 Byzantine Fault-Tolerant Consensus: Overview of Results Synchronous t-resilient algorithm – –iff t < n with authentication and weak unanimity –iff t < n/2 with authentication and strong unanimity –iff t < n/3 without authentication Eventually synchronous (ES) t-resilient algorithm –iff t < n/3 with or without authentication Homework problem: show the lower bound
20
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 20 Overcoming Byzantine Failures With 3t+1 Processes Recall what we did for crash failures – –We gathered “votes” from a majority in every ballot –Since every two majorities intersect, for every two ballots, at least one process votes in both But now, a faulty process can lie about what it did in the other ballot –We want a correct process in the intersection –Since n-t ≥ 2t+1, two sets of size n-t intersect by at least one correct process –Gather n-t votes in a ballot, to ensure that for every two ballots, at least one correct process votes in both
21
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 21 Byzantine Paxos Setting State machine replication Structured like Paxos: –Updates are sent to the current leader –Leader uses a consensus algorithm to have all replicas agree on the order of updates –Our focus today is the Consensus algorithm Used to implement BFS – Byzantine Fault Tolerant NFS –Only 3% slower than un-replicated NFS
22
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 22 Model n processes: {1,…n} Up to t Byzantine failures, t < n/3 –For simplicity, assume n = 3t+1 Authentication (PKI) Reliable links, no recovery (for now)
23
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 23 Reminder: Classic Paxos Phase I Periodically, until decision is reached do: if leader (by ) then BallotNum BallotNum.num+1, myId send (“prepare”, BallotNum) to all Upon receive (“prepare”, b) from i if b BallotNum then BallotNum b send (“ack”, b, AcceptNum, AcceptVal) to i
24
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 24 Reminder: Classic Paxos Phase II Upon receive (“ack”, BallotNum, b, val) from n-t if all vals = then myVal = initial value else myVal = received val with highest b send (“accept”, BallotNum, myVal) to all /* proposal */ Upon receive (“accept”, b, v) with b BallotNum AcceptNum b; AcceptVal v /* accept proposal */ send (“accept”, b, v) to all (first time only)
25
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 25 How Can Byzantine Failures Cause Problems?
26
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 26 Safety Problems: Leader Can Lie Problem 1: Leader can choose a value different than the highest accepted by n-t processes –Solution: Can “prove” he’s not lying by sending the signed “ack” (Phase 1) messages to all processes Problem 2: If no previous ballot was accepted, leader can send different new values to different processes
27
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 27 Solution to the 2 nd Problem Before accepting a value proposed by the leader, verify that the value was proposed to “enough” processes Byzantine Paxos Phases: –Phase 1: Prepare –Phase 2: Propose – echo leader’s proposal –Phase 3: Accept – now only if n-t proposed Add new variable: PropNum, initially 0
28
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 28 Safety Problems: Others Can Lie Problem 3: Faulty users can send invalid “accept” messages –Solution: Wait for n-t=2t+1 “accept” messages Problem 4: Faulty users can send invalid values with higher AcceptNums in “ack” messages –Solution: Can “prove” value is valid by forwarding signed “propose” messages –Add new variable: Proof, initially empty set
29
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 29 Liveness Problems Problem 5: Faulty leader can deadlock algorithm –Solution: Propose a new leader when the current does not deliver –Use rotating coordinator until one is correct, leader will be (BallotNum mod n)+1 Problem 6: Faulty processes may keep selecting new leaders all the time (livelock) –Solution: Accept a new ballot only if t+1 processes propose a new leader
30
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 30 And Now For Our Feature Presentation The Byzantine Paxos Consensus Algorithm
31
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 31 Byzantine Paxos Variables Int BallotNum, initially 0 Int PropNum, initially 0 Int AcceptNum, initially 0 Value { } AcceptVal, initially Message Set Proof, initially empty Define: Leader = (BallotNum mod n)+1
32
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 32 Byzantine Paxos Phase I: Prepare Upon timeout on Leader BallotNum BallotNum +1 send (“prepare”, BallotNum) to all Upon receive (“prepare”, b) from t+1 if (b < BallotNum) then return if (b > BallotNum) then BallotNum b send (“prepare”, BallotNum) to all send (“ack”, b, AcceptNum, AcceptVal, Proof) to Leader
33
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 33 Byzantine Paxos Phase II: Propose Upon receive (“ack”, BallotNum, b, val, proof) from n-t S = {received (signed) “ack” messages} if (all vals that have valid proofs in S are then myVal init value else myVal val that has valid proof with highest b in S send (“propose”, BallotNum, myVal, S) to all Upon receive (“propose”, BallotNum, v, S) if (BallotNum PropNum) then return if (v is not a valid choice given S) then return PropNum BallotNum send (“propose”, BallotNum, v, S) to all
34
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 34 Byzantine Paxos Phase III: Accept Upon receive (“propose”, b, v, S) from n-t if (b < BallotNum) then return AcceptNum b; AcceptVal v Proof set of n-t signed “propose” messages send (“accept”, b, v) to all Upon receive (“accept”, b, v) from n-t decide v
35
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 35 In Failure-Free Runs accept 1 prepareackpropose 2 n...... 1 2 n...... 11 2 n...... 1 2 n...... 1 2 n...... All send prepare All echo propose
36
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 36 Some Optimizations Prepare and its “ack” can be merged into one message round Proofs don’t have to be sent with messages: processes can have the information to check the proofs locally because the original messages are multicast
37
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 37 Invariant If proposals (b,v) and (b, v’) are accepted by correct processes i and j, (possibly i = j ) then v’=v Proof: –An accepted proposal is proposed by n-t processes –Two sets of n-t = 2t+1 processes have at least one correct process in common –A correct process sends no more than one propose message with the same b
38
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 38 Lemma 1 If a proposal (b,v) is accepted by t+1 correct processes, then for every proposal (b’, v’) with b’>b that is proposed by a correct process, v’=v. Again, follows from Lemma 2…
39
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 39 Lemma 2 If a proposal (b,v) is proposed by a correct process, then there is a set S including at least t+1 correct processes such that either –(1) no correct p in S accepts a proposal ranked less than b; or –(2) v is the value of the highest-ranked proposal among proposals ranked less than b accepted by correct processes in S
40
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 40 Proving Lemma 1 from Lemma 2 Assume (b,v) is accepted by t+1 correct processes, and consider the lowest ranked proposal (b’, v’) with b’>b proposed by a correct process Since two sets of t+1 correct processes have at least one correct process in common, case (1) of Lemma 2 is impossible, and by case (2), v’=v Continue by induction on ballot number
41
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 41 Proving Agreement Let v be a decided value. The first process that decides v receives a n-t accept messages for v with some ballot b, i.e., (b,v) is accepted by at least t+1 correct processes No other value is accepted by a correct process with the same b. Why? Let (v 1,b 1 ) be the first proposal accepted by n-t By Lemma 1, v 1 is the only possible decision value
42
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 42 Liveness Is the current leader making progress? –If yes, some correct process decides. This process can periodically forward the “proof” for its decision to others so they will decide too. –If not, all timeout on the leader and start a new ballot. Once there is a correct leader. –The n-t correct processes will send all the needed messages. –The t faulty processes will not be able to force a new ballot.
43
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 43 Atomic Broadcast: Issues Leader can propose invalid client requests Leader can refrain from proposing client requests Leader can lie to client about response Leader can refrain from sending client responses Solution: clients cannot trust a single server
44
Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 44 Byzantine Message Flow accept S1 prepareackpropose S2 Sn...... S1 S2 Sn...... S1 S2 Sn...... S1 S2 Sn...... propose S1 S2 Sn...... S1 request S2 Sn...... C C response
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.