Building Dependable Distributed Systems, Copyright Wenbing Zhao Chapter 7 Wenbing Zhao Department of Electrical and Computer Engineering Cleveland State University wenbing@ieee.org Building Dependable Distributed Systems, Copyright Wenbing Zhao

Outline Zyzzyva Byzantine general problem By Leslie Lamport, Robert Shostak, & Marshall Pease Practical Byzantine fault tolerance By Miguel Castro and Barbara Liskov, OSDI’99 http://www.pmg.csail.mit.edu/papers/osdi99.pdf Zyzzyva 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 2

The Byzantine Generals Problem Abstract model of a computer system that may have faulty components Faulty components may send conflicting information to different parts of the system Scenario where Byzantine Generals must reach agreement in the presence of traitors Generals must reach consensus among themselves => they all agree on an action: retreat or attack 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 3

The Byzantine Generals Scenario Commanding General General Lieutenant General General Byzantine Army Division Byzantine Army Division Enemy City Ask: why we insist on each lieutenant general to share what it has received from the commanding general to others? Lieutenant General General Traitorous General General Lieutenant General Byzantine Army Division Byzantine Army Division 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 4

Byzantine Generals Problem A commanding general must send an order to his n-1 lieutenants such that IC1. All loyal lieutenants obey the same order IC2. If the commanding general is loyal, then every loyal lieutenant obeys the order he sends IC1 = Agreement clause IC2 = Validity clause IC1 and IC2 are called interactive consistency Conditions Without IC2, we have a trivial solution: all lieutenants decide on attack or retreat 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 5

Byzantine Agreement Protocol Assumption: Every message that is sent is delivered correctly Traitors cannot interfere with messages they do not sent The receiver of a message knows who sent it Traitors cannot spoof messages The absence of a message can be detected Traitors cannot prevent an agreement by not sending => Synchronous system + no spoofing 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Byzantine Agreement Protocol (f=1) Round 1: the commander sends a value to each of the lieutenants Round 2: each of the lieutenants sends the value it received to its peers At the end of round 2, each lieutenant check to see if there is a majority opinion (attack or retreat). We have a solution if there is Question is: how many generals needed to tolerate f number of traitors? Note: f>=2 cases are much more complicated and expensive (Oral message algorithms) Why synchronous? We assume each round every message reaches the target reliably and in time 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 7

Unsolvable Situations – N=3, f=1 Commander Attack Attack He said Retreat lieutenant lieutenant A lieutenant cannot be sure if the commander or the other lieutenant is lying. Commander Attack Retreat lieutenant He said Retreat lieutenant 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 8

Byzantine Agreement Protocol (f=1) Commander Attack Retreat lieutenant He said Retreat lieutenant He said Attack He said Attack He said Attack Attack He said Retreat He said Attack lieutenant He said Attack If there are f traitors, then there must be at least 3f + 1 total generals for IC1 and IC2 to hold 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Byzantine Agreement Protocol (f=1) Under our assumption, if message digital signature is used and assuming the signature cannot be forged, we need only N=2f+1 to tolerate f traitors The commander still can send different information to different lieutenant, but a lieutenant cannot lie about what the commander has told him In asynchronous systems, N=2f+1 is not sufficient We have to stop after collecting f+1 input because the f faulty traitors could simply refrain from sending Unfortunately there might be f inputs from traitors 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Introduction to BFT Paper The growing reliance of industry and government on online information services Malicious attacks become more serious and successful More software errors due to increased size and complexity of software This paper presents “practical” algorithm for state machine replication that works in asynchronous systems like the Internet 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Assumptions Asynchronous distributed system The network may fail to deliver, delay, duplicate or deliver them out of order Faulty nodes may behave arbitrarily Independent node failures The adversary cannot delay correct nodes indefinitely All messages are cryptographically signed by their sender and these signatures cannot be subverted by the adversary 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Service Properties A (deterministic) service is replicated among ≥ 3f+1 processors. Resilient to ≤ f failures Safety: All non-faulty replicas guaranteed to process the same requests in the same order Liveness: Clients eventually receive replies to their requests 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Optimal Resiliency Imagine non-faulty processors trying to agree upon a piece of data by telling each other what they believe the data to be A non-faulty processor must be sure about a piece of data before it can proceed f replicas may refuse to send messages, so each processor must be ready to proceed after having received (n-1)-f messages Total of n-1 other replicas 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Optimal Resiliency But what if f of the (n-1)-f messages come from faulty replicas? To avoid confusion, the majority of messages must come from non-faulty nodes, i.e, (n-f-1)/2 ≥ f => Need a total of ≥3f+1 replicas 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

BFT Algorithm in a Nutshell Backup f + 1 Match (OK) Client Primary Backup Backup 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Replicas and Views ……… Set of replicas (R): |R| ≥ 3f + 1 R0 R0 R0 R1 For view v: primary p is assigned such that p= v mod |R| 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Safeguards If the client does not receive replies soon enough, it broadcasts the request to all replicas If the request has already been processed, the replicas simply re-send the reply (replicas remember the last reply message they sent to each client) If the primary does not multicast the request to the group, it will eventually be suspected to be faulty by enough replicas to cause a view change 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Normal Case Operation {REQUEST, o, t, c} Client Primary o – Operation t – Timestamp c - Client Timestamps are totally ordered such that later requests have higher timestamps than earlier ones 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Normal Case Operation Primary p receives a client request m , it starts a three-phase protocol Three phases are: pre-prepare, prepare, commit 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Pre-Prepare Phase Backup <<PRE-PREPARE, v, n, d> , m> Primary <<PRE-PREPARE, v, n, d> , m> Backup Note that d is the digest of the message m (i.e., hash value of m). It is NOT the digital signature for the pre-prepare message!!!! The digital signature is not explicitly included in the message shown in the illustration here and later!!! v – view number n – sequence number d – digest of the message D(m) m – message Backup 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 21

Prepare Phase A backup accepts the PRE-PREPARE message only if: The signatures are valid and the digest matches m It is in view v It has not accepted a PRE-PREPARE for the same v and n Sequence number is within accepted bounds 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Prepare Phase <PREPARE, v, n, d, i> Signatures are correct If backup i accepts the pre-prepare message it enters prepare phase by multicasting <PREPARE, v, n, d, i> to all other replicas and adds both messages to its log Otherwise does nothing Replica (including primary) accepts prepare message and adds them to its log, provided that Signatures are correct View numbers match the current view Sequence number is within accepted bounds d is message digest! NOT digital signature! 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 23

Prepare Phase At replica i, prepared (m, v, n, i) = true, iff 2f PREPARE from different backups that match the pre-prepare When prepared = true, replica i multicasts <COMMIT, v, n, d , i> to other replicas 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Agreement Achieved If primary is non-faulty then all 2f+1 non-faulty replicas agree on the sequence number If primary is faulty Either ≥f+1 non-faulty replicas (majority) agree on some other sequence and the rest realize that the primary is faulty Or, all non-faulty replicas will suspect the primary is faulty When a faulty primary is replaced, the minority of confused non-faulty replicas are brought up to date up by the majority 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Commit Phase Replicas accept commit messages and insert them in their log provided signatures are same Define committed and committed-local predicates as Committed (m, v, n) = true, iff prepared (m, v, n, i) is true for all i in some set of f+1 non-faulty replicas Committed-local (m, v, n, i) = true iff the replica has accepted 2f+1 commit message from different replicas that match the pre-prepare for m If Committed-local (m,v,n,i) is true for some non-faulty replica i, then committed (m,v,n) is true 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Commit Phase Replica i executes the operation requested by m after committed-local (m, v, n, i) = true and i’s state reflects the sequential execution of all requests with lower sequence numbers The PRE-PREPARE and PREPARE phases of the protocol ensure agreement on the total order of requests within a view The PREPARE and COMMIT phases ensure total ordering across views 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Normal Operation Reply All replicas sends the reply <REPLY, v, t, c, i, r>, directly to the client v = current view number t = timestamp of the corresponding request i = replica number r = result of executing the requested operation c = client id Client waits for f+1 replies with valid signatures from different replicas, and with same t and r, before accepting the result r 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Normal Case Operation: Summery Request Pre-prepare Prepare Commit Reply C Primary: 0 1 2 Faulty: 3 X 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Garbage Collection Used to discard messages from the log For the safety condition to hold, messages must be kept in a replica’s log until it knows that the requests have been executed by at least f+1 non-faulty replicas Achieved using a checkpoint, which occur when a request with sequence number (n) is divisible by some constant is executed May not get to this 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 30

Garbage Collection When a replica i produces a checkpoint it multicasts a message <CHECKPOINT, n, d, i> to other replicas Each replica collects checkpoint messages in its log until it has 2f+1 of them for sequence number n with same digest d This creates a stable checkpoint and the replica discards all the pre-prepare, prepare and commit messages n is the sequence number of the last request whose execution is reflected in the state d is digest of the checkpoint 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 31

View Changes Triggered by timeouts that prevent backups from waiting indefinitely for request to execute If the timer of backup expires in view v, the backup starts a view change to move to view v+1 by, Not accepting messages (other than checkpoint, view-change, and new-view messages) Multicasting a VIEW-CHANGE message What prompts a view change? When does the tiimer start? Stop? 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 32

View Changes VIEW-CHANGE message is defined as <VIEW-CHANGE, v+1, n, C, P, i> where, C = 2f + 1 checkpoint messages P = set of sets Pm Pm = a PRE-PREPARE msg + all PREPARE messages for all messages with committed = false Here n is the sequence number of the last stable checkpoint s known to i, 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 33

View Change - Primary Primary p of view v+1 receives 2f valid VIEW-CHANGE messages Multicasts a <NEW-VIEW, v+ 1, V, O> message to all other replicas where V = set of 2f valid VIEW-CHANGE messages O = set of reissued PRE-PREPARE messages Moves to view v+1 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

View Changes - Backups Accepts NEW-VIEW by checking V and O Sends PREPARE messages for everything in O These PREPARE messages carry view v+1 Moves to view v+1 The primary creates a new pre-prepare message for view v+1 for each sequence number between min-s and max-s. There are two cases: (1) there is at least one set in the P component of some view-change message in V with sequence number n, or (2) there is no such set. In the first case, the primary creates a new message <PRE-PREPARE,v+1,n,d>, where d is the request digest in the pre-prepare message for sequence number with the highest view number in V. In the second case, it creates a new preprepare message <PRE-PREPARE,v+1,d,d^null>, where d^null is the digest of a special null request; a null request goes through the protocol like other requests, but its execution is a no-op. 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 35

Events Before the View Change Before the view change we have two groups of non-faulty replicas: the Confused minority and the Agreed majority A non-faulty replica becomes Confused when it is kept by the faulty's from agreeing on a sequence number for a request It can't process this request and so it will time out, causing the replica to vote for a new view 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 36

Events Before the View Change The minority Confused replicas send a VIEW-CHANGE message and drop off the network The majority Agreed replicas continue working as long as the faulty's help with agreement The two groups can go out of synch but the majority keeps working until the faulty's cease helping with agreement 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 37

System State: Faulty Primary Is Erroneous View Change Possible? System State Confused Minority ≤f non-faulty replicas Agreed Majority ≥f+1 non-faulty replicas Agreed Majority ≥f+1 non-faulty replicas Confused Minority ≤f non-faulty replicas Adversary f non-faulty replicas P Adversary f non-faulty replicas P f faulty replicas f faulty replicas ≤2f replicas: NOT enough to change views 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 38

Events Before the View Change Given ≥f+1 non-faulty replicas that are trying to agree, the faulty replicas can either help that or hinder that If they help, then agreement on request ordering is achieved and the clients get ≥f+1 matching replies for all requests with the faulty's help If they hinder, then the ≥f+1 non-faulty's will time out and demand for a new view When the new majority is in favor of a view change, we can proceed to the new view Can we have a confused majority while the remaining minority correct replicas keep working? No because they have no way to get 2f+1 matching messages to do anything 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 39

System State: Faulty Primary Is it possible to continue processing requests? System State Confused Minority ≤f non-faulty replicas Confused Minority ≤f non-faulty replicas Agreed Majority ≥f+1 non-faulty replicas Agreed Majority ≥f+1 non-faulty replicas Adversary P Adversary f non-faulty replicas P f faulty replicas f faulty replicas YES ≥2f+1 replicas: enough for agreement 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 40

System State: Faulty Primary Majority now large enough to independently move to a new view Confused Minority ≤f non-faulty replicas Agreed Majority ≥f+1 non-faulty replicas Adversary f non-faulty replicas YES ≥2f+1 replicas: enough for agreement Confused Majority 2f+1 non-faulty replicas Enough to agree to change views P Adversary f non-faulty replicas P f faulty replicas f faulty replicas Faulty replicas cease helping with agreement 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao 41

Liveness Replicas must move to a new view if they are unable to execute a request To avoid starting a view change too soon, a replica that multicasts a view-change message for view v+1, waits for 2f+1 view-change messages and then starts the timer T If the timer T expires before receiving new-view message it starts the view change for view v+2 The timer will wait 2T before starting a view-change from v+2 to v+3 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Liveness If a replica receives f+1 valid view-change messages from other replicas for views greater than its current view, it sends a view-change message for the smallest view in the set, even if T has not expired Faulty replicas cannot cause a view-change by sending a view-change message since a view-change will happen only if at least f+1 replicas send view-change message The above techniques guarantee liveness, unless message delays grow faster than the timeout period indefinitely 4/4/2019 EEC688/788: Secure & Dependable Computing Building Dependable Distributed Systems, Copyright Wenbing Zhao Wenbing Zhao

Zyzzyva: Speculative BFT A replica speculatively executes a request as soon as it receives a valid pre-prepare msg Commitment of a request is moved to the client A request is said to have completed at hthe issuing client if the corresponding reply can be safely delivered If a request completes at a client, the request will eventually be committed at the server replicas Prepare and commit phases are reduced to a single phase View change has one more additional phase Building Dependable Distributed Systems, Copyright Wenbing Zhao

Zyzzyva: Speculative BFT History hash: helps client determine if its request has been ordered appropriately Server replica maintains a history hash for each request ordered and appends the history hash hs = H(hs-1,ds) to the reply for the request with sequence number s ds: digest for the request Previous hash: hs-1, new hash with request s executed: hs H(): hash function Prefix concept: his is a prefix of hsj if sj > si, and there exist a set of requests with sequence numbers si+1, si+2, …, sj-1 with digests dsi+1, dsi+2,…,dsj-1 such that hsi+1=H(hsi, dsi+1), hsi+2 = H(hsi+1,dsi+2),…,hsj=H(hsj-1,dsj) Building Dependable Distributed Systems, Copyright Wenbing Zhao

Zyzzyva: Speculative BFT Safety: given any two requests that have completed, they must have been assigned two different sequence numbers. Furthermore, if the two sequence numbers are i and j an i < j, the history hash hi must be a prefix of hj Liveness: if a nonfaulty client issues a request, the request eventually completes Building Dependable Distributed Systems, Copyright Wenbing Zhao

Zyzzyva: Agreement Protocol A client maintains a complete timer for each request A request may complete at the client in one of two cases: Case 1: client receives 3f+1 matching replies => all replicas have executed the request in the same total order Case 2: client receives at least 2f+1 matching replies when timer expires. The client then initiates another round of msg exchange with the server replicas before the request is declared as complete Building Dependable Distributed Systems, Copyright Wenbing Zhao

Zyzzyva: Agreement Protocol, Case 1 Building Dependable Distributed Systems, Copyright Wenbing Zhao

Zyzzyva: Agreement Protocol, Case 2 Building Dependable Distributed Systems, Copyright Wenbing Zhao

Building Dependable Distributed Systems, Copyright Wenbing Zhao Zyzzyva: Case 2 Commit msg: contains a commit certificate: A list of 2f+1 replica ids Signed component of spec-response from each of the 2f+1 replicas Local-commit: a replica sends a local-commit when it receives a valid commit msg from client Verify history hash When the client receives 2f+1 or more local-commit, the request is completed and can be delivered Building Dependable Distributed Systems, Copyright Wenbing Zhao

Zyzzyva: View Change Protocol When a view change is triggered Sufficient number of backups timed out the current primary When the client receives spec-response msgs with different sequence numbers or different history hash values In the 2nd case, client broadcasts a POM msg to all replicas Replica initiates a view change A replica also rebroadcasts POM upon receiving one => to speed up view change Building Dependable Distributed Systems, Copyright Wenbing Zhao

Zyzzyva: View Change Protocol What is special in Zyzzyva At most one round msg exchange during normal operation for agreement => equivalent to prepare phase (or commit phase), for case 2. Replicas would have commit certificate For case 1, replicas would not possess a commit certificate Impact on view change: Need an additional round msg exchange – “I hate the primary” Need to change condition on including a request in new-view msg Cannot use PBFT view change protocol: may lose liveness Faulty primary could force f nonfaulty replicas to suspect itself, but cooperate with other f+1 nonfaulty replicas f faulty replicas stops sending spec-response Client cannot complete any request: only receive f+1 matching replies Building Dependable Distributed Systems, Copyright Wenbing Zhao

Zyzzyva: View Change Protocol PBFT view change A replica abandons the current view as soon as it suspects the primary: stops participating agreement for current view Zyzzyva view change Make sure a view change will actually take place before abandoning the current view Accomplished by using the “I hate primary” msg exchange: a replica abandons the current view only when it receives f+1 “I hate primary” msgs Building Dependable Distributed Systems, Copyright Wenbing Zhao

Zyzzyva: View Change Protocol Zyzzyva view change: deal with case 1 A replica includes all order-req msgs received since latest stable checkpoint, or the most recent commit certificate New primary adopts request-seq# binding if f+1 or more matching order-req msgs The primary may see multiple sets for same seq# (for different requests): can select any of the bindings because none of them could have been completed A backup should accept the decision by the new primary for the ordering Building Dependable Distributed Systems, Copyright Wenbing Zhao