1. Byzantine fault tolerance
Jinyang Li
With PBFT slides from Liskov

2. What we’ve learnt so far: tolerate fail-stop failures
Traditional RSM tolerates benign failures
Node crashes
Network partitions
A RSM w/ 2f+1 replicas can tolerate f simultaneous crashes

3. A traditional RSM client N0: primary N1: backup N2: backup
Req-x
Reply-x
Req-y
vid=0, seqno=2,req-y
vid=0, seqno=2,reply
Reply-y
N0: primary
vid=0, seqno=1,reply
vid=0, seqno=1,req-x
N1: backup
N2: backup
What must N2 check before executing
<vid=0, seqno=1, req>?

4. A reconfigurable RSM (lab 6)
client
Req-x
Req-x
N0: primary
vid=1, seqno=2,req-x
N1: backup
vid=0, seqno=1,req-x
Prepare vid=1, myh=N0:1
Accept vid=1, N0:1, {N0,N1}, lastseq=1
decide vid=1, N0:1, {N0,N1}, lastseq=1
N2: backup

5. Byzantine fault tolerance
Nodes fail arbitrarily
they lie
they collude
Causes
Malicious attacks
Software errors
Seminal work is PBFT
Practical Byzantine Fault Tolerance, M. Castro and B. Liskov, SOSP 1999

6. What does PBFT achieve?
Achieve sequential consistency (linearizability) if …
Tolerate f faults in a 3f+1-replica RSM
What does that mean in a practical sense?

7. Practical attacks that PBFT prevents (or not)
Prevent consistency attacks
E.g. a bad node fools clients into accepting at a stale bank balance
Protection is achieved only when <= f nodes fail
With enough bad nodes, attacker can fool clients into accept arbitrary bank balance, not just a stale one.
If an attacker hacks into one server, how likely is he to hack into the rest?
Does not prevent attacks like:
Turn a machine into a botnet node
Steal SSNs from servers

8. Why doesn’t traditional RSM work with Byzantine nodes?
Cannot rely on the primary to assign seqno
Malicious primary can assign the same seqno to different requests!
Cannot use Paxos for view change
Paxos uses a majority accept-quorum to tolerate f benign faults out of 2f+1 nodes
Does the intersection of two quorums always contain one honest node?
Bad node tells different things to different quorums!
E.g. tell N1 accept=val1 and tell N2 accept=val2

9. Paxos under Byzantine faults
Prepare vid=1, myn=N0:1
OK val=null N2 N0 N1
nh=N0:1
nh=N0:1
Prepare vid=1, myn=N0:1
OK val=null

10. Paxos under Byzantine faults
accept vid=1, myn=N0:1, val=xyz OK N2 X N0 N1
N0 decides on
Vid1=xyz
nh=N0:1
nh=N0:1

11. Paxos under Byzantine faults
prepare vid=1, myn=N1:1, val=abc
OK val=null N2 X N0 N1
N0 decides on
Vid1=xyz
nh=N0:1
nh=N0:1

12. Paxos under Byzantine faults
accept vid=1, myn=N1:1, val=abc OK N2 X N0 N1
N0 decides on
Vid1=xyz
nh=N0:1
nh=N1:1
N1 decides on
Vid1=abc
Agreement conflict!

13. PBFT main ideas Static configuration (same 3f+1 nodes)
To deal with malicious primary
Use a 3-phase protocol to agree on sequence number
To deal with loss of agreement
Use a bigger quorum (2f+1 out of 3f+1 nodes)
Need to authenticate communications

14. BFT requires a 2f+1 quorum out of 3f+1 nodes… … … …
1. State:
2. State:
3. State:
4. State: A A A
Servers X
write A
write A
write A
write A
Clients
For liveness, the quorum size must be at most N - f

15. BFT Quorums
1. State: …
2. State: …
3. State: …
4. State: … A A B B B
Servers X
write B
write B
write B
write B
Clients
For correctness, any two quorums must intersect at least
one honest node: (N-f) + (N-f) - N >= f N >= 3f+1

16. PBFT Strategy Primary runs the protocol in the normal case
Replicas watch the primary and do a view change if it fails

17. Replica state A replica id i (between 0 and N-1)
Replica 0, replica 1, …
A view number v#, initially 0
Primary is the replica with id
i = v# mod N
A log of <op, seq#, status> entries
Status = pre-prepared or prepared or committed 17

18. Normal Case
Client sends request to primary
or to all

19. Normal Case Primary sends pre-prepare message to all
Pre-prepare contains <v#,seq#,op>
Records operation in log as pre-prepared
Keep in mind that primary might be malicious
Send different seq# for the same op to different replicas
Use a duplicate seq# for op

20. Normal Case Replicas check the pre-prepare and if it is ok:
Record operation in log as pre-prepared
Send prepare messages to all
Prepare contains <i,v#,seq#,op>
All to all communication

21. Normal Case Replicas wait for 2f+1 matching prepares
Record operation in log as prepared
Send commit message to all
Commit contains <i,v#,seq#,op>
Trust the group, not the individuals

22. Normal Case Replicas wait for 2f+1 matching commits
Record operation in log as committed
Execute the operation
Send result to the client

23. Normal Case
Client waits for f+1 matching replies

24. BFT Client Primary Replica 2 Replica 3 Replica 4 Request Pre-Prepare
Commit
Reply

25. View Change Replicas watch the primary Request a view change
Commit point: when 2f+1 replicas have prepared

26. View Change Replicas watch the primary Request a view change
send a do-viewchange request to all
new primary requires 2f+1 requests
sends new-view with this certificate
Rest is similar

27. Additional Issues State transfer
Checkpoints (garbage collection of the log)
Selection of the primary
Timing of view changes

28. Possible improvements
Lower latency for writes (4 messages)
Replicas respond at prepare
Client waits for 2f+1 matching responses
Fast reads (one round trip)
Client sends to all; they respond immediately

29. BFT Performance Phase BFS-PK BFS NFS-sdt 1 25.4 0.7 0.6 2 1528.6 39.8
26.9 3
80.1
34.1
30.7 4
87.5
41.3
36.7 5
2935.1
265.4
237.1
total
4656.7
381.3
332.0
Table 2: Andrew 100: elapsed time in seconds
M. Castro and B. Liskov, Proactive Recovery in a Byzantine-Fault-Tolerant System, OSDI 2000

30. PBFT inspires much follow-on work
BASE: Using abstraction to improve fault tolerance, R. Rodrigo et al, SOSP 2001
R.Kotla and M. Dahlin, High Throughput Byzantine Fault tolerance. DSN 2004
J. Li and D. Mazieres, Beyond one-third faulty replicas in Byzantine fault tolerant systems, NSDI 07
Abd-El-Malek et al, Fault-scalable Byzantine fault-tolerant services, SOSP 05
J. Cowling et al, HQ replication: a hybrid quorum protocol for Byzantine Fault tolerance, OSDI 06
Zyzzyva: Speculative Byzantine fault tolerance SOSP 07
Tolerating Byzantine faults in database systems using commit barrier scheduling SOSP 07
Low-overhead Byzantine fault-tolerant storage SOSP 07
Attested append-only memory: making adversaries stick to their word SOSP 07

31. A2M’s main insights
Main worry in PBFT is that malicious nodes lie differently to different replicas
Decide on
Seq1=req-x
Decide on
Seq1=req-y

32. A2M’s proposal
Introduce a trusted abstraction: attested append-only-memory
A2M properties:
Trusted (Attacker can corrupt the RSM implementation, but not A2M itself)
Prevent malicious nodes from making different lies to different replicas

33. A2M’s abstraction A2M implements a trusted log
Append: append a value to log
Lookup: lookup the value at position i
End: lookup the value at the end of log
Truncate: garbage collection old values
Advance: skip positions in log

34. What A2M achieves Smaller quorum size for BFT
Achieve correctness & liveness if <= f Byzantine faults with 2f+1 nodes Or
Achieve correctness if <= 2f nodes fail. Achieve liveness if <=f nodes fail