1. Systems Research Barbara Liskov October 2007

2. Replication Goal: provide reliability and availability by storing information at several nodes

3. Single Server Server Clients

4. Single Server Server Clients X

5. Replicated Servers Servers Clients X

6. Replication Issues Semantics What is being replicated Failure assumptions

7. Issue 1: Semantics One-copy consistency Or weaker Servers Clients

8. Issue 2: Type of Operations Only reads and writes General operations acct.deposit($$); acct.withdraw($$$);

9. Replication protocols Data replication Quorums and voting Operations State machine replication System performs a sequence of operations

10. Issue 3: Failure Assumptions Network is asynchronous Eventual delivery Network is malicious Corruption Replay Spoofing Handled via cryptography Nodes are failstop or Byzantine

11. Failstop Failures Nodes fail by crashing A machine is either working correctly or it is doing nothing! The assumption made in the 1980s

12. Failstop failures Requires 2f+1 replicas Operations must intersect at at least one replica In general want availability for both reads and writes: f+1 nodes is sufficient Read and write quorums

13. Quorums Servers Clients State: … … … write A X

14. Quorums Servers Clients State: … … … A A X

15. Quorums Servers Clients State: … … A write B X State: … A X

16. Data Replication R.H. Thomas, A majority consensus approach to concurrency control for multiple copy databases, ACM TODS, 1979 D.K. Gifford, Weighted voting for replicated data, SOSP 1979 H. Attiya, A. Bar-Noy, and D. Dolev, Sharing memory robustly in message-passing systems, JACM, Jan. 1995

17. Quorum Consensus Each data item has a version number A sequence of values write(d, val, v#) Waits for f+1 oks read(d) returns (val, v#) Waits for f+1 matching v#’s Else does a write-back

18. State Machine Replication Replicas must execute operations in the same order Implies replicas will have the same state, assuming replicas start in the same state operations are deterministic

19. Failstop Replication Viewstamped replication: a new primary copy method to support highly available distributed systems, B. Oki and B. Liskov, PODC 1988 Thesis, May 1988 Replication in the Harp file system, S. Ghemawat et. al, SOSP 1991 The part-time parliament, L. Lamport, TOCS 1998 Paxos made simple, L. Lamport, Nov. 2001

20. Approach Use a primary It orders the operations Other replicas obey this order

21. Views System moves through a sequence of views Primary runs the protocol Replicas watch the primary and do a view change if it fails

22. Normal Case Client sends request to primary Primary sends prepare message

23. Normal Case Client sends request to primary Primary sends prepare message Replicas receive prepare Send prepare-ok message to the primary

24. Normal Case Client sends request to primary Primary sends prepare message to all Replicas receive prepare Send prepare-ok message to the primary Primary waits for f prepare-oks Sends response to client

25. Normal Case A 2-phase protocol: Prepare; commit Only 3 message delays

26. Byzantine Failures Nodes fail arbitrarily They lie, they collude Causes Malicious attacks Non-deterministic software errors

27. Quorums 3f+1 replicas are needed to survive f failures 2f+1 replicas is a quorum Insures intersection The minimum in an asynchronous network

28. State: … A … A … A … Quorums Servers Clients write A X

29. State: … A … A B … B … B Quorums Servers Clients write B X

30. BFT M. Castro and B. Liskov, Practical Byzantine faulty tolerance and proactive recovery, ACM TOCS, 2002

31. Strategy Primary runs the protocol in the normal case Replicas watch the primary and do a view change if it fails Key difference: replicas might lie Solution: add a pre-prepare phase

32. Normal Case Client sends request to primary

33. Normal Case Client sends request to primary Primary sends pre-prepare message to all

34. Normal Case Client sends request to primary Primary sends pre-prepare message to all Why not a prepare message? Because primary might be malicious

35. Normal Case Client sends request to primary Primary sends pre-prepare message to all Replicas check the pre-prepare and if it is ok: Send prepare messages to all

36. Normal Case Replicas wait for 2f+1 matching prepares Send commit message to all

37. Normal Case Replicas wait for 2f+1 matching prepares Send commit message to all Replicas wait for 2f+1 matching commits Execute operation and send result to client

38. Follow-on Work BASE: using abstraction to improve fault tolerance, R. Rodrigo et al, SOSP 2001 R.Kotla and M. Dahlin, High Throughput Byzantine Fault tolerance. DSN 2004 J. Li and D. Mazieres, Beyond one-third faulty replicas in Byzantine fault tolerant systems, NSDI 07 Abd-El-Malek et al, Fault-scalable Byzantine fault- tolerant services, SOSP 05 HQ replications: a hybrid quorum protocol for Byzantine Fault tolerance, OSDI 06

39. Papers in SOSP 07 Monday 1:30-3:30 Zyzzyva: Speculative Byzantine fault tolerance Tolerating Byzantine faults in database systems using commit barrier scheduling Low-overhead Byzantine fault-tolerant storage Attested append-only memory: making adversaries stick to their word Tuesday: 11:00-12:00 PeerReview: practical accountability for distributed systems