1. 1 Attested Append-Only Memory: Making Adversaries Stick to their Word Byung-Gon Chun (ICSI) October 15, 2007 Joint work with Petros Maniatis (Intel Research, Bekeley), Scott Shenker, and John Kubiatowicz (UC Berkeley)

2. 2 Context

3. 3

4. 4 Centralized Server inc (1) 10 dec (1) 10 Alice Bob Counter 11 Alice: inc(1) Counter 11 Bob: dec(1) Counter 10 11

5. 5 Liveness: making progress Centralized Server inc (1) 10 dec (1) 9 Alice Bob Counter 11 Alice: inc(1) Counter 11 Bob: dec(1) Counter 10 Bob: dec(1) Counter 9 Alice: inc(1) Counter 10 Linearizability: 1. A serial schedule of operations 2. External order

6. 6 2 4 1 3 Replicated Servers inc (1) Byzantine Fault Tolerant Replicated Systems e.g., PBFT[TOCS02] S1: 11 S2: 11 S3: 11 11 Alice Bob 11 BFT algorithms can tolerate up to 1/3 faulty replicas. e.g., f out of 3f+1 If the fault assumption is violated, there is no guarantee! 10 Linearizability and liveness 10 11

7. 7 2 4 1 3 Servers Equivocating to Servers inc (1) 11 9 Alice Bob dec (9) 10 Sequence 101 Alice:inc(1) Sequence 101 Bob:dec(1)

8. 8 24 1 3 Servers Equivocating to Clients 11 9 Alice Bob S1: 11 S2: 11 S3: 11 S1: 9 S2: 9 S4: 9 Sequence 101 Alice:inc(1) Sequence 101 Bob:dec(1)

9. 9 Questions Does preventing equivocation help at all? –Can we improve upon the 1/3 Byzantine fault bound? How do we prevent equivocation? –Is there any minimal system support?

10. 10 Talk Outline Introduction and Motivation Attested Append-Only Memory (A2M) A2M Protocols Evaluation Conclusion

11. 11 High-level View 24 13 Application + Protocol Service 24 13 Application + Protocol Service Equivocation guard Non-equivocation

12. 12 Attested Append-Only Memory (A2M) A set of numbered logs Each log entry contains –Sequence number –Stored value –Crypto digest of entire log lookup / end –Get a log entry –Attest (sequence number, value, history digest) –Attest freshness –Attest the end of log append / advance –Cannot overwrite

13. 13 An A2M Usage Pattern A2M Sending replica Result: the sending replica is forced to say the same msg for n Replicas need to agree on msg in sequence number n append(h(msg)) lookup(n)  msg, msg, msg,

14. 14 A2M Implementation Scenarios Third-party service Remote Software isolation Local Virtual machine Local Virtual machine monitor Local Trusted hardware Local Faulty app Faulty operator Faulty app Faulty operator

15. 15 Talk Outline Introduction and Motivation Attested Append-Only Memory (A2M) A2M Protocols Evaluation Conclusion

16. 16 A2M protocols A2M State Machine Replication –A2M-PBFT-E –A2M-PBFT-EA A2M-Storage (SUNDR-like) A2M-Q/U

17. 17 Background: PBFT Assumptions –Byzantine faults –Secure cryptography –Weak synchrony Guarantee linearizability and liveness with up to f faults out of 3f+1 replicas Three phase protocol View change

18. 18 Background: PBFT time Primary Client1 PrepreparePrepareCommit Request Reply Execute     s1 s2 s3 s4 Quorum = 3 [ 1,a ] Client2 [ 1,b ] Quorum: matching messages from different replicas req,resp Agreement Execution

19. 19 A2M-PBFT-E (Execution) time Primary Client1 PrepreparePrepareCommit Request Reply Execute     s1 s2 s3 s4 Quorum = 3 Attested by A2M req,resp, Request log A2M

20. 20 Intuition Client1 S1: req1,resp1, S2: req1,resp1, S3: req1,resp1, Client2 S1: req2,resp2, S2: req2,resp2, S4: req2,resp2,

21. 21 Liveness Problems of A2M-PBFT-E time Primary Client1 Preprepare Prepare Commit Request Reply Execute     s1 s2 s3 s4 Quorum = 3 Attested by A2M req,resp,

22. 22 A2M-PBFT-EA( Execution+Agreement ) time Primary Client1 PrepreparePrepareCommit Request Reply Execute     s1 s2 s3 s4 Quorum = 3 Attested by A2M req,resp,

23. 23 A2M-PBFT-EA (2f + 1 replicas) time Primary Client1 PrepreparePrepareCommit Request Reply Execute    s1 s2 s3 Quorum = 2 Attested by A2M req,resp,

24. 24 Intuition Quorum1 (2f + 1) Quorum2 (2f + 1) f + 1 1 non-faulty replica PBFT (3f + 1) Quorum1 (f + 1) Quorum2 (f + 1) 1 1 A2M A2M-PBFT-EA (2f + 1)

25. 25 A2M-PBFT-EA (Three phase) time Primary Client1 PrepreparePrepareCommit Request Reply Execute    s1 s2 s3 Quorum = 2 Attested by A2M req,resp,

26. 26 A2M-PBFT-EA (Two phase) time Primary Client1 PrepareCommit Request Reply Execute    s1 s2 s3 Attested by A2M req,resp,

27. 27 Other Results A2M-PBFT-EA View change A2M-Storage: achieve linearizability in an untrusted single-server system A2M-Q/U: require 4f+1 replicas (instead of 5f+1 replicas) to tolerate f faults

28. 28 Talk Outline Introduction and Motivation Attested Append-Only Memory (A2M) A2M Protocols Evaluation Conclusion

29. 29 Protocol Trade-offs 3f+1 2/31/3 A2M-PBFT-E 1/2 A2M-PBFT-EA PBFT 1/3

30. 30 Evaluation Setup Implemented A2M-PBFT-E and A2M-PBFT-EA A2M protocols use signatures or MACs for authentication Four replicas in a LAN. Each replica has its own A2M. Microbenchmarks –Null operation with various request or response sizes Macrobenchmarks: NFS –Software package compilation

31. 31 Macro-benchmarks: NFS NFS Step -S-PBFT-A2M- PBFT-E (sig) -A2M- PBFT-E (MAC) -A2M- PBFT- EA (3 phase) (sig) -A2M- PBFT- EA (3 phase) (MAC) Copy Uncompress Untar Configure Make Clean 0.219 1.015 2.322 12.748 7.241 0.180 0.709 3.027 4.448 12.412 7.461 0.298 1.026 4.378 6.826 19.173 9.778 0.640 0.728 3.103 4.553 12.659 7.500 0.312 2.141 8.601 12.896 26.181 11.379 0.742 0.763 3.236 4.669 13.040 7.510 0.311 Total (seconds) 23.72528.355 (0%) 41.821 (47.5%) 28.854 (1.8%) 61.940 (118.4%) 29.528 (4.1%)

32. 32 Trustworthy system Untrusted Trustworthy system + Small trusted primitives Untrusted Broader Implications What small trusted primitives to put to make systems better –e.g., trusted logical clocks for weak consistency guarantees –e.g., network interface card attestation More classes of components with different fault characteristics –trusted, semi-trusted, untrusted

33. 33 Conclusions Present A2M, a small trusted, log-based primitive –Simple and easily implementable –Prevent equivocation Improve fault tolerance by forcing servers to commit to a single history of operations –Improve fault bounds of BFT state machine replication –Achieve linearizability in an untrusted single-server system –The benefits are achieved with small performance overhead A2M has broader implications on structuring trustworthy systems

34. 34 Thank you! Questions? SOSP 2007

35. 35 Related Work Weaken the guarantee –fork* consistency [NSDI07] –fork consistency [OSDI04] Standard trusted hardware like TPM –does not improve the fault bound Auditing –PeerReview [SOSP07], CATS [FAST07] Shared file servers –SUNDR[OSDI04], Ivy [OSDI02], Plutus[FAST03] Separating agreement from execution Symmetric faults – hybrid fault model Group communication