1. Prophecy: Using History for High- Throughput Fault Tolerance Siddhartha Sen Joint work with Wyatt Lloyd and Mike Freedman Princeton University

2. Non-crash failures happen Model as Byzantine (malicious)

3. Mask Byzantine faults Clients Service

4. Mask Byzantine faults Replicated service Clients Throughput

5. Mask Byzantine faults Replicated service Clients Throughput Linearizability (strong consistency)

6. Byzantine fault tolerance (BFT) Low throughput Modifies clients Long-lived sessions

7. Prophecy High throughput + good consistency No free lunch: – Read-mostly workloads – Slightly weakened consistency

8. Byzantine fault tolerance (BFT) Low throughput Modifies clients Long-lived sessions D-Prophecy Prophecy

9. Traditional BFT reads Clients Replica Group application Agree?

10. A cache solution Clients Replica Group application cache Agree?

11. A cache solution Clients Replica Group application cache Agree? Problems: Huge cache Invalidation

12. A compact cache Clients Replica Group application cache RequestsResponses req1resp1 req2resp2 req3resp3

13. A compact cache Clients Replica Group application cache RequestsResponses sketch(req1)sketch(resp1) sketch(req2)sketch(resp2) sketch(req3)sketch(resp3) RequestsResponses

14. A sketcher Clients Replica Group application sketcher

15. A sketcher Clients Replica Group …… …… …… sketchwebpage

16. Executing a read Clients Replica Group …… ………… …… Agree? Fast, load-balanced reads sketchwebpage

17. Executing a read Clients Replica Group …… …… …… …… Agree? sketchwebpage

18. Executing a read Clients Replica Group …… …… …… sketchwebpage key-value store replicated state machine

19. Executing a read Clients Replica Group …… …… …… Agree? sketchwebpage …… Maintain a fresh cache

20. NO! Did we achieve linearizability?

21. Executing a read Clients Replica Group …… …… …… sketchwebpage ……

22. Executing a read Clients Replica Group …… …… …… Agree? sketchwebpage ……

23. Executing a read Clients Replica Group …… …… …… Agree? sketchwebpage …… Fast reads may be stale

24. Load balancing Clients Replica Group …… …… …… Agree? sketchwebpage …… Pr(k stale) = g k

25. Traditional BFT: Each replica executes read Linearizability D-Prophecy: One replica executes read “Delay-once” linearizability D-Prophecy vs. BFT Clients Replica Group

26. Byzantine fault tolerance (BFT) Low throughput Modifies clients Long-lived sessions D-Prophecy Prophecy

27. Key-exchange overhead 11% 3%

28. Internet services Clients Replica Group

29. Sketcher A proxy solution Clients Replica Group Proxy Consolidate sketchers

30. Sketcher Clients Replica Group Trusted A proxy solution Sketcher must be fail-stop

31. Sketcher Clients Replica Group Trusted A proxy solution Trust middlebox already Small and simple

32. …… …… Sketcher …… Executing a read Clients Replica Group Trusted q ReqResp s(q)s(q)  …… …… …… Fast, load-balanced reads Prophecy

33. Sketcher Clients Replica Group Trusted ………… …… …… …… …… …… Fast reads may be stale …… ReqResp s(q)s(q) 

34. Delay-once linearizability

35.  W, R, W, R, W, R  Delay-once linearizability Read-after-write property

36.  W, R, W, W, R, R, W, R  Delay-once linearizability Read-after-write property

37. Example application Upload embarrassing photos 1. Remove colleagues from ACL 2. Upload photos 3. (Refresh) Weak may reorder Delay-once preserves order

38. Byzantine fault tolerance (BFT) Low throughput Modifies clients Long-lived sessions D-Prophecy Prophecy

39. Implementation Modified PBFT – PBFT is stable, complete – Competitive with Zyzzyva et. al. C++, Tamer async I/O – Sketcher:  2000 LOC – PBFT library:  1140 LOC – PBFT client:  1000 LOC

40. Evaluation Prophecy vs. proxied-PBFT – Proxied systems D-Prophecy vs. PBFT – Non-proxied systems

41. Evaluation Prophecy vs. proxied-PBFT – Proxied systems We will study: – Performance on “null” workloads – Performance with real replicated service – Where system bottlenecks, how to scale

42. Basic setup Sketcher Clients (100) Replica Group (PBFT) (concurrent)

43. Fraction of failed fast reads Alexa top sites: < 15%

44. Small benefit on null reads

45. Apache webserver setup Sketcher Clients Replica Group

46. Large benefit on real workload 3.7x 2.0x

47. Benefit grows with work 94  s (Apache) Null workloads are misleading!

48. Benefit grows with work

49. Single sketcher bottlenecks

50. Scaling out

51. Scales linearly with replicas

52. Summary Prophecy good for Internet services – Fast, load-balanced reads D-Prophecy good for traditional services Prophecy scales linearly while PBFT stays flat Limitations: – Read-mostly workloads (meas. study corroborates) – Delay-once linearizability (useful for many apps)

53. Thank You

54. Additional slides

55. Transitions Prophecy good for read-mostly workloads Are transitions rare in practice?

56. Measurement study Alexa top sites Access main page every 20 sec for 24 hrs

57. Mostly static content

58. 15%

59. Dynamic content Rabin fingerprinting on transitions 43% differ by single contiguous change Sampled 4000 of them, over half due to: – Load balancing directives – Random IDs in links, function parameters