1. Rex: Replication at the Speed of Multi-core Zhenyu Guo, Chuntao Hong, Dong Zhou*, Mao Yang, Lidong Zhou, Li Zhuang Microsoft ResearchCMU* 1

2. Tension between Replication and Multi- core Most applications are multi-threaded But, to replicate, you can only use single-thread Sacrifices performance for replication 2 Database Lock Server File Server Key-value Stores Multi-core Replication

3. Rex: Replication at the Speed of Multi-core 3 Replication Multi-core

4. Outline Motivation System Overview Implementation Evaluation 4

5. State Machine Replication To replicate a service: 1.Model as deterministic state machine 2.Order requests with consensus protocol 3.Execute with single-thread 5 Consensus Server Sequential Execution Consistent States Parallel Execution Multi-core Server Inconsistent States requests

6. Why Multi-thread Breaks State Machine Replication Non-deterministic decisions: locking order, etc… Replicas make decisions independently 6 Server 1 Server 2 Performance Consistency

7. Rex: Execute-Agree-Follow 7 Primary Traces Consensus Traces Secondary Execute AgreeFollow

8. Programming With Rex 1.Model app as RexRSM 2.Use Rex to make non-deterministic decisions RexLocks, RexCond, … RexTimeStamp, RexRand, etc. 8

9. Outline Motivation System Overview Implementation Evaluation 9

10. Normal Execution: Primary 10 2 lockA 3 unlockA 4 1 request 1 1 request 2 2 lockA reply 1 3 4 unlockA reply 2 Primary Trace: (t1, 1, request 1) … Causal edge((t1, 3)->(t2, 2)) … (t1, 4, reply 1)... …

11. Normal Execution: Secondary 11 1 request 1 1 request 2 lockA 2 3 unlockA 4 reply 1 Secondary (t1, 1, request 1) … Causal edge((t1, 3)->(t2, 2)) … (t1, 4, reply 1)... … 3 4 unlockA reply 2 2 waited event

12. Primary Failover Primary restart from checkpoint rejoin Secondary upgrade to primary switch replay -> record 12 Committed Uncommitted Crash

13. Unique Challenges: Integrating Replication and Record/Replay Inconsistency cut “Holes” in logs Causal edge pruning Hybrid execution … 13

14. The Inconsistent Cut Problem Collects logs at each thread asynchronously Inconsistent cut contains destination nodes without source node Problem: not be able to follow 14

15. Solving Inconsistent Cut Problem Define consensus on last consistent cut Drop C1-C2 when primary fail Reply only when reply contained in a committed consistent cut Use vector clock to track 15

16. Outline Motivation System Overview Implementation Evaluation 16

17. Experiment Setup Real-world Applications Micro-benchmark: for lock contention ratio Servers: 12-core, 24-thread, 10GE network 17 AppDescription ThumbnailGenerating and storing thumbnails XLockLock server similar to Chubby File ServerFile server Kyoto CabinetKey-value store LevelDBLocal storage behind BigTable MemCachedCache server

18. Performance Overview 18 Rex scales as nonreplicated <24% overhead

19. LevelDB in Detail 19 # cores Waited events grows with # threads, so does overhead overhead drops with more threads to schedule

20. Lock Conflict Ratio 20 Overhead < 15%

21. Summary Rex: execute-agree-follow Applied to six real-world applications Preserves scalability and low overhead 21

22. Thanks! Q&A 22

23. Backups 23

24. Dealing with Data Races Reply logging & compare Resource version checking Lock-free data structures: NATIVE_EXEC Experience shows that getting rid of data races is doable 24

25. Workloads Thumbnail: 1 pic per request K-V stores: 1M pairs 16 byte key, 100 byte value 10% write File system: 16KB random requests 20% write Xlock: 90% lease renew 100B – 5KB file 25

26. Lock Granularity 26

27. Request Granularity 27 10% computation in locks 1% conflict ratio

28. Experimental Results: Scalability 28

29. Causal Events & Performance 29

30. Improving Performance: Causal Edge Pruning with Vector Clock More causal edges, more overhead Causal edge pruning: trades primary performance for secondary 30 Reduces 58% ~ 99% causal edges

31. Replicated State Machine 31

32. Rex: Causal Order Replication 32

33. Correctness Correctness guaranteed by: 1.Captures all non-determinism with Rex 2.Consensus on traces 3.Agreed trace is a continuous sequence (no holes) 33

34. Inconsistent Cut: Why Is It Bad? 34 Trace: t1 unlock -> t2 lock -> t2 unlock -> t3 lock reply: 0 Replay: t1 unlock -> t3 lock -> t3 unlock -> t2 lockreply: 1 Should we reply 0 or 1?

35. Inconsistent Cut: Solving the Reply Problem Reply only when reply and all its dependencies are committed Use a vector clock to detect 35