Samsara: Efficient Deterministic Replay with Hardware Virtualization Extensions Peking University Shiru Ren, Chunqi Li, Le Tan, and Zhen Xiao July 27 ， 2015

Introduction 1 Background & Motivation 2 Samsara Overview 3 R & R the Memory Interleaving with HAV 4 2 Conclusion 6 Evaluation 5 Table of Contents

Introduction 3 Deterministic Replay  Gives computer users the ability to travel backward in time, recreating past states and events in the computer.  Checkpoint + record all non-deterministic events Checkpoint Execute same instruction stream Inject NDEs in logged points Initial State Instruction stream Final State Non-determinism Events (NDEs) (e.g. user / network input, interrupts… ) Final State’ NDEs log Recording Phase Replay Phase Application Scenarios  For debugging: cyclic debugging  For security: forensics, intrusion detection, malware analysis  For fault tolerance: hot standby, data recovery

Introduction 4 Deterministic Replay for Multi-processor  Deterministic replay for single processor is relatively mature and well- developed  Challenge on the multi-processor system: memory interleaving

Introduction 1 Background & Motivation 2 Samsara Overview 3 R & R the Memory Interleaving with HAV 4 5 Conclusion 6 Evaluation 5 Table of Contents

6 Software-only schemes  Modify OS, compiler, runtime libraries or VMM  Virtualization-based approaches—CREW protocol  CREW: Concurrent-Read & Exclusive-Write Background & Motivation Issues  Each memory access operation must be checked for logging  Serious performance degradation (more than 10X)  Huge log size (approximately 1 MB/processor/second)

Background & Motivation 7 Hardware-based schemes  Use special hardware support for recording memory-access interleaving  Redesign the cache coherence protocol Issues  Huge space overhead which limits the duration of the recorded interval  Modeled only using software simulations  Only support Sequential Consistency (SC)  Impractical for use in realistic systems We believe software-only approaches will remain in the focus for optimizations as commercial processors with dedicated hardware-based R&R features are not commonly available yet.

8 Evaluation results show that our system:  Incurs less than 3X overhead when compared to native execution  Reduce 90% log size (even smaller than hardware-based scheme) Combine the merits of both solutions  An Efficient way to record memory interleaving  Without hardware changes  Utilize current hardware acceleration as much as possible Hardware-assisted virtualization  Some hardware characteristics are available to boost performance  Efficient full virtualization using help from hardware capabilities Background & Motivation

Introduction 1 Background & Motivation 2 Samsara Overview 3 R & R the Memory Interleaving with HAV 4 9 Conclusion 6 Evaluation 5 Table of Contents

Samsara Overview 10

Samsara Overview 11 System composition  Controller  DMA recorder  R&R Component  Log recorder daemon Record and Replay Non-deterministic Events  Synchronous Events: record the contents  Asynchronous Events: record timestamp  Compound Events: DMA, record both  Memory interleaving: most important challenge

Introduction 1 Background & Motivation 2 Samsara Overview 3 R & R the Memory Interleaving with HAV 4 12 Conclusion 6 Evaluation 5 Table of Contents

R&R the Memory Interleaving with HAV Extensions 13 Motivation  point-to-point logging approach  Record dependences between pairs of instructions( log size, record overhead )  Avoid the large number of memory access detections  Chunk-based schemes ( only the total sequence of chunks is recorded ) Chunk-based Strategy  Restrict virtual processors’ execution into a series of chunks  Merely need to record commit order  Chunk execution must satisfy:  Atomicity  Serializability

14  Serializability: COW, conflict detection strategy  Atomicity: some instructions that hard to undo R&R the Memory Interleaving with HAV Extensions P0 Chunk Start LD (A) COW ST (A) ST (B) COW Chunk Complete Truncation Reason: I/O Instruction Commit P1 LD (A) Squash & Rollback LD (B) ST (B) Re-execution LD (D) ST (D) Conflict Detection R-set { A } W-set { A, B } Truncation Reason: Chunk Size Limit R-set { D } W-set { D } R-set { A, B } W-set { B } ……

15 Obtain R&W-set Efficiently via HAV Extensions  VM-based approaches: VM exit (hardware page protection)  Our approach: a single EPT traversal  Accessed and Dirty Flags of EPT  Optimization: tree-based design of EPT W(b) R(b) R(a) R(c) W(b) R(b) W(e) W(b) R(b) R(a) R(c) W(b) R(b) W(e) a single EPT traversal Just the first write to each memory page will trigger an EPT violation VM exit  Reduce at least 50% extra VM exits R&R the Memory Interleaving with HAV Extensions

16 R&R the Memory Interleaving with HAV Extensions Observations  Chunk commit is a time-consuming process  The time consumed on waiting for this lock is excessive (40%)  Update write-back operation involves serious performance degradation P0 Chunk Complete Wait for Lock Detect Conflict Broadcast Updates Subsequent Chunk Lock Obtain R&W-set Write-back Updates

17 Reduce Lock Granularity with a Decentralized Three-Phase Commit Protocol  Pages committed concurrently by different chunks have no intersection  Move this out of the synchronized block  Chunk committing out-of-order R&R the Memory Interleaving with HAV Extensions P0 Chunk Complete Wait for Lock Detect Conflict Broadcast Updates Write-back Updates Subsequent Chunk Lock Insert into committing list Update Chunk Info Check Committing List Obtain R&W-set

18 Replay Memory Interleaving  Guarantee all chunks will be properly re-built and executed in the original order  1. Truncate a chunk at the recorded timestamp (hardware performance counter)  2. Ensure that all preceding chunks have been committed successfully before the subsequent chunk starts  Allowing processors execute concurrently in replay R&R the Memory Interleaving with HAV Extensions

Introduction 1 Background & Motivation 2 Samsara Overview 3 R & R the Memory Interleaving with HAV 4 19 Conclusion 6 Evaluation 5 Table of Contents

Evaluation 20 Experimental Setup  4-core Intel Core i processor, 12GB memory, 1TB Hard Drive  Host: Ubuntu with Linux kernel version and Qemu  Guest: Ubuntu with Linux kernel version WorkloadApplication Domain Parallelization Granularity Data Usage Working Set SharingExchange blackscholesFinancial Analysiscoarselow small bodytrackComputer Visionmediumhighmedium raytraceRenderingmediumhighlowunbounded swaptionsFinancial Analysiscoarselow medium freqmineData Miningcoarsehighmediumunbounded x264Media Processingcoarsehigh medium Workloads  PARSEC 3.0

Evaluation 21 Log Size  Samsara generates log at an average rate of MB/s and MB/s for recoding two and four processors, respectively  Reduce 90% log size (even smaller than hardware-based schemes)  For comparison : MB/s (single processor)

Evaluation 22 Log Size  The size of the chunk commit log is practically negligible compared with other non-deterministic events  2.33% with 2 processors and 7.37% with 4 processors on average

Evaluation 23 Recording Overhead Compared to Native Execution  Measure the overhead of the system relative to the base platform it runs on (can not reflect the actual execution time of the system in real life)  2.6X and 5.0X for recording these workloads on two and four processors

Conclusion 24 We made the first attempt to leverage HAV extensions to achieve an efficient software-based replay system  Record processors’ execution as a series of chunks  Avoid the large number of memory access detections by performing a single EPT traversal at the end of each chunk  Propose a decentralized three-phase commit protocol to reduce the lock granularity of the chunk commit process

Thanks 25