Execution Replay for Multiprocessor Virtual Machines George W. Dunlap Dominic Lucchetti Michael A. Fetterman Peter M. Chen

Big ideas Detection and replay of memory races is possible on commodity hardware Overhead high for some workloads …but surprisingly low for other workloads

Execution Replay CPU Memory Disk Network Keyboard, mouse Interrupts

Uses of Execution Replay Reconstructing state –Fault tolerance Reconstructing execution –Debugging –Realistic trace generation Both –Intrusion analysis

Single-processor Replay Basic principles well understood –Log all non-deterministic inputs –Timing of asynchronous events Minimal overhead (Dunlap02) –13% worst case –Log for months or years Available commercially –VMWare: Record/Replay

Replay for Multiprocessors Memory races in multiprocessor VMs The Ordering Requirement The CREW Protocol –Implementing with page protections –Relation to the Ordering Requirement –Generating constrants from CREW events DMA-capable devices and CREW Performance

The Multiprocessor Challenge Interleaved reads and writes –Fine-grained non-determinism –Much more difficult Existing solutions –Hardware modification –Software instrumentation SMP-ReVirt –Hardware MMU to detect sharing

Multiprocessor Replay P2 Memory P1 P2 n=3 n=5 if (n<4)

Ordering Memory Accesses Preserving order will reproduce execution –a→b: “a happens-before b” –Ordering is transitive: a→b, b→c means a→c Two instructions must be ordered if: –they both access the same memory, and –one of them is a write

Constraints: Enforcing order To guarantee a→d: –a→d–a→d –b→d–b→d –a→c–a→c –b→c–b→c Suppose we need b→c –b→c is necessary –a→d is redundant P1 a b c d P2 overconstrained

CREW Protocol Each shared object in one of two states: –Concurrent-Read: all processors can read, none can write –Exclusive-Write: one processor (the owner) can read and write; others have no access

CREW protocol, con’t Enforced with hardware MMU –Read/write –Read-only –None Change CREW states on demand –Fault, fixup, re-execute CREW event –Increasing or reducing permission due to CREW state changes

CREW Property If two instructions on different processors: –access the same page, –and one of them is a write, –there will be a CREW event on each processor between them.

Generating Constraints State: Concurrent Read –All processors read-only d*: CREW fault New state: P2 Exclusive r: privilege reduction –Read to None i: privilege increase –Read to Read/write Log timing of r and i Constraint: –r → i P1 a d P2 r i d*

Direct Memory Access Device accesses memory directly Logically another processor –Reads and writes need to be ordered –IOMMU: can’t fault/fixup/re-execute Observation: Transaction model Device: non-preemptible actor

Prototype: SMP-ReVirt Modified Xen hypervisor Implement logging, CREW protocol Details in paper

Evaluation questions What is the overhead? What affects performance? –In paper When might I want to use MP? –Log with 1, 2, or N cpus?

Evaluation Workloads SPLASH2 parallel application suite –FMM, LU, ocean, radix, water-spatial, radiosity Kernel-build Dbench

Predicting results Key changes in sharing attributes –4096-byte sharing granularity –“Miss” is very expensive SPLASH2 –Good: high spatial locality / low false sharing –Bad: random access patterns / high false sharing The Linux kernel –Tuned to 16-byte cacheline –Involving the kernel may be expensive

Single-processor Xen guests

Log Growth Rate WorkloadLog growth(GB/day)Days to fill 300GB FMM LU Ocean Radix Water-spatial Kernel-build Radiosity Dbench

2-processor Xen guests

2-processor, con’t

Log Growth Rate WorkloadLog growth(GB/day)Days to fill 300GB FMM LU Ocean Radix Water-spatial Kernel-build Radiosity Dbench

4-processor Xen guests

Recap Memory races in multiprocessor VMs The Ordering Requirement The CREW Protocol –Implementing with page protections –Relation to the Ordering Requirement –Generating constrants from CREW events DMA-capable devices and CREW Performance

Big ideas Detection and replay of memory races is possible on commodity hardware Overhead high for some workloads …but surprisingly low for other workloads

Questions