Execution Replay for Multiprocessor Virtual Machines George W. Dunlap Dominic Lucchetti Michael A. Fetterman Peter M. Chen
Big ideas Detection and replay of memory races is possible on commodity hardware Overhead high for some workloads …but surprisingly low for other workloads
Execution Replay CPU Memory Disk Network Keyboard, mouse Interrupts
Uses of Execution Replay Reconstructing state –Fault tolerance Reconstructing execution –Debugging –Realistic trace generation Both –Intrusion analysis
Single-processor Replay Basic principles well understood –Log all non-deterministic inputs –Timing of asynchronous events Minimal overhead (Dunlap02) –13% worst case –Log for months or years Available commercially –VMWare: Record/Replay
Replay for Multiprocessors Memory races in multiprocessor VMs The Ordering Requirement The CREW Protocol –Implementing with page protections –Relation to the Ordering Requirement –Generating constrants from CREW events DMA-capable devices and CREW Performance
The Multiprocessor Challenge Interleaved reads and writes –Fine-grained non-determinism –Much more difficult Existing solutions –Hardware modification –Software instrumentation SMP-ReVirt –Hardware MMU to detect sharing
Multiprocessor Replay P2 Memory P1 P2 n=3 n=5 if (n<4)
Ordering Memory Accesses Preserving order will reproduce execution –a→b: “a happens-before b” –Ordering is transitive: a→b, b→c means a→c Two instructions must be ordered if: –they both access the same memory, and –one of them is a write
Constraints: Enforcing order To guarantee a→d: –a→d–a→d –b→d–b→d –a→c–a→c –b→c–b→c Suppose we need b→c –b→c is necessary –a→d is redundant P1 a b c d P2 overconstrained
CREW Protocol Each shared object in one of two states: –Concurrent-Read: all processors can read, none can write –Exclusive-Write: one processor (the owner) can read and write; others have no access
CREW protocol, con’t Enforced with hardware MMU –Read/write –Read-only –None Change CREW states on demand –Fault, fixup, re-execute CREW event –Increasing or reducing permission due to CREW state changes
CREW Property If two instructions on different processors: –access the same page, –and one of them is a write, –there will be a CREW event on each processor between them.
Generating Constraints State: Concurrent Read –All processors read-only d*: CREW fault New state: P2 Exclusive r: privilege reduction –Read to None i: privilege increase –Read to Read/write Log timing of r and i Constraint: –r → i P1 a d P2 r i d*
Direct Memory Access Device accesses memory directly Logically another processor –Reads and writes need to be ordered –IOMMU: can’t fault/fixup/re-execute Observation: Transaction model Device: non-preemptible actor
Prototype: SMP-ReVirt Modified Xen hypervisor Implement logging, CREW protocol Details in paper
Evaluation questions What is the overhead? What affects performance? –In paper When might I want to use MP? –Log with 1, 2, or N cpus?
Evaluation Workloads SPLASH2 parallel application suite –FMM, LU, ocean, radix, water-spatial, radiosity Kernel-build Dbench
Predicting results Key changes in sharing attributes –4096-byte sharing granularity –“Miss” is very expensive SPLASH2 –Good: high spatial locality / low false sharing –Bad: random access patterns / high false sharing The Linux kernel –Tuned to 16-byte cacheline –Involving the kernel may be expensive
Single-processor Xen guests
Log Growth Rate WorkloadLog growth(GB/day)Days to fill 300GB FMM LU Ocean Radix Water-spatial Kernel-build Radiosity Dbench
2-processor Xen guests
2-processor, con’t
Log Growth Rate WorkloadLog growth(GB/day)Days to fill 300GB FMM LU Ocean Radix Water-spatial Kernel-build Radiosity Dbench
4-processor Xen guests
Recap Memory races in multiprocessor VMs The Ordering Requirement The CREW Protocol –Implementing with page protections –Relation to the Ordering Requirement –Generating constrants from CREW events DMA-capable devices and CREW Performance
Big ideas Detection and replay of memory races is possible on commodity hardware Overhead high for some workloads …but surprisingly low for other workloads
Questions