Deterministic Logging/Replaying of Applications

Motivation Run-time framework goals –Collect a complete trace of a program’s user-mode execution –Keep the tracing overhead for both space and time low –Re-simulate the traced execution deterministically based on the collected trace with full fidelity down to the instruction level Full fidelity: user mode only, no tracing of kernel, only user-mode I/O callbacks Advantages –Complete program trace that can be analyzed from multiple perspectives (replay analyzers: debuggers, locality, etc) –Trace can be collected on one machine and re-played on other machines (or perform live analysis by streaming) Challenges: Trace Size and Performance

Original Record-Replay Approaches InstantReplay ’87 – Record order or memory accesses – overhead may affect program behavior RecPlay ’00 – Record only synchronizations – Not deterministic if have data races Netzer ’93 – Record optimal trace – too expensive to keep track of all memory locations Bacon & Goldstein ’91 – Record memory bus transactions with hardware – high logging bandwidth

Motivation Increasing use and development for multi-core processors MT program behavior is non-deterministic To effectively debug software, developers must be able to replay executions that exhibit concurrency bugs –Shared memory updates happen in different order

Related Concepts Runtime interpretation/translation of binary instructions –Requires no static instrumentation, or special symbol information –Handle dynamically generated code, self modifying code –Recording/Logging: ~ x More recent logging –Proposed hardware support (for MT domain) –FDR (Flight Data Recorder) –BugNet (cache bits set on first load) –RTR (Regulated Transitive Reduction) –DeLorean (ISCA chunks of instructions) –Strata (time layer across all the logs for the running threads) –iDNA (Diagnostic infrastructure using NirvanA- Microsoft)

Deterministic Replay Re-execute the exact same sequence of instructions as recorded in a previous run Single threaded programs –Record Load Values needed for reproducing behavior of a run (Load Log) –Registers updated by system calls and signal handlers (Reg Log) –Output of special instructions: RDTSC, CPUID (Reg Log) –System call (virtualization- cloning arguments, updates) –Checkpointing (log summary ~10Million) Multi-threaded programs –Log interleaving among threads (shared memory updates ordering – SMO Log)

PinSEL – System Effect Log (SEL) Logging program load values needed for deterministic replay: –First access from a memory location –Values modified by the system (system effect) and read by program –Machine and time sensitive instructions (cpuid,rdtsc) Load A; (A = 111) Logged Not Logged Syscall modifies location (B  0) and (C  99) Load C; (C = 99) Load D; (D = 10) Store A; (A  111) Store B; (B  55) Load B; (B = 0) system call Program execution Load C; (C = 9) Load D; (D = 10) Trace size is ~4-5 bytes per instruction

Optimization: Trace select reads Observation: Hardware caches eliminate most off-chip reads Optimize logging: –Logger and replayer simulate identical cache memories –Simple cache (the memory copy structure) to decide which values to log. No tags or valid bits to check. If the values mismatch they are logged. Average trace size is <1 bit per instruction i = 1; for (j = 0; j < 10; j++) { i = i + j; } k = i; // value read is 46 System_call(); k = i; // value read is 0 (not predicted) The only read not predicted and logged follows the system call

Example Overhead –PinSEL and PinPLAY Initial work (2006) with single threaded programs: –SPEC2000 ref runs: 130x slowdown for pinSEL and ~80x for PinPLAY (w/o in-lining) Working with a subset of SPLASH2 benchmarks: 230x slowdown for PinSEL –Now: Geo-mean SPEC2006 Pin 1.4x Logger 83.6x Replayer 1.4x

Example: Microsoft iDNA Trace Writer Performance Application Simulated Instructions (millions) Trace File Size Trace File Bits / Instruction Native Execution Time Execution Time While Tracing Execution Overhead Gzip24, MB s187s15.98 Excel1,78199 MB s105s5.76 Power Point 7, MB s247s5.66 IE1165 MB s6.94s13.90 Vulcan2, MB s46.6s17.01 Satsolver9, MB s127s12.98 Memchecker and valgrind are in 30-40x range on CPU 2006 iDNA ~11x, (does not log shared-memory dependences explicitly) Use a sequential number for every lock prefixed memory operation: offline data race analysis

Logging Shared Memory Ordering –Emulation of Directory Based Cache Coherence Identifies RAW, WAR, WAW dependences Indexed by hashing effective address Each entry represents an address range Store A Load B Program execution hash Dir Entry Directory

Directory Entries –Every DirEntry maintains: Thread id of the last_writer A timestamp is the # of memory ref. the thread has executed Vector of timestamps of last access for each thread to that entry On Loads: update the timestamp for the thread in the entry On Stores: update the timestamp and the last_writer fields Program execution Thread T1 Thread T2 Last writer id: 1: Store A 2: Load A DirEntry: [A:D] Last writer id: DirEntry: [E:H] Directory T1:T2: T1:T2: 1: Load F 2: Store A 3: Load F 3: Store F T1 1 1 T T1 3 Vector

Detecting Dependences –RAW dependency between threads T and T’ is established if: T executes a load that maps to the directory entry A T’ is the last_writer for the same entry –WAW dependency between T and T’ is established if: T executes a store that maps to the directory entry A T’ is the last_writer for the same entry –WAR dependency between T and T’ is established if: T executes a store that maps to the directory entry A T’ has accessed the same entry in the past and T is not the last_writer

Example Program execution Thread T1 Thread T2 Last writer id: 1: Store A 2: Load A DirEntry: [A:D] Last writer id: DirEntry: [E:H] T1:T2: T1:T2: 1: Load F 2: Store A 3: Load F 3: Store F T1 1 1 T T1 3 WAW RAW WAR T1 2 T2 2 T1 3 T2 3 T2 2 T1 1 SMO logs: Thread T1 cannot execute memory reference 2 until T2 executes its memory reference 2 Thread T2 cannot execute memory reference 2 until T1 executes its memory reference 1 Last access to the DirEntry Last_writer Last access to the DirEntry

Ordering Memory Accesses (Reducing log size) Preserving order will reproduce execution –a→b: “a happens-before b” –Ordering is transitive: a→b, b→c means a→c Two instructions must be ordered if: –they both access the same memory, and –one of them is a write

Constraints: Enforcing Order To guarantee a→d: –a→d–a→d –b→d–b→d –a→c–a→c –b→c–b→c Suppose we need b→c –b→c is necessary –a→d is redundant P1 a b c d P2 overconstrained

Reproduce exact same conflicts: no more, no less Problem Formulation ld A Thread I Thread J Recording st B st C sub ld B add st C ld B st A st C Thread I Thread J Replay Log ld D st D ld A st B st C sub ld B add st C ld B st A st C ld D st D Conflicts (red) Dependence (black)

 Detect conflicts  Write log Log All Conflicts ld A Thread I Thread J Replay st B st C sub ld B add st C ld B st A st C ld D st D Log J : 2  3 1  4 3  5 4  6 Log I : 2  3 Log Size: 5*16=80 bytes (10 integers) Dependence Log 16 bytes Assign IC (logical Timestamps) But too many conflicts

Netzer’s Transitive Reduction ld A Thread I Thread J Replay st B st C sub ld B add st C ld B st A st C ld D st D TR reduced Log J : 2  3 3  5 4  6 Log I : 2  3 Log Size: 64 bytes (8 integers) TR Reduced Log

RTR (Regulated Transitive Reduction): Stricter Dependences to Aid Vectorization ld A Thread I Thread J Replay st B st C sub ld B add st C ld B st A st C ld D st D Log J : 2  3 4  5 Log I : 2  3 Log Size: 48 bytes (6 integers) New Reduced Log stricter Reduced 4% Overhead.2 MB/core/second logging (Apache)