Software & Services Group PinADX: Customizable Debugging with Dynamic Instrumentation Gregory Lueck, Harish Patil, Cristiano Pereira Intel Corporation CGO 2012, San Jose, USA 1

Software & Services Group Hypothetical Problem 1 2 (gdb) run Program received signal SIGSEGV, Segmentation fault. 0x in ?? () (gdb) bt #0 0x in ?? () #1 0x in ?? () (gdb) run Program received signal SIGSEGV, Segmentation fault. 0x in ?? () (gdb) bt #0 0x in ?? () #1 0x in ?? () Crash with bad PC and no stack trace Corrupted return address someplace... Want to stop BEFORE bad “ret” instruction

Software & Services Group Hypothetical Problem 2 3 thread stack ? ? ? ?... Massively threaded application How much stack space needed? At what point does each thread use its max stack?

Software & Services Group Traditional Debugger Breakpoints? 4 Original Application … 1.Debugger catches trap 2.Check if “ret” is to good PC 3.If yes, resume How can debugger find all “ret” instructions? Horribly slow to trap on each return Foo: ret Bar: … ret Application In Debugger … Foo: trap Bar: … trap Overwrite each “ret” with trap ret

Software & Services Group Dynamic Binary Instrumenation (DBI) 5 Application if (return to bad PC) Breakpoint() Application if (stack too big) Breakpoint() Much faster – avoids trap overhead DBI can find all “ret” instructions reliably General approach – solves stack problem (and others) BUT difficult to integrate with debugger sub 0x60, %sp … Foo: ret Bar: … ret Foo: Bar: sub 0x10, %sp … … Instrumentation

Software & Services Group Pin Overview Tool JIT compiler Application Code Cache Traces Tool controls instrumentation (e.g. “if return to bad PC”) JIT compiler fetches application instructions, calls tool to instrument Instrumented instructions stored in code cache for efficiency Fetch Instrument Optimize Store & execute 6

Software & Services Group Original CodeCode Cache JIT Compiler Overview 1’ ’ 6’ Pin 4 3’ 5’ Tool inserts instrumentation (e.g. check if return to bad PC) Dynamic recompilation makes debugging hard 7

Software & Services Group Process running under Pin PinADX Architecture 8 Tool Pin Application Debugger PinADX presents “pure” view of application. Hides effect of instrumentation and recompilation. Tool extends debugger via instrumentation. PinADX core GDB or Microsoft Visual Studio 11 Supports Linux & Windows

Software & Services Group Rest of the Talk Introduction / Motivation Example: Using “Stack Debugger” extension Example: Authoring “Stack Debugger” extension Implementing PinADX 9

Software & Services Group $ gdb./my-application (gdb) target remote :1234 (gdb) break PrintHello Breakpoint 1 at 0x4004dd: file hw.c, line 13 (gdb) cont Breakpoint 1, PrintHello () at hw.c:13 (gdb) backtrace #0 PrintHello () at hw.c:13 #1 main () at hw.c:7 (gdb) x/2i $pc => 0x4004dd : mov $0x4005e8,%edi 0x4004e2 : callq 0x4003b8 Example – Stack Debugger 10 $ pin –appdebug –t stack-debugger.so --./my-application Application stopped until continued from debugger. Start GDB, then issue this command at the (gdb) prompt: target remote :1234 Run application under Pin Debugger connected to Pin (gdb) cont Breakpoint 1, PrintHello () at hw.c:13 (gdb) backtrace #0 PrintHello () at hw.c:13 #1 main () at hw.c:7 (gdb) x/2i $pc => 0x4004dd : mov $0x4005e8,%edi 0x4004e2 : callq 0x4003b8 (gdb) break PrintHello Breakpoint 1 at 0x4004dd: file hw.c, line 13

Software & Services Group (gdb) monitor stackbreak 4000 Break when thread uses 4000 stack bytes (gdb) cont Stopped: Thread uses 4004 bytes of stack (gdb) backtrace #0 0x3f in _dl_runtime_resolve () #1 0x e7 in PrintHello () at hw.c:13 #2 0x d2 in main () at hw.c:7 (gdb) monitor stackbreak Break when thread uses stack bytes (gdb) break exit Breakpoint 2 at 0x7fffe60f9650 (gdb) cont Breakpoint 2, 0x7fffe60f9650 in exit () (gdb) monitor stats Maximum stack usage: 8560 bytes. Example – Stack Debugger 11 Stop when application uses too much stack (gdb) monitor stackbreak 4000 Break when thread uses 4000 stack bytes (gdb) cont Stopped: Thread uses 4004 bytes of stack (gdb) monitor stackbreak Break when thread uses stack bytes (gdb) break exit Breakpoint 2 at 0x7fffe60f9650 (gdb) cont Breakpoint 2, 0x7fffe60f9650 in exit () (gdb) backtrace #0 0x3f in _dl_runtime_resolve () #1 0x e7 in PrintHello () at hw.c:13 #2 0x d2 in main () at hw.c:7 (gdb) monitor stats Maximum stack usage: 8560 bytes.

Software & Services Group Rest of the Talk Introduction / Motivation Example: Using “Stack Debugger” extension Example: Authoring “Stack Debugger” extension Implementing PinADX 12

Software & Services Group Stack Debugger – Instrumentation 13 Thread Start: […] sub$0x60, %esp cmp%esi, %edx jle size = StackBase - %esp; if (size > MaxStack) MaxStack = size; if (size > StackLimit) TriggerBreakpoint(); StackBase = %esp; MaxStack = 0; After each stack-changing instruction Record initial stack

Software & Services Group VOID Instruction(INS ins, VOID *) { if (INS_RegWContain(ins, REG_STACK_PTR)) { IPOINT where = (INS_HasFallThrough(ins)) ? IPOINT_AFTER : IPOINT_TAKEN_BRANCH; INS_InsertCall(ins, where, (AFUNPTR)OnStackChange, IARG_REG_VALUE, REG_STACK_PTR, IARG_THREAD_ID, IARG_CONTEXT, IARG_END); } } VOID OnStackChange(ADDRINT sp, THREADID tid, CONTEXT *ctxt) { size_t size = StackBase - sp; if (size > StackMax) StackMax = size; if (size > StackLimit) { ostringstream os; os << "Stopped: Thread uses " << size << " stack bytes."; PIN_ApplicationBreakpoint(ctxt, tid, FALSE, os.str()); } } Stack Debugger – Implementation 14 Instrumentation Analysis Instrument only instructions that change $SP Call after each instruction

Software & Services Group int main() { […] PIN_AddDebugInterpreter(HandleDebugCommand, 0); } BOOL HandleDebugCommand(const string &cmd, string *result) { if (cmd == "stats") { ostringstream os; os << "Maximum stack usage: " << StackMax << " bytes.\n"; *result = os.str(); return TRUE; } else if (cmd.find("stackbreak ") == 0) { StackLimit = /* parse limit */; ostringstream os; os << "Break when thread uses " << limit << " stack bytes."; *result = os.str(); return TRUE; } return FALSE; // Unknown command } Stack Debugger – Implementation 15

Software & Services Group Visual Studio IDE Extension 16

Software & Services Group Other Debugger Extensions Intel Inspector XE Product –Memory Checker –Thread Checker Intel SDE: Instruction emulation Debug from log file (PinPlay, CGO 2010) Dynamic slicing (Rajiv Gupta, UC Riverside) Cmp$im: Cache simulator Write your own! 17

Software & Services Group Rest of the Talk Introduction / Motivation Example: Using “Stack Debugger” extension Example: Authoring “Stack Debugger” extension Implementing PinADX 18

Software & Services Group Process running under Pin PinADX Architecture 19 Tool Pin Application Debugger PinADX presents “pure” view of application. Hides effect of instrumentation and recompilation. Tool extends debugger via instrumentation. PinADX core GDB or Microsoft Visual Studio 11

Software & Services Group Communication Details 20 Very low level Symbol processing in debugger Expression evaluation in debugger Extension of GDB’s remote debugging protocol Commands Read / write registers, memory Set breakpoints Continue, single-step, stop Notifications Breakpoint triggered Caught signal Application exited Debugger PinADX core Pin

Software & Services Group Communication Details 21 Breakpoint alternatives Insert real INT3 trap instruction Virtualize inside Pin VM See paper for details Commands Read / write registers, memory Set breakpoints Continue, single-step, stop Notifications Breakpoint triggered Caught signal Application exited Debugger PinADX core Pin

Software & Services Group set breakpoint at 4 continue Code CacheOriginal Code Breakpoint 1’ breakpoint notification Execution stops in Pin Waits for GDB to continue BP 2’ 3’ PinADX core Debugger 22

Software & Services Group Code CacheOriginal Code Single Step 1’ step complete notification Debugger Execution stops in Pin Waits for GDB to continue do single-step PinADX core 23

Software & Services Group Thanks Mark Charney – SDE software emulator Andria Pazarloglou – Created VS11 GUI plugin Gregg Miskelly – Microsoft VS11 debugger architect Robert Cohn – Father of Pin 24

Software & Services Group Summary DBI can implement powerful debugger features API allows Pin tools to extend debugger easily Multi-platform –Linux: GDB –Windows: Microsoft Visual Studio 11 (soon) Works with off-the-shelf debuggers 25