PinADX: Customizable Debugging with Dynamic Instrumentation Gregory Lueck, Harish Patil, Cristiano Pereira Intel Corporation CGO 2012, San Jose, USA

Hypothetical Problem 1 (gdb) run Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () (gdb) bt #0 0x0000000000000000 in ?? () #1 0x0000000000000000 in ?? () Crash with bad PC and no stack trace Corrupted return address someplace ... Want to stop BEFORE bad “ret” instruction

Hypothetical Problem 2 thread stack thread stack thread stack thread stack ... ? ? ? ? Massively threaded application How much stack space needed? At what point does each thread use its max stack?

Traditional Debugger Breakpoints? Original Application Application In Debugger Overwrite each “ret” with trap Foo: Foo: … … Debugger catches trap Check if “ret” is to good PC If yes, resume ret trap ret Bar: Bar: … … ret trap ret How can debugger find all “ret” instructions? Horribly slow to trap on each return

Dynamic Binary Instrumenation (DBI) Application Application if (return to bad PC) Breakpoint() if (stack too big) Breakpoint() Foo: Foo: … … sub 0x60, %sp ret Bar: … Instrumentation Bar: ret … sub 0x10, %sp Much faster – avoids trap overhead DBI can find all “ret” instructions reliably General approach – solves stack problem (and others) BUT difficult to integrate with debugger

Pin Overview Tool controls instrumentation (e.g. “if return to bad PC”) Tool Application Code Cache Instrument Optimize Traces Instrumented instructions stored in code cache for efficiency Fetch JIT compiler Store & execute JIT compiler fetches application instructions, calls tool to instrument

JIT Compiler Overview Original Code Code Cache Tool inserts instrumentation (e.g. check if return to bad PC) 1 1’ Dynamic recompilation makes debugging hard 3’ 2 2’ 3 5’ 6’ 4 5 6 Pin

Process running under Pin Microsoft Visual Studio 11 PinADX Architecture Tool extends debugger via instrumentation. Process running under Pin Debugger Application Tool PinADX core GDB or Microsoft Visual Studio 11 Pin PinADX presents “pure” view of application. Hides effect of instrumentation and recompilation. Supports Linux & Windows

Rest of the Talk Introduction / Motivation Example: Using “Stack Debugger” extension Example: Authoring “Stack Debugger” extension Implementing PinADX

Example – Stack Debugger $ pin –appdebug –t stack-debugger.so -- ./my-application Application stopped until continued from debugger. Start GDB, then issue this command at the (gdb) prompt: target remote :1234 Run application under Pin $ gdb ./my-application (gdb) target remote :1234 (gdb) break PrintHello Breakpoint 1 at 0x4004dd: file hw.c, line 13 (gdb) cont Breakpoint 1, PrintHello () at hw.c:13 (gdb) backtrace #0 PrintHello () at hw.c:13 #1 main () at hw.c:7 (gdb) x/2i $pc => 0x4004dd <PrintHello+4>: mov $0x4005e8,%edi 0x4004e2 <PrintHello+9>: callq 0x4003b8 Debugger connected to Pin (gdb) break PrintHello Breakpoint 1 at 0x4004dd: file hw.c, line 13 (gdb) cont Breakpoint 1, PrintHello () at hw.c:13 (gdb) backtrace #0 PrintHello () at hw.c:13 #1 main () at hw.c:7 (gdb) x/2i $pc => 0x4004dd <PrintHello+4>: mov $0x4005e8,%edi 0x4004e2 <PrintHello+9>: callq 0x4003b8

Example – Stack Debugger Stop when application uses too much stack (gdb) monitor stackbreak 4000 Break when thread uses 4000 stack bytes (gdb) cont Stopped: Thread uses 4004 bytes of stack (gdb) monitor stackbreak 4000 Break when thread uses 4000 stack bytes (gdb) cont Stopped: Thread uses 4004 bytes of stack (gdb) backtrace #0 0x3f07214445 in _dl_runtime_resolve () #1 0x00004004e7 in PrintHello () at hw.c:13 #2 0x00004004d2 in main () at hw.c:7 (gdb) monitor stackbreak 10000 Break when thread uses 10000 stack bytes (gdb) break exit Breakpoint 2 at 0x7fffe60f9650 Breakpoint 2, 0x7fffe60f9650 in exit () (gdb) monitor stats Maximum stack usage: 8560 bytes. (gdb) backtrace #0 0x3f07214445 in _dl_runtime_resolve () #1 0x00004004e7 in PrintHello () at hw.c:13 #2 0x00004004d2 in main () at hw.c:7 (gdb) monitor stackbreak 10000 Break when thread uses 10000 stack bytes (gdb) break exit Breakpoint 2 at 0x7fffe60f9650 (gdb) cont Breakpoint 2, 0x7fffe60f9650 in exit () (gdb) monitor stats Maximum stack usage: 8560 bytes.

Rest of the Talk Introduction / Motivation Example: Using “Stack Debugger” extension Example: Authoring “Stack Debugger” extension Implementing PinADX

Stack Debugger – Instrumentation Thread Start: […] sub $0x60, %esp cmp %esi, %edx jle <L1> Record initial stack StackBase = %esp; MaxStack = 0; After each stack-changing instruction size = StackBase - %esp; if (size > MaxStack) MaxStack = size; if (size > StackLimit) TriggerBreakpoint();

Stack Debugger – Implementation Instrument only instructions that change $SP Call after each instruction VOID Instruction(INS ins, VOID *) { if (INS_RegWContain(ins, REG_STACK_PTR)) { IPOINT where = (INS_HasFallThrough(ins)) ? IPOINT_AFTER : IPOINT_TAKEN_BRANCH; INS_InsertCall(ins, where, (AFUNPTR)OnStackChange, IARG_REG_VALUE, REG_STACK_PTR, IARG_THREAD_ID, IARG_CONTEXT, IARG_END); } } VOID OnStackChange(ADDRINT sp, THREADID tid, CONTEXT *ctxt) { size_t size = StackBase - sp; if (size > StackMax) StackMax = size; if (size > StackLimit) { ostringstream os; os << "Stopped: Thread uses " << size << " stack bytes."; PIN_ApplicationBreakpoint(ctxt, tid, FALSE, os.str()); } } Instrumentation Analysis

Stack Debugger – Implementation int main() { […] PIN_AddDebugInterpreter(HandleDebugCommand, 0); } BOOL HandleDebugCommand(const string &cmd, string *result) { if (cmd == "stats") { ostringstream os; os << "Maximum stack usage: " << StackMax << " bytes.\n"; *result = os.str(); return TRUE; else if (cmd.find("stackbreak ") == 0) { StackLimit = /* parse limit */; ostringstream os; os << "Break when thread uses " << limit << " stack bytes."; *result = os.str(); return TRUE; return FALSE; // Unknown command

Visual Studio IDE Extension

Other Debugger Extensions Intel Inspector XE Product Memory Checker Thread Checker Intel SDE: Instruction emulation Debug from log file (PinPlay, CGO 2010) Dynamic slicing (Rajiv Gupta, UC Riverside) Cmp$im: Cache simulator Write your own!

Rest of the Talk Introduction / Motivation Example: Using “Stack Debugger” extension Example: Authoring “Stack Debugger” extension Implementing PinADX

Process running under Pin Microsoft Visual Studio 11 PinADX Architecture Tool extends debugger via instrumentation. Process running under Pin Debugger Application Tool PinADX core GDB or Microsoft Visual Studio 11 Pin PinADX presents “pure” view of application. Hides effect of instrumentation and recompilation.

Communication Details Commands Read / write registers, memory Set breakpoints Continue, single-step, stop Pin PinADX core Debugger Notifications Breakpoint triggered Caught signal Application exited Very low level Symbol processing in debugger Expression evaluation in debugger Extension of GDB’s remote debugging protocol

Communication Details Commands Read / write registers, memory Set breakpoints Continue, single-step, stop Pin PinADX core Debugger Notifications Breakpoint triggered Caught signal Application exited Breakpoint alternatives Insert real INT3 trap instruction Virtualize inside Pin VM See paper for details

Breakpoint BP Original Code Code Cache 1 1’ 2 2’ 5 3 3’ Execution stops in Pin Waits for GDB to continue 6 BP 4 PinADX core Debugger set breakpoint at 4 continue breakpoint notification

Single Step Original Code Code Cache 1 1’ 2 5 3 Execution stops in Pin Waits for GDB to continue 6 4 PinADX core Debugger step complete notification do single-step

Thanks Mark Charney – SDE software emulator Andria Pazarloglou – Created VS11 GUI plugin Gregg Miskelly – Microsoft VS11 debugger architect Robert Cohn – Father of Pin

Summary DBI can implement powerful debugger features API allows Pin tools to extend debugger easily Multi-platform Linux: GDB Windows: Microsoft Visual Studio 11 (soon) Works with off-the-shelf debuggers http://pintool.org