1. PinADX: Customizable Debugging with Dynamic Instrumentation
Gregory Lueck, Harish Patil, Cristiano Pereira
Intel Corporation
CGO 2012, San Jose, USA

2. Hypothetical Problem 1
(gdb) run
Program received signal SIGSEGV, Segmentation fault.
0x in ?? ()
(gdb) bt
#0 0x in ?? ()
#1 0x in ?? ()
Crash with bad PC and no stack trace Corrupted return address someplace ... Want to stop BEFORE bad “ret” instruction

3. Hypothetical Problem 2
thread stack
thread stack
thread stack
thread stack
... ? ? ? ?
Massively threaded application How much stack space needed? At what point does each thread use its max stack?

4. Traditional Debugger Breakpoints?
Original
Application
Application
In Debugger
Overwrite each “ret” with trap
Foo:
Foo: … …
Debugger catches trap
Check if “ret” is to good PC
If yes, resume
ret
trap
ret
Bar:
Bar: … …
ret
trap
ret
How can debugger find all “ret” instructions? Horribly slow to trap on each return

5. Dynamic Binary Instrumenation (DBI)
Application
Application
if (return to bad PC)
Breakpoint()
if (stack too big)
Breakpoint()
Foo:
Foo: … …
sub 0x60, %sp
ret
Bar: …
Instrumentation
Bar:
ret …
sub 0x10, %sp
Much faster – avoids trap overhead DBI can find all “ret” instructions reliably General approach – solves stack problem (and others) BUT difficult to integrate with debugger

6. Pin Overview Tool controls instrumentation
(e.g. “if return to bad PC”)
Tool
Application
Code Cache
Instrument
Optimize
Traces
Instrumented instructions stored in code cache for efficiency
Fetch
JIT compiler
Store &
execute
JIT compiler fetches application instructions, calls tool to instrument

7. JIT Compiler Overview
Original Code
Code Cache
Tool inserts instrumentation (e.g. check if return to bad PC) 1 1’
Dynamic recompilation makes debugging hard 3’ 2 2’ 3 5’ 6’ 4 5 6
Pin

8. Process running under Pin Microsoft Visual Studio 11
PinADX Architecture
Tool extends debugger via instrumentation.
Process running under Pin
Debugger
Application
Tool
PinADX core
GDB or
Microsoft Visual Studio 11
Pin
PinADX presents “pure” view of application. Hides effect of instrumentation and recompilation.
Supports Linux & Windows

9. Rest of the Talk Introduction / Motivation
Example: Using “Stack Debugger” extension
Example: Authoring “Stack Debugger” extension
Implementing PinADX

10. Example – Stack Debugger
$ pin –appdebug –t stack-debugger.so -- ./my-application
Application stopped until continued from debugger.
Start GDB, then issue this command at the (gdb) prompt:
target remote :1234
Run application under Pin
$ gdb ./my-application
(gdb) target remote :1234
(gdb) break PrintHello
Breakpoint 1 at 0x4004dd: file hw.c, line 13
(gdb) cont
Breakpoint 1, PrintHello () at hw.c:13
(gdb) backtrace
#0 PrintHello () at hw.c:13
#1 main () at hw.c:7
(gdb) x/2i $pc
=> 0x4004dd <PrintHello+4>: mov $0x4005e8,%edi
0x4004e2 <PrintHello+9>: callq 0x4003b8
Debugger connected to Pin
(gdb) break PrintHello
Breakpoint 1 at 0x4004dd: file hw.c, line 13
(gdb) cont
Breakpoint 1, PrintHello () at hw.c:13
(gdb) backtrace
#0 PrintHello () at hw.c:13
#1 main () at hw.c:7
(gdb) x/2i $pc
=> 0x4004dd <PrintHello+4>: mov $0x4005e8,%edi
0x4004e2 <PrintHello+9>: callq 0x4003b8

11. Example – Stack Debugger
Stop when application uses too much stack
(gdb) monitor stackbreak 4000
Break when thread uses 4000 stack bytes
(gdb) cont
Stopped: Thread uses 4004 bytes of stack
(gdb) monitor stackbreak 4000
Break when thread uses 4000 stack bytes
(gdb) cont
Stopped: Thread uses 4004 bytes of stack
(gdb) backtrace
#0 0x3f in _dl_runtime_resolve ()
#1 0x e7 in PrintHello () at hw.c:13
#2 0x d2 in main () at hw.c:7
(gdb) monitor stackbreak 10000
Break when thread uses stack bytes
(gdb) break exit
Breakpoint 2 at 0x7fffe60f9650
Breakpoint 2, 0x7fffe60f9650 in exit ()
(gdb) monitor stats
Maximum stack usage: 8560 bytes.
(gdb) backtrace
#0 0x3f in _dl_runtime_resolve ()
#1 0x e7 in PrintHello () at hw.c:13
#2 0x d2 in main () at hw.c:7
(gdb) monitor stackbreak 10000
Break when thread uses stack bytes
(gdb) break exit
Breakpoint 2 at 0x7fffe60f9650
(gdb) cont
Breakpoint 2, 0x7fffe60f9650 in exit ()
(gdb) monitor stats
Maximum stack usage: 8560 bytes.

12. Rest of the Talk Introduction / Motivation
Example: Using “Stack Debugger” extension
Example: Authoring “Stack Debugger” extension
Implementing PinADX

13. Stack Debugger – Instrumentation
Thread Start:
[…]
sub $0x60, %esp
cmp %esi, %edx
jle <L1>
Record initial stack
StackBase = %esp;
MaxStack = 0;
After each stack-changing instruction
size = StackBase - %esp;
if (size > MaxStack) MaxStack = size;
if (size > StackLimit) TriggerBreakpoint();

14. Stack Debugger – Implementation
Instrument only instructions that change $SP
Call after each instruction
VOID Instruction(INS ins, VOID *) { if (INS_RegWContain(ins, REG_STACK_PTR)) { IPOINT where = (INS_HasFallThrough(ins)) ? IPOINT_AFTER : IPOINT_TAKEN_BRANCH; INS_InsertCall(ins, where, (AFUNPTR)OnStackChange, IARG_REG_VALUE, REG_STACK_PTR, IARG_THREAD_ID, IARG_CONTEXT, IARG_END); } }
VOID OnStackChange(ADDRINT sp, THREADID tid, CONTEXT *ctxt) { size_t size = StackBase - sp; if (size > StackMax) StackMax = size; if (size > StackLimit) { ostringstream os; os << "Stopped: Thread uses " << size << " stack bytes."; PIN_ApplicationBreakpoint(ctxt, tid, FALSE, os.str()); } }
Instrumentation
Analysis

15. Stack Debugger – Implementation
int main() {
[…]
PIN_AddDebugInterpreter(HandleDebugCommand, 0); }
BOOL HandleDebugCommand(const string &cmd, string *result) { if (cmd == "stats") {
ostringstream os;
os << "Maximum stack usage: " << StackMax << " bytes.\n";
*result = os.str();
return TRUE;
else if (cmd.find("stackbreak ") == 0) { StackLimit = /* parse limit */; ostringstream os; os << "Break when thread uses " << limit << " stack bytes."; *result = os.str(); return TRUE;
return FALSE; // Unknown command

16. Visual Studio IDE Extension

17. Other Debugger Extensions
Intel Inspector XE Product
Memory Checker
Thread Checker
Intel SDE: Instruction emulation
Debug from log file (PinPlay, CGO 2010)
Dynamic slicing (Rajiv Gupta, UC Riverside)
Cmp$im: Cache simulator
Write your own!

18. Rest of the Talk Introduction / Motivation
Example: Using “Stack Debugger” extension
Example: Authoring “Stack Debugger” extension
Implementing PinADX

19. Process running under Pin Microsoft Visual Studio 11
PinADX Architecture
Tool extends debugger via instrumentation.
Process running under Pin
Debugger
Application
Tool
PinADX core
GDB or
Microsoft Visual Studio 11
Pin
PinADX presents “pure” view of application. Hides effect of instrumentation and recompilation.

20. Communication Details
Commands
Read / write registers, memory
Set breakpoints
Continue, single-step, stop
Pin
PinADX core
Debugger
Notifications
Breakpoint triggered
Caught signal
Application exited
Very low level
Symbol processing in debugger
Expression evaluation in debugger
Extension of GDB’s remote debugging protocol

21. Communication Details
Commands
Read / write registers, memory
Set breakpoints
Continue, single-step, stop
Pin
PinADX core
Debugger
Notifications
Breakpoint triggered
Caught signal
Application exited
Breakpoint alternatives
Insert real INT3 trap instruction
Virtualize inside Pin VM
See paper for details

22. Breakpoint BP Original Code Code Cache 1 1’ 2 2’ 5 3 3’
Execution stops in Pin
Waits for GDB to continue 6 BP 4
PinADX core
Debugger
set breakpoint at 4
continue
breakpoint notification

23. Single Step Original Code Code Cache 1 1’ 2 5 3 Execution stops in Pin
Waits for GDB to continue 6 4
PinADX core
Debugger
step complete notification
do single-step

24. Thanks Mark Charney – SDE software emulator
Andria Pazarloglou – Created VS11 GUI plugin
Gregg Miskelly – Microsoft VS11 debugger architect
Robert Cohn – Father of Pin

25. Summary DBI can implement powerful debugger features
API allows Pin tools to extend debugger easily
Multi-platform
Linux: GDB
Windows: Microsoft Visual Studio 11 (soon)
Works with off-the-shelf debuggers