iProbe: A Lightweight User- Space Instrumentation Tool Nipun Arora, Hui Zhang, Junghwan Rhee, Kenji Yoshihira, Guofei Jiang Autonomic Management Group Princeton, NJ

Motivation

} Background } The current state-of-art monitoring mechanisms use Trampoline (DTrace, DynInst) Just-in-Time Compilation (PIN, Valgrind) Source-code /Compiler-Driven } Blackbox but slow } Developer -driven but fast Ensuring stability & robustness when rewriting the binary Context Switch to Kernel or Instrumentation Functions Fixing and simulating overwritten Instructions

} } Background Trampoline (DTrace, DynInst) Just-in-Time Compilation (PIN, Valgrind) Source-code based techniques (Log4j, Log4c) Compiler driven techniques(gprof) Blackbox but slow } Developer/Compiler driven but fast H/W Interrupt Trap Mechanism Trampoline mechanism Overhead because of extra-jumps, and simulating overwritten instructions High overhead because of h/w trap mechanism Monitored Application Monitored Application High overhead because of context switch to kernel space *High Overhead because of complex Safety checks Kernel Space Logs User Space Logs D-Trace/SystemTap DynInst

The Core Idea: Hybrid Instrumentation Traditional Instrumentation uses either purely compiler based techniques or purely binary based techniques Hybrid Instrumentation uses both compiler and binary instrumentation to gain a significant performance advantage Development Phase Production Phase Source Code Files iProbe GCC Compiler Flags + iProbe packaged software Run-time Monitoring iProbe packaged software

State-Diagram ColdPatch Phase HotPatch Phase Compile source code files with GNU compiler flag “-finstrument-functions" Use cold-patch script to replace all instrumentation calls with NOP instructions Create a meta-data file with location of each NOP placeholder iProbe presents the user the set of functions they can select to instrument at run-time using the probe-list The HotPatcher then replaces the NOP instructions with a call to the instrumentation function Extremely low overhead since there is no overwriting of instructions <Basic Block Begin> <func_foo>: push %EBP call <foo_begin> pop inc …. … call <foo_end> <Basic Block End> <Basic Block Begin> <func_foo>: push %EBP NOP <90> pop inc …. … <Basic Block End> <Basic Block Begin> <func_foo>: push %EBP call<begin_instr> pop inc …. call<end_instr> <Basic Block End> <Basic Block Begin> <func_foo>: push %EBP pop inc …. … <Basic Block End> NOP replaced in run-time with calls to instrumentation functions Native binary Compiled with instrumentation flag Replaced by NOP OpCode

Evaluation : ColdPatch An iProbe enabled cold-patched application has NOP instructions added as placeholders to the binary Evaluation on SPEC CPU 2006 benchmarks - negligible on most applications (<1%)

Evaluation: HotPatch Evaluation of application performance in comparison to existing tools was found to have an order of magnitude better performance and scaled significantly better 10-2 10-1 10 102 103 104 10-3 102 103 104 105 106 107 108 10

Conclusion iProbe can provide for extremely light-weight instrumentation in the user-space, with a stable and robust design, which avoids most complications that other tools deal with. No developer effort needed (new instrumentation etc.), uses pre-existing compiler flags We have an advanced version which supports binary rewriting or user-driven macros to generate place-holders, and can be used instead of compiler flags for customized instrumentation points. iProbe-enabled applications can provide secure instrumentation especially when used with code-obfuscation techniques iProbe can be used as a monitoring framework to develop further more intelligent instrumentation and monitoring applications

Please visit us !!! nipun@nec-labs.com Nipun Arora NEC Laboratories America Princeton, NJ http://www.nipunarora.net/iprobe-demo.html nipun@nec-labs.com