1. iProbe: A Lightweight User- Space Instrumentation Tool
Nipun Arora, Hui Zhang, Junghwan Rhee, Kenji Yoshihira, Guofei Jiang
Autonomic Management Group
Princeton, NJ

2. Motivation

3. } Background } The current state-of-art monitoring mechanisms use
Trampoline (DTrace, DynInst)
Just-in-Time Compilation (PIN, Valgrind)
Source-code /Compiler-Driven }
Blackbox but slow }
Developer -driven
but fast
Ensuring stability & robustness
when rewriting the binary
Context Switch to Kernel
or Instrumentation Functions
Fixing and simulating
overwritten Instructions

4. } } Background Trampoline (DTrace, DynInst)
Just-in-Time Compilation (PIN, Valgrind)
Source-code based techniques (Log4j, Log4c)
Compiler driven techniques(gprof)
Blackbox but slow }
Developer/Compiler driven but fast
H/W Interrupt Trap Mechanism
Trampoline mechanism
Overhead because of extra-jumps, and simulating overwritten instructions
High overhead because of h/w trap mechanism
Monitored Application
Monitored Application
High overhead because of context switch to kernel space
*High Overhead because of complex
Safety checks
Kernel Space Logs
User Space Logs
D-Trace/SystemTap
DynInst

5. The Core Idea: Hybrid Instrumentation
Traditional Instrumentation uses either purely compiler based techniques or purely binary based techniques
Hybrid Instrumentation uses both compiler and binary instrumentation to gain a significant performance advantage
Development Phase
Production Phase
Source Code Files
iProbe
GCC Compiler Flags +
iProbe packaged
software
Run-time
Monitoring
iProbe packaged
software

6. State-Diagram ColdPatch Phase HotPatch Phase
Compile source code files with GNU compiler flag “-finstrument-functions"
Use cold-patch script to replace all instrumentation calls with NOP instructions
Create a meta-data file with location of each NOP placeholder
iProbe presents the user the set of functions they can select to instrument at run-time using the probe-list
The HotPatcher then replaces the NOP instructions with a call to the instrumentation function
Extremely low overhead since there is no overwriting of instructions
<Basic Block Begin>
<func_foo>:
push %EBP
call <foo_begin>
pop
inc …. …
call <foo_end>
<Basic Block End>
<Basic Block Begin>
<func_foo>:
push %EBP
NOP <90>
pop
inc …. …
<Basic Block End>
<Basic Block Begin>
<func_foo>:
push %EBP
call<begin_instr>
pop
inc ….
call<end_instr>
<Basic Block End>
<Basic Block Begin>
<func_foo>:
push %EBP
pop
inc …. …
<Basic Block End>
NOP replaced in run-time with calls to instrumentation functions
Native binary
Compiled with instrumentation flag
Replaced by NOP OpCode

7. Evaluation : ColdPatch
An iProbe enabled cold-patched application has NOP instructions added as placeholders to the binary
Evaluation on SPEC CPU 2006 benchmarks - negligible on most applications (<1%)

8. Evaluation: HotPatch
Evaluation of application performance in comparison to existing tools was found to have an order of magnitude better performance and scaled significantly better
10-2
10-1 10
102
103
104
10-3
102
103
104
105
106
107
108 10

9. Conclusion
iProbe can provide for extremely light-weight instrumentation in the user-space, with a stable and robust design, which avoids most complications that other tools deal with.
No developer effort needed (new instrumentation etc.), uses pre-existing compiler flags
We have an advanced version which supports binary rewriting or user-driven macros to generate place-holders, and can be used instead of compiler flags for customized instrumentation points.
iProbe-enabled applications can provide secure instrumentation especially when used with code-obfuscation techniques
iProbe can be used as a monitoring framework to develop further more intelligent instrumentation and monitoring applications

10. Please visit us !!! nipun@nec-labs.com Nipun Arora
NEC Laboratories America
Princeton, NJ