1 JIFL: JIT Instrumentation Framework for Linux Marek Olszewski Adam Czajkowski Keir Mierle University of Toronto

2 Instrumenting Operating Systems  Operating systems are growing in complexity Becoming harder to understand  Kernel instrumentation is a well know method of combating this problem Used for: debugging, profiling, monitoring, coverage testing, security auditing...  Dynamic instrumentation is especially useful No recompilation & no reboot Good for debugging systemic problems Feasible in production settings

3 Dynamic Instrumentation  All dynamic instrumentation tools for operating systems are probe based Overwrite existing code with jump/trap instructions  Efficient on fixed length architectures  Slow on variable length architectures Must use trap instruction (and hash table lookup)  JIT-based instrumentation can be more efficient Proven itself for user space (Pin, Valgrind) Probe-based instrumentation is seldom used in user-space

4 Probe-based Instrumentation OS Code Instrumentation Code Instrumentation Code Trap Handler 1.Look up which instrumentation to call 2.Call instrumentation 3.Emulate overwritten instruction

5 JIT Instrumentation  JIT instrumentation rewrites the OS code with calls to instrumentation code Creates a duplicate instrumented copy of the OS Called the code-cache  Since instrumentation is dynamic, it is not feasible to rewrite the entire operating system up-front Instrumentation is performed just-in-time, basic block by basic block, right before each new basic block is executed.  The resulting code is fast No hash table lookup required Though there is some cost in executing in the code- cache

6 JIT Instrumentation OS Code Instrumentation Code Instrumentation Code Duplicate Copy of OS

7 Software Architecture

8 Using JIFL  Use JIFLs API and build system to easily write loadable instrumentation “plugins”  Start by implementing four key functions: plugin_init plugin_exit plugin_start plugin_stop  Load/Unload plugin with Linux’s insmod/rmmod tools  Start/Stop instrumentation with the JIFL starter tool

9 Example Plugin #include "jifl.h" syscall_t syscall; long long count; // Executed for every instrumented basic block static void inc_count(long long *counter_ptr, long size) { *counter_ptr += size; } // Called for every newly discovered basic block static void bb_inst(bb_t *bb, void *arg) { long bb_size = bb_size(bb); bb_insertcall(bb, inc_count, ARG_VOID_PTR, &count, ARG_INT32, bb_size, ARG_END); } // Start instrumentation of clone system call static void plugin_start() { syscall_init(&syscall, __NR_clone); Syscall_add_bb_instrumentation(&syscall, bb_inst, NULL); syscall_start_instrumenting(&syscall); } // Stop instrumentation of clone system call static void plugin_stop() { syscall_stop_instrumenting(&syscall); printk("Clone system call executed %lld instructions\n", count); }

10 Performance Evaluation Apache Web Server Throughput

11 Conclusions  JIT instrumentation viable for operating systems  Fine grained instrumentation now possible for kernel space on variable length architectures Intel’s x86 AMD’s AMD64  Great performance Though it comes with a fixed cost