GILK: A Dynamic Instrumentation Tool for the Linux Kernel David J. Pearce, Paul H.J. Kelly, Tony Field and Uli Harder

Introduction  Do programmers really know what is going on? »Need deeper understanding of program behaviour »Debug/model/validate performance issues »Program instrumentation can provide this  GILK is about doing this for the Linux Kernel »Adds instrumentation on the fly »No need to restart »No need to recompile »No need to interrupt the web server!

Introduction  Do programmers really know what is going on? »Need deeper understanding of program behaviour »Debug/model/validate performance issues »Program instrumentation can provide this  GILK is about doing this for the Linux Kernel »Adds instrumentation on the fly »No need to restart »No need to recompile »No need to interrupt the web server!

GILK – An Introduction

Motivating Example  Research into Self Imperial »Using TCPDUMP to obtain network timing information »Produced fair amount of data »But occasional zero or negative inter-arrival time was spotted »Are results so far wasted? »Is TCPDUMP useable or not?  GILK deployed for comparison with TCPDUMP »GILK provides high resolution packet arrival times »Previous results valid – statistical properties unchanged »TCPDUMP does generate erroneous results  What We Did »Next slide

Motivating Example  Research into Self Imperial »Using TCPDUMP to obtain network timing information »Produced fair amount of data »But occasional zero or negative inter-arrival time was spotted »Are results so far wasted? »Is TCPDUMP useable or not?  GILK deployed for comparison with TCPDUMP »GILK provides high resolution packet arrival times »Previous results valid – statistical properties unchanged »TCPDUMP does generate erroneous results  What We Did »Next slide

Motivating Example  Research into Self Imperial »Using TCPDUMP to obtain network timing information »Produced fair amount of data »But occasional zero or negative inter-arrival time was spotted »Are results so far wasted? »Is TCPDUMP useable or not?  GILK deployed for comparison with TCPDUMP »GILK provides high resolution packet arrival times »Previous results valid – statistical properties unchanged »TCPDUMP does generate erroneous results  What We Did »Next slide

speedo_rx sock_recvmsg What We Did BSD Socket Layer Ethernet Driver { { Linux Network Stack

speedo_rx sock_recvmsg What We Did BSD Socket Layer Ethernet Driver { { Linux Network Stack

Where was the Success?  We’re not Kernel Programmers »Don’t know beforehand what was needed »Time wasted trying different instrumentation  GILK is fast »Turn around time only a few minutes »Allows us to try many different instrumentations »Fits exploratory nature of debugging

Where was the Success?  We’re not Kernel Programmers »Don’t know beforehand what was needed »Time wasted trying different instrumentation  GILK is fast »Turn around time only a few minutes »Allows us to try many different instrumentations »Fits exploratory nature of debugging

Where was the Success?  We’re not Kernel Programmers »Don’t know beforehand what was needed »Time wasted trying different instrumentation  GILK is fast »Turn around time only a few minutes »Allows us to try many different instrumentations »Fits exploratory nature of debugging

How it Works  GILK uses code splicing technology »Allows instrumentation of active kernel »Instrumentation performed at machine code level »Works on unmodified kernel image »No patching, restarting, recompiling etc  Instruments written in ‘C’ »Implemented as Kernel Modules »Access to full language features »Access to kernel functionality  System Requirements »Only Intel x86 Architecture supported »Kernel versions 2.0 – 2.2 supported -Extension to 2.4 is trivial

How it Works  GILK uses code splicing technology »Allows instrumentation of active kernel »Instrumentation performed at machine code level »Works on unmodified kernel image »No patching, restarting, recompiling etc  Instruments written in ‘C’ »Implemented as Kernel Modules »Access to full language features »Access to kernel functionality  System Requirements »Only Intel x86 Architecture supported »Kernel versions 2.0 – 2.2 supported -Extension to 2.4 is trivial

How it Works  GILK uses code splicing technology »Allows instrumentation of active kernel »Instrumentation performed at machine code level »Works on unmodified kernel image »No patching, restarting, recompiling etc  Instruments written in ‘C’ »Implemented as Kernel Modules »Access to full language features »Access to kernel functionality  System Requirements »Only Intel x86 Architecture supported »Kernel versions 2.0 – 2.2 supported -Extension to 2.4 is trivial

Before After instrumentation splice Relocated sequence Kernel Function { Kernel Function { Code Patch { Instruction Boundaries unused

Conclusion  GILK quickly instruments Linux Kernel »Runtime code splicing useful for instrumentation »Much left unsaid, however  Future work … »Simplifying instrumentation process -GUI, Instrumentation Strategies, Visualization »Programmatic instrumentation is possible -Paradyn (WISC) attempts this »Non-instrumentation based options -Modifying Program Behaviour »Non-Intel platforms? -KernInst/Paradyn (WISC) on UltraSparc/Solaris »User-Space programs? -User-Space Threading is an issue … »Our Research group is now doing this for Java …

Conclusion  GILK quickly instruments Linux Kernel »Runtime code splicing useful for instrumentation »Much left unsaid, however  Future work … »Simplifying instrumentation process -GUI, Instrumentation Strategies, Visualization »Programmatic instrumentation is possible -Paradyn (WISC) attempts this »Non-instrumentation based options -Modifying Program Behaviour »Non-Intel platforms? -KernInst/Paradyn (WISC) on UltraSparc/Solaris »User-Space programs? -User-Space Threading is an issue … »Our Research group is now doing this for Java …

Conclusion  GILK quickly instruments Linux Kernel »Runtime code splicing useful for instrumentation »Much left unsaid, however  Future work … »Simplifying instrumentation process -GUI, Instrumentation Strategies, Visualization »Programmatic instrumentation is possible -Paradyn (WISC) attempts this »Non-instrumentation based options -Modifying Program Behaviour »Non-Intel platforms? -KernInst/Paradyn (WISC) on UltraSparc/Solaris »User-Space programs? -User-Space Threading is an issue … »Our Research group is now doing this for Java …