1. GILK: A Dynamic Instrumentation Tool for the Linux Kernel David J. Pearce, Paul H.J. Kelly, Tony Field and Uli Harder d.pearce@doc.ic.ac.uk www.doc.ic.ac.uk/~djp1/gilk.html

2. Introduction  Do programmers really know what is going on? »Need deeper understanding of program behaviour »Debug/model/validate performance issues »Program instrumentation can provide this  GILK is about doing this for the Linux Kernel »Adds instrumentation on the fly »No need to restart »No need to recompile »No need to interrupt the web server!

3. Introduction  Do programmers really know what is going on? »Need deeper understanding of program behaviour »Debug/model/validate performance issues »Program instrumentation can provide this  GILK is about doing this for the Linux Kernel »Adds instrumentation on the fly »No need to restart »No need to recompile »No need to interrupt the web server!

4. GILK – An Introduction

7. Motivating Example  Research into Self Similarity @ Imperial »Using TCPDUMP to obtain network timing information »Produced fair amount of data »But occasional zero or negative inter-arrival time was spotted »Are results so far wasted? »Is TCPDUMP useable or not?  GILK deployed for comparison with TCPDUMP »GILK provides high resolution packet arrival times »Previous results valid – statistical properties unchanged »TCPDUMP does generate erroneous results  What We Did »Next slide

8. Motivating Example  Research into Self Similarity @ Imperial »Using TCPDUMP to obtain network timing information »Produced fair amount of data »But occasional zero or negative inter-arrival time was spotted »Are results so far wasted? »Is TCPDUMP useable or not?  GILK deployed for comparison with TCPDUMP »GILK provides high resolution packet arrival times »Previous results valid – statistical properties unchanged »TCPDUMP does generate erroneous results  What We Did »Next slide

9. Motivating Example  Research into Self Similarity @ Imperial »Using TCPDUMP to obtain network timing information »Produced fair amount of data »But occasional zero or negative inter-arrival time was spotted »Are results so far wasted? »Is TCPDUMP useable or not?  GILK deployed for comparison with TCPDUMP »GILK provides high resolution packet arrival times »Previous results valid – statistical properties unchanged »TCPDUMP does generate erroneous results  What We Did »Next slide

10. speedo_rx sock_recvmsg What We Did BSD Socket Layer Ethernet Driver { { Linux Network Stack

11. speedo_rx sock_recvmsg What We Did BSD Socket Layer Ethernet Driver { { Linux Network Stack

12. Where was the Success?  We’re not Kernel Programmers »Don’t know beforehand what was needed »Time wasted trying different instrumentation  GILK is fast »Turn around time only a few minutes »Allows us to try many different instrumentations »Fits exploratory nature of debugging

13. Where was the Success?  We’re not Kernel Programmers »Don’t know beforehand what was needed »Time wasted trying different instrumentation  GILK is fast »Turn around time only a few minutes »Allows us to try many different instrumentations »Fits exploratory nature of debugging

14. Where was the Success?  We’re not Kernel Programmers »Don’t know beforehand what was needed »Time wasted trying different instrumentation  GILK is fast »Turn around time only a few minutes »Allows us to try many different instrumentations »Fits exploratory nature of debugging

15. How it Works  GILK uses code splicing technology »Allows instrumentation of active kernel »Instrumentation performed at machine code level »Works on unmodified kernel image »No patching, restarting, recompiling etc  Instruments written in ‘C’ »Implemented as Kernel Modules »Access to full language features »Access to kernel functionality  System Requirements »Only Intel x86 Architecture supported »Kernel versions 2.0 – 2.2 supported -Extension to 2.4 is trivial

16. How it Works  GILK uses code splicing technology »Allows instrumentation of active kernel »Instrumentation performed at machine code level »Works on unmodified kernel image »No patching, restarting, recompiling etc  Instruments written in ‘C’ »Implemented as Kernel Modules »Access to full language features »Access to kernel functionality  System Requirements »Only Intel x86 Architecture supported »Kernel versions 2.0 – 2.2 supported -Extension to 2.4 is trivial

17. How it Works  GILK uses code splicing technology »Allows instrumentation of active kernel »Instrumentation performed at machine code level »Works on unmodified kernel image »No patching, restarting, recompiling etc  Instruments written in ‘C’ »Implemented as Kernel Modules »Access to full language features »Access to kernel functionality  System Requirements »Only Intel x86 Architecture supported »Kernel versions 2.0 – 2.2 supported -Extension to 2.4 is trivial

18. Before After instrumentation splice Relocated sequence Kernel Function { Kernel Function { Code Patch { Instruction Boundaries unused

19. Conclusion  GILK quickly instruments Linux Kernel »Runtime code splicing useful for instrumentation »Much left unsaid, however  Future work … »Simplifying instrumentation process -GUI, Instrumentation Strategies, Visualization »Programmatic instrumentation is possible -Paradyn (WISC) attempts this »Non-instrumentation based options -Modifying Program Behaviour »Non-Intel platforms? -KernInst/Paradyn (WISC) on UltraSparc/Solaris »User-Space programs? -User-Space Threading is an issue … »Our Research group is now doing this for Java …

20. Conclusion  GILK quickly instruments Linux Kernel »Runtime code splicing useful for instrumentation »Much left unsaid, however  Future work … »Simplifying instrumentation process -GUI, Instrumentation Strategies, Visualization »Programmatic instrumentation is possible -Paradyn (WISC) attempts this »Non-instrumentation based options -Modifying Program Behaviour »Non-Intel platforms? -KernInst/Paradyn (WISC) on UltraSparc/Solaris »User-Space programs? -User-Space Threading is an issue … »Our Research group is now doing this for Java …

21. Conclusion  GILK quickly instruments Linux Kernel »Runtime code splicing useful for instrumentation »Much left unsaid, however  Future work … »Simplifying instrumentation process -GUI, Instrumentation Strategies, Visualization »Programmatic instrumentation is possible -Paradyn (WISC) attempts this »Non-instrumentation based options -Modifying Program Behaviour »Non-Intel platforms? -KernInst/Paradyn (WISC) on UltraSparc/Solaris »User-Space programs? -User-Space Threading is an issue … »Our Research group is now doing this for Java … d.pearce@doc.ic.ac.uk www.doc.ic.ac.uk/~djp1/gilk.html