FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 FUDConBrussels THIS IS SYSTEMTAP Dynamic instrumentation for the Linux kernel Bryn Reeves Software maintenance engineer Fosdem 2007

FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 Overview Project goals and motivations Architecture & requirements Running scripts SystemTap scripting Examples Future work Links & docs

FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 Project Founded in 2005 Project members include Red Hat, IBM, Intel and Hitachi Free software infrastructure to simplify the gathering of information about the running Linux kernel SystemTap is possible because Linux is open source Complements related tools including Oprofile, LTT, Frysk, (k)gdb, crash Project pages hosted at

FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 Motivation Some projects require deep insight into OS internals Current performance tools are useful, but inadequate for many tasks Application centric tools narrow in scope Tools with system-wide scope present a static view of system behavior but do not permit further probing Many problems are not readily exposed by traditional tools A more flexible solution is needed

FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 SystemTap architecture ● Infrastructure to access low-level kernel probes ● A scripting language for writing instrumentation ● User space translator/driver and daemon (stap & stpd) ● Kernel space runtime infrastructure ● System supplied tapset libraries for commonly used probes and functions

FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 SystemTap Infrastructure Kernel probe.ko kprobes jprobes kretprobes relay fs runtime proc fs Userspace stpd stap.k o tapsets runtime source.stp output timers, profili ng

FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 SystemTap requirements Supported from Fedora Core 5 onward RPM dependencies GCC toolchain kernel-devel kernel-debuginfo Now split into common, xen, kdump, and PAE sub-packages FC5 kernel-debuginfo was around 1G on i686! Default yum configuration now knows about debuginfo

FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 How do I use it? SystemTap provides a scripting interface probe begin { print("Hello World!\n") ; exit() } These scripts name events and give them handlers An event and its handler is a probe When a specified event occurs, the kernel executes the handler and then goes back to whatever it was doing Similar idea and implementation to a breakpoint Low overhead (30,000 probes/sec with 1% overhead)

FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 Running probes stap [ OPTIONS ] [-e SCRIPT|FILENAME] [ ARGUMENTS ] SystemTap uses sudo for privileged actions Logs to terminal or file Binary dumps for post-mortem analysis (LKET)

FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 Probe execution

FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 Show me some code, dammit! global reads probe begin { print(“probe begins\n”) } probe syscall.read { reads[execname()] <<< count } probe end { foreach (progname in reads) { printf(“%s reads: %d, “, printf(“total bytes: %d, } Global variables Script startup and shutdown events Built-in functions Associative arrays Aggregation operator and stats functions Pre-defined tapsets

FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 Variables, startup and shutdown global reads probe begin { print(“probe begins\n”) } probe syscall.read { reads[execname()] <<< count } probe end { foreach (progname in reads) { printf(“%s reads: %d, “, printf(“total bytes: %d, } Global variables Script startup and shutdown events Built-in functions Associative arrays Aggregation operator and stats functions Pre-defined tapsets

FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 Functions, arrays, stats and tapsets global reads probe begin { print(“probe begins\n”) } probe syscall.read { reads[execname()] <<< count } probe end { foreach (progname in reads) { printf(“%s reads: %d, “, printf(“total bytes: %d, } Global variables Script startup and shutdown events Built-in functions Associative arrays Aggregation operator and stats functions Pre-defined tapsets

FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 Some examples Trace and analyze open(2) Who's messing with my file?... and what about my device? Implementing strace(1) as a SystemTap script Gathering memory and performance statistics Discovering process properties Abusing SystemTap for fun and profit

FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 Future Work Introducing utrace – new infrastructure for user-space debugging and tracing Replaces ptrace layer in the kernel Kernel support for perfmon2 (performance counter access) Improve SystemTap binary portability SystemTap user interfaces More tapsets

FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 Want to know more? Man pages stap(1), stapex(5), stapprobes(5), stapfuncs(5) Website: Wiki: Mailing list: IRC channel: #systemtap on irc.freenode.net

FUDConBrussels Fedora ProjectFedora Project – 24 February 2007 The team