1. Systemtap Frank Ch. Eigler software developer

2. Systemtap review Script language is safely compiled into C kernel module Module uses kprobes and other event sources Probes may just trace, or analyze, aggregate, act Includes reusable probe libraries User-space probing coming soon

3. Tracing/probing degrees of freedom Probe points – where to probe Probe handling – what to do there Cost of dormant probe Active probe dispatch cost

4. Dynamic probing Probe point: any location, may be identified by source-level “co-ordinates” Probe handler: a bunch of safe code to extract and process context values; online processing possible Dormant probes are free Dispatch cost high (trap handling)

5. Static tracing Probe point: wherever hooks are compiled in Fixed probe handler: collect fixed pool of context data, dump it to buffer; off-line post-processing Low cost dormant probes (how low?) Dispatch cost low (how low?)

6. Static instrumentation markers Decoupling probe point and handler To create: place it, name it, parametrize it. That's it: STAP_MARK_NN(context_switch,prev->pid,next->pid); To use from systemtap: probe kernel.mark(“context_switch”) { print($arg1) } #define STAP_MARK_NN(n,a1,a2) do { \ static void (*__stap_mark_##n##_NN)(int64_t,int64_t); \ if (unlikely (__stap_mark_##n##_NN)) \ (void) (__stap_mark_##n##_NN((a1),(a2))); \ } while (0)

7. How systemtap uses markers: Compile probe handler the usual way Find static variable's location from symbol table Demangle event name, parameter types/arity for custom glue function During probe session startup, cmpxchg function pointer Thence kernel calls probe handler with params During shutdown, cmpxchg it back to NULL

8. What's next? Convert J Axboe's blocktrace probe points to static markers Convert associated tracing probe handlers to systemtap script Write non-systemtap alternative backend for tracing If successful, lobby others to convert their tracer widgets similarly