Self-Propelled Instrumentation Alex Mirgorodskiy Barton Miller Computer Sciences Department University of Wisconsin-Madison 1210 West Dayton Street Madison, WI USA

Self-propelled Instrumentation 2 Motivation Finding the causes of bugs and performance problems in production systems is hard Systems are complex and non-transparent –Multiple components, different vendors Anomalies are common –Intermittent –Environment-specific Users have little debugging expertise

Self-propelled Instrumentation 3 Vision New family of run-time tools: Autonomous Detailed Low-overhead Host A Host B Process P Process Q Agent network

Self-propelled Instrumentation 4 Applications Diagnostics of E-commerce systems –Find bugs by following the paths of individual requests through the system of Web and Database servers Diagnostics of Cluster and Grid systems –Work without process control facilities: Red Storm, BG/L –Start up without batch scheduler support Diagnostics of Interactive and Real-time systems –Find the causes of latency problems in multimedia –Trace a word said by one user through a phone network Enforcement of Security –Sandboxing and controlling execution

Self-propelled Instrumentation 5 Self-Propelled Instrumentation: Overview Insert an initial probe into a process The probe starts propagating –Inserts more probes ahead of the control flow The probes may monitor or control execution –Collect run-time data –Look for anomalies in the data

Self-propelled Instrumentation 6 Instrumentation without an external tool –Ultra-fast code insertion –Same mechanism for applications and the kernel Propagation –Instrument only what is necessary –Make decisions autonomously and on the fly Self-Propelled Instrumentation: Key Ideas

a.out bar 8430: 8431: 8433: 8444: 8446: 8449: 844b: 844c: push mov... int mov xor pop ret foo %ebp %esp,%ebp 0x80 %ebp,%esp %eax,%eax %ebp call jmp Patch1 instrument(foo) foo 0x8405 6cf5: 6d20: 6d27: 6d49: push... call... iret sys_call: %eax *%eax call jmp instrument(%eax) *%eax 0x6d27 Patch3 instrumenter.so /dev/instrumenter ioctl int jmp (INSTRUMENT_SYS) 0x80 0x8446 Patch2 patch jmp %ebp %esp,%ebp foo %ebp,%esp %ebp push mov... call mov pop ret 83f0: 83f1: 83f3: 8400: 8405: 8413: 8414: OS Kernel patch jmp Inject Activate Propagate

Self-propelled Instrumentation 8 Prototype Tool: spTracer Collects traces of function calls and returns Activates/deactivates at run time –Trace between two user-specified events Can trace unmodified applications Can trace the OS kernel code –System calls – follow control flow into the kernel –Hardware interrupts

Self-propelled Instrumentation 9 Propagation and Trace Collection call instrument(B) call traceEntry(B) call B call traceExit(B) jmp back... call C()... return... call B()... return jmp ApatchABB Overwrite call sites in the code –Fast: done with local memory operations Invoke tracing routines before and after each call

Self-propelled Instrumentation 10 Visualization: Nupshot Time-diagram of all function calls Nested rectangles = nested function calls

Self-propelled Instrumentation 11 Evaluation: DVD playback study Choppy DVD playback under Linux Same hardware works fine under Windows

Self-propelled Instrumentation 12 Evaluation: DVD playback study Activation –Start on the alarm timer to skip lengthy initialization/warm-up in MPlayer Propagation –Instrument both MPlayer and the kernel –Collect function-level traces Deactivation –Stop tracing after 250 msec (~6 frames)

Evaluation: DVD playback study Freeze is neither in a loop nor in a system call Also happened in other places in the code –Could not discover it with cumulative metrics (gprof) Preemption? User-level trace of MPlayer

Evaluation: DVD playback study User-level trace of MPlayer Kernel interrupts while MPlayer was running

Self-propelled Instrumentation 15 Evaluation: DVD playback study The time is spent handling interrupts from the DVD drive The root cause – ide_input_data that reads data from the drive 4 bytes at a time Solution – enable DMA ! Kernel interrupts under magnification

Self-propelled Instrumentation 16 Ongoing Work Cross process and host boundaries –Jump from one process to another –Reconstruct system-wide flows –Analyze collected data for anomalies Diagnostics of Cluster and Grid environments –Debugging the SCore batch scheduler –Joint work with Naoya Maruyama and Satoshi Matsuoka, Tokyo Institute of Technology Diagnostics of E-commerce environments

Self-propelled Instrumentation 17 Conclusion Analysis by following the control flow –Autonomous, detailed, low-overhead Analysis across the kernel boundary –Foundation for full-system analysis For further information: – –Come see the demo: Friday, CS 7355 –Paper: Mirgorodskiy and Miller, Autonomous analysis of interactive systems with self-propelled instrumentation,