Dynamic Tracing using DTrace tm By Sameer Niphadkar

Overview Introduction to Tracing Current Alternatives What is DTrace ? Instrumentation Significance D Programming language DTrace Basic Architecture Case Studies Further Information References

Introduction to Tracing Trace = find by careful investigation How's software tracing useful ? Tracks performance problems across many layers of software for further debugging by tracing system_calls( ) for one or more processes. Determines actions to be taken by investigation of bottlenecks. Quick iterative analysis saves precious time on production systems.

Introduction to Tracing (cont) Highlights and tunes software systems for better performance and management No recompilation (Dtrace) No performance hit (Dtrace) Useful for I/O, CPU, VM stats, user functions and other personalized views (DTrace)

Current Alternatives Process statistics using – truss, trace (s, p, l), apptrace, debuggers – mdb, adb etc Limitations Non-evasive methods : proc file systems Concentrate only on processes system calls and no system wide introspection : threads, functions, user applications No dynamic instrumentation of various code paths and data records into buffers.

Current Alternatives (cont) Other tracing tools Linux Trace Toolkit – Limited no. of instrumentation points, No arbitrary action, Lacks inclusion of higher level language to describe actions DProbes – For OS/2 ported to Linux. Similar to Dtrace but lossy technique for various CPUs. Lacks data aggression and predicate support. Invalid safety support K42 – Research kernel having its own static instrumentation. Limitations similar to LTT Kerninst – Dynamic instrumentation framework. Zero probe effect like Dtrace. Highly aggressive instrumentation can introduce fatal errors. No support for arbitrary actions.

What is Dtrace ? Dtrace is a comprehensive dynamic tracing facility that is built into the Sun Solaris 10 and the Open Solaris kernel Features of Dtrace Normal custom tracing mechanisms + your own personal programs and provides answers to all your queries : D Programming language Binds a request to a probe after firing for specific actions like recording a stack trace, time stamp or arguments to a function : programming sensor Dtrace + probe = Information of desired location. All instrumentation is dynamic : probes enabled discretely while you are using them i.e. system does not experience any performance degradation. Zero probe effect.

What is Dtrace ? (cont) Dynamic instrumentation both in the user and kernel level : data & control flow across boundary Data Integrity : Reports only non-corrupt and true data values Predicates : Actions to be taken only when user specified conditions are met. Pruning at source. Speculative tracing : Deferring decision to commit/discard at a later time. Less post-processing for aberrant behaviour. Scalable Architecture : Thousands of instrumentation points. Probes subsets to be enabled Virtualized Consumer : Multiple consumers can enable the same probe in different ways. Single consumer – single probe in different ways.

Instrumentation Significance Systemic Scope : Entire system to be instrumentable. Performance Analysis on production systems. Types of instrumentation tracings - Static : Operate by instrumenting binary offline and running it in lieu of the original binary. Limitations – No systemic insights, No integration from disjoint applications, No OS instrumentation For e.g. – ATOM, Purify, Pin

Instrumentation Significance Steps for statically defined tracings for user applications (Dtrace) Name your provider Make probe points by adding macro calls to the code Create yourprovider.d file containing descriptors and arguments of the probe Modify build process to run dtrace all over your object (.o) file Add yourprovider.o to the link library

D Programming language Main features High level language based on C : 90% of syntax from ANSI C Programming structure suitable for tracing + Set of primitives for more advanced data Dtrace compiler in userland converts requests from high-level D Programming language to simple virtual machine. Every probe in DTrace has two names: a unique integer ID and a human-readable string name Easy to understand for any new user as well.

D Programming language (cont) Sample Dtrace operation - # dtrace -n BEGIN -n END dtrace: description 'BEGIN' matched 1 probe dtrace: description 'END' matched 1 probe CPU ID FUNCTION:NAME 0 1 :BEGIN ^C 0 2 :END # Sample D Code - BEGIN { trace("hello, world"); exit(0); } O/P - # dtrace -s hello.d dtrace: script 'hello.d' matched 1 probe CPU ID FUNCTION:NAME 0 1 :BEGIN hello, world #

D Programming language (cont) Providers & Probes : DTrace probes come from set of layered kernel modules called providers. Each of which performs a kind of instrumentation. Tracing action is then bound to these probes. # dtrace -l | wc -l => More than 30,000 probes on 1 system + you can create newer ones if need. Id: provider: module: function: name Compilation & Instrumentation : D language compiler => OS kernel execution => No instrumented code for in-active probes. Therefore no system pause, no trace switch. Intermediate form of D program valid for safety. Can handle runtime errors

Dtrace Architecture

Dtrace Architecture (cont) ECB : Enabling Control Block – Probe is enabled ECB is created. Actions & Predicates : ECB has an optional predicate associated with it. ECB has a list of actions if predicate is satisfied. Buffers : Each Dtrace consumer has a set of in-kernel per-CPU buffers. When ECB action indicated data to be recorded, then its done in consumer's per CPU buffer. If no space in buffer then drop count is incremented and the next ECB is tackled. EPID : Enabled Probe Identifier : Data record layout in per CPU buffer. 1 to many mappings with ECBs.

Dtrace Architecture (cont) DIF : Action-Predicates described in VM instruction set emulated in the kernel during probe fire. RISC instruction set for emulation and on the fly code generation. Execution Mechanism : Command run Probe fire Record in buffer Buffer processed User Level Kernel Level

Case Studies Dtrace Scripts (recepies) : iosnoop, execsnoop, opensnoop, prustat, dtruss...endless Production Systems : SunRay server 10 CPUs, 32 GB memory, 170 SunRay users. Dtrace's sysinfo provider helped in resolving cross-call per CPU problem. Other useful ones :

Further Information References Dtrace BigAdmin – System Admin Portal Dtrace source for download Dtrace manual/ documentation Bryan Cantrill's blog Dtrace tools users.tpg.com.au/adsln4yb/DTrace/smc.html

Summary Conclusion : Dtrace can be considered as one of the best comprehensive tool of instrumentation on a production system.

Thank You !