November 25, KFT & Tracing Collaboration Tim Bird Sony Electronics

November 25, KFT Quickie Overview KFT = Kernel Function Trace –Uses a compiler switch (-finstrument_functions) to instrument EVERY function in kernel –Low-intrusion patch to kernel source –Optimized for finding latency problems Filters for long-duration function calls Maintains a function stack from which short duration functions are constantly popped Historically used to find places to improve bootup time

November 25, KFT Example kftstatic.conf Output: (cat /proc/kft_data | addr2sym -m System.map) trigger start entry start_kernel trigger end entry to_userspace filter mintime 500 Entry Delta PID Function Called At start_kernel L6+0x setup_arch start_kernel+0x setup_memory setup_arch+0x2a register_bootmem_low_pages setup_memory+0x8f...

November 25, More Information Has triggers, filters Can trace very early - from first kernel instruction Uses procfs for control and data (everything ASCII) Post-processing tool shows: –functions sorted by duration, call count, local time, etc. –call tree, including interrupts Google: “KernelFunctionTrace” (all one word)

November 25, Tracing Collaboration

November 25, Why Collaborate Want to avoid duplicating effort Want to avoid conflicts and re-inforce efforts –E.g. Kernel will support AT MOST one set of static tracepoints Want to produce awesome tools with great capabilities and usefulness It’s open source, dang-it! How to proceed? –Groups have talked before - was there any outcome?

November 25, Tracing Terminology Phases: –trace definition/configuration –trace generation activation, control, capture, termination –post-processing formatting the data and visualization Kinds of data gathering –event-based –sample-based

November 25, Tracing Terminology tracepoint/probe point - a location in the traced software, which, when encountered, “emits” an event trace/probe handler - the code to execute when an event occurs or a sample is taken trace definition - the set of tracepoints, handlers and constraints which determine what events are collected and how they are processed post-processing - manipulation of the trace data after the trace is collected aggregation - updating statistics or other analytical information, based on trace events or samples filters - criteria used to limit the events that are captured or processed triggers - criteria used to start and stop tracing automatically visualization tool - tool to present trace data graphically to make it easier to analyze Tracepoints and trace handlers can be defined either in-source (as compiled-in, or static tracepoints) or externally (inserted as dynamic tracepoints)

November 25, Tracer Comparison Chart Flexibility KFT LTTng SystemTap Specialized tracers

November 25, Tracer Comparison Table KFTLTTngSystemTAP Target user embedded developer system admin Instrumentation every function, by compiler static definitions in source programmable, using external definition Control interface echo and cat, using /proc lttctl and lttd, using netlink stap, using insmod and kprobe Overhead mediumlowhigh Trace format ascii, fixedbinary, with xml schema programmable, printf and ascii-art Post-processing kd, text- oriented lttv, graphicalsystemtapgui??

November 25, Tracer Commonalities Ummm…

November 25, Ideas Integrate KFT into LTTng Have LTTng support externally-defined, but compiled-in, tracepoints Stabilize trace buffer mechanism in kernel Build best-of-systemtap functions module, which can be compiled-in Standardize trace file format What else???