Kernel Code Coverage Nilofer Motiwala Computer Sciences Department University of Wisconsin 1210 W. Dayton Street Madison, WI 53706-1685 USA

Motivation Code coverage answers the basic question: How much of my code did I test? Workloads exercise only a subset of the programs functionality Code coverage for an OS kernel Critical due to complex interactions in kernel More difficult to extract information from kernel (Kerninst makes it easy)

Motivation Kernel code coverage is important Workload does not guarantee the same execution pattern over multiple runs Networking behavior Filesystem behavior Error checking code is rarely executed Coverage at basic block level Make sure we are precise in our analysis

Basic Strategy Dynamically instrument all basic blocks Increment a counter at beginning of basic block Periodically sample counter values De-instrument if block has been reached Reduces execution overhead Similar to DynInst code coverage tool for user-level applications, but with new challenges…

Basic Strategy Dynamically instrument all basic blocks Increment a counter at beginning of basic block Periodically sample counter values De-instrument if block has been reached Reduces execution overhead Similar to DynInst code coverage tool for user-level applications, but with new challenges… First application of mass instrumentation

Previous Applications Kernel performance tool (kperfmon) Incremental approach Mass instrumentation not encouraged As a result, no significant issues regarding allocation of space for large number of instrumentation code patches

Previous Applications Kernel performance tool (kperfmon) Incremental approach Mass instrumentation not encouraged As a result, no significant issues regarding allocation of space for large number of instrumentation code patches However, a code coverage tool calls for mass instrumentation, and a perf tool can benefit too

The Problem Challenge is mass instrumentation in kernel Sparc-Solaris7 kernel is large (~3MB) 96 modules 13,000 functions 188,000 basic blocks Jump to instrumentation code using a single instruction Jump range limitation of +/- 8MB Finding space within 8MB for large quantity of instrumentation code is a problem

Instrumentation method Patch Area foo() counter++ (bb1 entry) displaced code counter++ (bb2 entry) displaced code Patch area reached via a branch instruction Replace one instruction w/ branch to patch area However, +/-8MB range limitation of branch Need 188,000 patch areas within range

Instrumentation method Springboards Patch Area foo() jmp patch_addr counter++ (bb1 entry) displaced code jmp patch_addr (bb2 entry) displaced code Mitigate problem by adding indirection Springboard is smaller than a patch area Now, only Springboards have to be in 8MB range Still, need 188,000 Springboards nearby

Patch Area v/s Springboard Kerninst is set up to allocate patch areas anywhere in kernel virtual address space Would like to allocate them close to the code we are instrumenting However, for purposes of code coverage, Springboards are given preference Tradeoff between efficiency and instrumenting more blocks

How much space is needed? Springboard size dictated by patch area address Jump to 32-bit address  three instructions Need ~2 MB in 8 MB range Jump to 64-bit address  six instructions Need ~4 MB in 8 MB range Fortunately, we can place all patches in 32 bit address space

How much space is needed? Springboard size dictated by patch area address Jump to 32-bit address  three instructions Need ~2 MB in 8 MB range Jump to 64-bit address  six instructions Need ~4 MB in 8 MB range Fortunately, we can place all patches in 32 bit address space Still need 2 MB in 8 MB range Trade off space v/s execution effeciency Patch heap space for incrementing counter Seven instructions (confirm!!) Jump to 64-bit address  six instructions  Need ~2.4 MB in 8 MB range Want to place all patch areas in 32-bit address space 7 instructions to inc. counter  need ~2.8 MB in 32-bit address space

Kernel Address Space(64 bit) 0x000.0000.0000 Invalid Kernel Address Space(64 bit) 0x000.1000.0000 Kernel Nucleus 0x000.1040.0000 0x000.1080.0000 0x000.7802.0000 32 bit Kernel heap 0x000.7c00.0000 0x300.0000.0000 64 bit kernel heap 0x302.0000.0000

Kernel Address Space(64 bit) 0x000.0000.0000 Invalid Kernel Address Space(64 bit) 0x000.1000.0000 Kernel Text Segment 0x000.1040.0000 Kernel Data Segment Memory Mgmt Structures 0x000.1080.0000 0x000.7802.0000 32 bit Kernel heap 0x000.7c00.0000 File System Cache 0x300.0000.0000 64 bit kernel heap 0x302.0000.0000

Kernel Address Space(64 bit) 0x000.0000.0000 Invalid Kernel Address Space(64 bit) 0x000.1000.0000 Kernel Text Segment 0x000.1040.0000 Kernel Data Segment Memory Mgmt Structures 8 MB Range 0x000.1080.0000 0x000.7802.0000 32 bit Kernel heap 0x000.7c00.0000 0x300.0000.0000 64 bit kernel heap 0x302.0000.0000

Kernel Address Space(64 bit) 0x000.0000.0000 Invalid Kernel Address Space(64 bit) 0x000.1000.0000 Trap table Kernel Text Segment 0x000.1040.0000 Kernel Data Segment Memory Mgmt Structures 0x000.1080.0000 0x000.7802.0000 32 bit Kernel heap 0x000.7c00.0000 0x300.0000.0000 64 bit kernel heap 0x302.0000.0000

Finding Springboard Space Use all available space in nucleus Allocate free memory directly Overwrite routines that will not be invoked Load dummy modules in kernel 256 MB invalid region in kernel space Unmapped but could it be mapped and used?? 6 MB of free space will be within our branch range

Kernel Address Space(64 bit) 0x000.0000.0000 Invalid Kernel Address Space(64 bit) 0x000.1000.0000 Kernel Text Segment 0x000.1040.0000 Kernel Data Segment Memory Mgmt Structures 8 MB Range 0x000.1080.0000 0x000.7802.0000 32 bit Kernel heap 0x000.7c00.0000 0x300.0000.0000 64 bit kernel heap 0x302.0000.0000

Current Allocations Inside nucleus, able to obtain 260 KB via nucleus malloc (mach_mod_alloc) 64 KB via dummy module and 8 KB by overwriting routines 28,000 Springboards for basic blocks inside nucleus Can successfully allocate all patch heap space in 32 bit kernel heap Able to instrument afs (18,000 basic blocks) 210 KB Springboard space in 32 bit kernel heap Not yet able to instrument the entire kernel

Code Coverage kerninstd Information Flow Kernel Space Data Heap Instrumentation request Sampling request kerninstd ioctl() Patch Heap Data Heap (counters) /dev/kerninst Kernel Space

Code Coverage kerninstd Information Flow Kernel Space Data Heap Instrumentation request Individual counter values Sampling request kerninstd ioctl() Patch Heap Data Heap (counters) /dev/kerninst Kernel Space

Experimental Results Instrumented afs Workload Measurements Build for a large program (code coverage) Find through multiple layers of directories Read all files in /usr/include Measurements Number of functions covered Number of basic blocks covered

Experimental Results for AFS Basic blocks: 18163 Functions: 922

Current Challenges Memory allocation issues Parsing issues More springboard space needed to instrument all basic blocks Parsing issues 206 of 13,000 functions, currently cannot be parsed Unanalyzable jump instructions

Future Work Reduction in number of instrumentation points needed Dominator tree analysis Dyninst Code Coverage tool achieves a reduction of 33%-49% De-instrumentation policy Based on time interval Based on de-instrumentation request queue size Sampling optimization Only samples reflecting a change in value should be sent

Conclusion Kernel Code Coverage Mass instrumentation Gives the ability to know at a fine level, what kernel code is exercised by a given workload Dynamically de-instrument Ongoing work