1. Kernel Code Coverage Nilofer Motiwala Computer Sciences Department
University of Wisconsin
1210 W. Dayton Street
Madison, WI
USA

2. Motivation Code coverage answers the basic question:
How much of my code did I test?
Workloads exercise only a subset of the programs functionality
Code coverage for an OS kernel
Critical due to complex interactions in kernel
More difficult to extract information from kernel (Kerninst makes it easy)

3. Motivation Kernel code coverage is important
Workload does not guarantee the same execution pattern over multiple runs
Networking behavior
Filesystem behavior
Error checking code is rarely executed
Coverage at basic block level
Make sure we are precise in our analysis

4. Basic Strategy Dynamically instrument all basic blocks
Increment a counter at beginning of basic block
Periodically sample counter values
De-instrument if block has been reached
Reduces execution overhead
Similar to DynInst code coverage tool for user-level applications, but with new challenges…

5. Basic Strategy Dynamically instrument all basic blocks
Increment a counter at beginning of basic block
Periodically sample counter values
De-instrument if block has been reached
Reduces execution overhead
Similar to DynInst code coverage tool for user-level applications, but with new challenges…
First application of mass instrumentation

6. Previous Applications
Kernel performance tool (kperfmon)
Incremental approach
Mass instrumentation not encouraged
As a result, no significant issues regarding allocation of space for large number of instrumentation code patches

7. Previous Applications
Kernel performance tool (kperfmon)
Incremental approach
Mass instrumentation not encouraged
As a result, no significant issues regarding allocation of space for large number of instrumentation code patches
However, a code coverage tool calls for mass instrumentation, and a perf tool can benefit too

8. The Problem Challenge is mass instrumentation in kernel
Sparc-Solaris7 kernel is large (~3MB)
96 modules
13,000 functions
188,000 basic blocks
Jump to instrumentation code using a single instruction
Jump range limitation of +/- 8MB
Finding space within 8MB for large quantity of instrumentation code is a problem

9. Instrumentation method
Patch Area
foo()
counter++
(bb1 entry)
displaced code
counter++
(bb2 entry)
displaced code
Patch area reached via a branch instruction
Replace one instruction w/ branch to patch area
However, +/-8MB range limitation of branch
Need 188,000 patch areas within range

10. Instrumentation method
Springboards
Patch Area
foo()
jmp patch_addr
counter++
(bb1 entry)
displaced code
jmp patch_addr
(bb2 entry)
displaced code
Mitigate problem by adding indirection
Springboard is smaller than a patch area
Now, only Springboards have to be in 8MB range
Still, need 188,000 Springboards nearby

11. Patch Area v/s Springboard
Kerninst is set up to allocate patch areas anywhere in kernel virtual address space
Would like to allocate them close to the code we are instrumenting
However, for purposes of code coverage, Springboards are given preference
Tradeoff between efficiency and instrumenting more blocks

12. How much space is needed?
Springboard size dictated by patch area address
Jump to 32-bit address  three instructions
Need ~2 MB in 8 MB range
Jump to 64-bit address  six instructions
Need ~4 MB in 8 MB range
Fortunately, we can place all patches in 32 bit address space

13. How much space is needed?
Springboard size dictated by patch area address
Jump to 32-bit address  three instructions
Need ~2 MB in 8 MB range
Jump to 64-bit address  six instructions
Need ~4 MB in 8 MB range
Fortunately, we can place all patches in 32 bit address space
Still need 2 MB in 8 MB range
Trade off space v/s execution effeciency
Patch heap space for incrementing counter
Seven instructions (confirm!!)
Jump to 64-bit address  six instructions
 Need ~2.4 MB in 8 MB range
Want to place all patch areas in 32-bit address space
7 instructions to inc. counter
 need ~2.8 MB in 32-bit address space

14. Kernel Address Space(64 bit)0x
Invalid
Kernel Address Space(64 bit) 0x
Kernel Nucleus 0x 0x 0x
32 bit Kernel heap
0x000.7c 0x
64 bit kernel heap 0x

15. Kernel Address Space(64 bit)0x
Invalid
Kernel Address Space(64 bit) 0x
Kernel Text Segment 0x
Kernel Data Segment
Memory Mgmt Structures 0x 0x
32 bit Kernel heap
0x000.7c
File System Cache 0x
64 bit kernel heap 0x

16. Kernel Address Space(64 bit)0x
Invalid
Kernel Address Space(64 bit) 0x
Kernel Text Segment 0x
Kernel Data Segment
Memory Mgmt Structures
8 MB Range 0x 0x
32 bit Kernel heap
0x000.7c 0x
64 bit kernel heap 0x

17. Kernel Address Space(64 bit)0x
Invalid
Kernel Address Space(64 bit) 0x
Trap table
Kernel Text Segment 0x
Kernel Data Segment
Memory Mgmt Structures 0x 0x
32 bit Kernel heap
0x000.7c 0x
64 bit kernel heap 0x

18. Finding Springboard Space
Use all available space in nucleus
Allocate free memory directly
Overwrite routines that will not be invoked
Load dummy modules in kernel
256 MB invalid region in kernel space
Unmapped but could it be mapped and used??
6 MB of free space will be within our branch range

19. Kernel Address Space(64 bit)0x
Invalid
Kernel Address Space(64 bit) 0x
Kernel Text Segment 0x
Kernel Data Segment
Memory Mgmt Structures
8 MB Range 0x 0x
32 bit Kernel heap
0x000.7c 0x
64 bit kernel heap 0x

20. Current Allocations Inside nucleus, able to obtain
260 KB via nucleus malloc (mach_mod_alloc)
64 KB via dummy module
and 8 KB by overwriting routines
28,000 Springboards for basic blocks inside nucleus
Can successfully allocate all patch heap space in 32 bit kernel heap
Able to instrument afs (18,000 basic blocks)
210 KB Springboard space in 32 bit kernel heap
Not yet able to instrument the entire kernel

21. Code Coverage kerninstd Information Flow Kernel Space Data Heap
Instrumentation request
Sampling request
kerninstd
ioctl()
Patch Heap
Data Heap
(counters)
/dev/kerninst
Kernel Space

22. Code Coverage kerninstd Information Flow Kernel Space Data Heap
Instrumentation request
Individual counter values
Sampling request
kerninstd
ioctl()
Patch Heap
Data Heap
(counters)
/dev/kerninst
Kernel Space

23. Experimental Results Instrumented afs Workload Measurements
Build for a large program (code coverage)
Find through multiple layers of directories
Read all files in /usr/include
Measurements
Number of functions covered
Number of basic blocks covered

24. Experimental Results for AFS
Basic blocks: 18163
Functions: 922

25. Current Challenges Memory allocation issues Parsing issues
More springboard space needed to instrument all basic blocks
Parsing issues
206 of 13,000 functions, currently cannot be parsed
Unanalyzable jump instructions

26. Future Work Reduction in number of instrumentation points needed
Dominator tree analysis
Dyninst Code Coverage tool achieves a reduction of 33%-49%
De-instrumentation policy
Based on time interval
Based on de-instrumentation request queue size
Sampling optimization
Only samples reflecting a change in value should be sent

27. Conclusion Kernel Code Coverage Mass instrumentation
Gives the ability to know at a fine level, what kernel code is exercised by a given workload
Dynamically de-instrument
Ongoing work