Code Coverage Testing Using Hardware Performance Monitoring Support Alex Shye, Matthew Iyer, Vijay Janapa Reddi and Daniel A. Connors University of Colorado at Boulder Department of Electrical and Computer Engineering DRACO Architecture Research Group

Introduction Code coverage is simple but useful for software testing Common method of code coverage analysis is through program instrumentation –Insertion of software probes statically or dynamically –Incurs a high overhead! (~50-200% overhead) Modern processors support contain a hardware Performance Monitoring Unit (PMU) –Itanium, Pentium 4, Power PC –Allow for low overhead sampling of low level information PMU represents a low-overhead alternative to full instrumentation

PMU PMUs are becoming more advanced –Coarse-grained and fine-grained features DCPI, Oprofile- PC sampling –But PMU can do more… –For example, branch vectors on Itanium Obstacles to PMU profiling –Non-deterministic (sampling) –Sample aliasing –Sampling = Less information Offline analysis can extend PMU information! FeaturesDescription Event CountersCounts of course grained events. ex. cpu cycles, flushes,etc. Branch Trace Buffer (BTB) Record branch vector of last 4 branches executed. Filters: T/NT, predicted correct/mispredicted,etc. Instruction Event Address Registers (IEAR) Sample Icache/ITLB missed. Addresses and latency Data Event Address Registers (DEAR) Sample Dcache, DTLB, ALAT misses. Addresses and latency Itanium-2 PMU Features Goal: Explore PMU-based code coverage by sampling branch vectors and performing offline compiler analysis

Code Coverage Framework PMU Branch Vectors … Partial Paths Offline Code Coverage Intermediate File Kernel Buffer Branch Vector Hash Table Online Branch Vectors Interrupt on kernel buffer overflow Terminology Branch Vector: Series of addresses from BTB Partial Path: Path of ops in compiler IR Address Map Annotated Binary Dominator Analysis Configured to sample only taken branches

Dominator Analysis –Finds all blocks guaranteed to execute Cannot be performed effectively online But is standard in any compiler infrastructure Partial Path from Branch Vector Basic Blocks added with Dom. Analysis BTB Branch Vector Terminology Dominator: u dominates v if all paths from Entry to v include u Post Dominator: u post-dominates v if all paths from v to Exit include u

Methodology Experiments run on Itanium-2 with kernel Developed tool using perfmon kernel interface and libpfm-3.1 to interface with PMU –Only sample taken branches to elongate branch vectors Set of SPEC2000 benchmarks –Compiled with the OpenIMPACT Research Compiler With annotations OpenIMPACT module for offline analysis Compared to full code coverage information from a Pin code coverage tool Benchmark#Ops# Covered Ops 164.gzip6,4663,063 (47%) 175.vpr23,57312,229 (52%) 177.mesa89,0067,390 (8%) 179.art2,2011,515 (69%) 181.mcf1,9731,401 (71%) 183.equake3,0332,265 (75%) 188.ammp19,5625,835 (30%) 197.parser17,54111,271 (64%) 256.bzip25,0953,138 (62%) 300.twolf40,49015,705 (39%) Number of Instructions and Actual Code Covered Coverage percentage is the percent of actually covered code discovered with PMU sampling and offline analysis

Effect of Sampling Period Sampling Overhead due to: –Copy BTB to kernel buffer, interrupt on kernel buffer overflow, copy from kernel buffer into hash table

PMU vs Actual Instruction Distribution Kullback-Leibler Divergence –Relative entropy of p with respect to q –d =  k=0 p k log 2 (p k /q k )

Code Coverage

Multiple Runs Regular Sampling: 1) gzip, parser, twolf improve greatly Randomized Sampling may discover code regular sampling cannot

Conclusion Motivates and presents initial results and rational for PMU-based code coverage An example of using advanced PMU feature with branch vectors Illustrates how simple offline analysis can extend PMU information Indicates PMU could be very useful for low overhead profiling and program understanding Could be promising for profiling of released software Questions?