Using Set Operations on Code Coverage Data to Discover Program Properties by Nick Rutar

Motivation Many Programs already have code coverage data  Various Code Coverage Tools Available  Widely Explored Area of Research  Regression tests with coverage data becoming more common Code coverage data contains wealth of information about the program Data usually limited to how program reports it Want to milk the data for all it is worth  Possibly useful for finding errors in the program

Code Coverage Three Main Types  Statement Every line of code  Conditional Every decision in program (if/else)  Path Every path in the program Program usually Instrumented  Dynamic or Static Usually presented as a composite of separate tests

Using Set Operations Why use set operations? Most developers familiar with sets Data for statement coverage maps nicely onto sets Possible to manipulate data easily and give glimpses of properties of the code Most code coverage tools implicitly use sets anyway

Set Operations Union  Traditional Coverage Intersection  Lines ran on all tests Difference  Potential for Locating Errors  Probably biggest stretch from what data is currently being used for

Set Operations At Work int main(int argc, char *argv) { int x, y, z; x = y = z =0; if (argc == 2) x = atoi(argv[1]); if (x == 1) y = 3; else if (x == 2) y = 4; if (y > 0) z = 5; else z = -2; return z; } Inputs No input 1 Union Intersection Difference 2

Off the Beaten Path Sets Diff, - Union, U Intersection, I U/I Bad Sets - U Good Sets  Sometimes give better basis for finding bad code  Closest example of prior work only dealt with one bad run at a time Any given test - itself  Gives you the empty set U (I of Sets & (U/I Bad Sets - U Good Sets))  Gives you a very rough slice of program that went bad Manipulate data as seen fit for what you are looking for …

Other Code Coverage Info Pareto principle  Better known as rule  Pareto noticed 80% of the land in Italy owned by 20% of people  Shows up in all kinds of domains Nick’s high school - 80% of girls dated 20% of the boys  Software rule 20% of the lines of code is 80% of the runtime of the software Code Coverage often has frequency information Use that information for performance bottlenecks

Implementation Create tool that can use the set information Implementation details  Created in Java  Based on output of format from LCOV coverage tool  Takes in pre-generated coverage information as input  Supports Union, Difference, and Intersection  Supports Frequency Information

Demo

Evaluation Test Large Program against its regression test  Use Dyninst for evaluation C++ program that does binary instrumentation 100+ Source Files ~30,000 LOC instrumented to create coverage data Nightly build already has coverage capability with regression tests  Verify Union matches coverage data given by tool  Use Difference to try to find errors Series of tests with various inputs See which inputs cause failure and locate lines to discover error

Future Work For the Tool  Create Template for Insertion into program This program doesn’t care what language you are using Just needs input format to generate initial sets Specify format in text file, program uses it to input data  Better Visualization to specify points of interest Highlight source code that still has active lines  Usability Write now more of a proof of concept than a battle hardened tool In General  More evaluation of using Diff for finding errors in the program  Evaluation of software bottlenecks  IDE integration

Questions???