Coverage Estimating the quality of a test suite

2 Code Coverage A code coverage model calls out the parts of an implementation that must be exercised to satisfy an implementation-based test model Coverage, as a metric, is the percentage of these parts exercised by a test suite Commercial tools frequently contain coverage analyzers

3 Coverage usefulness 100% coverage of some aspect is never a guarantee of bug-free software Coverage reports can point out inadequate test suites suggest the presence of surprises, such as blind spots in the test design Help identify parts of the implementation that require implementation-based test design

4 Coverage example TEX and AWK are widely used programs with comprehensive test suites Coverage analysis showed SystemSegmentBranchP-useC-use TEX AWK

5 How to measure coverage? The source code is instrumented Depending on the code coverage model, code that writes to a trace file is inserted in every branch, statement etc. Most commercial tools measure segment and branch coverage

6 Code coverage models Statement Coverage Branch Coverage Multiple-Condition Coverage Object Code Coverage Data Flow Coverage

7 Control Flow Graphs A segment is one or more lexically contiguous statements with no conditionally executed statements. That is, once a segment is entered, all statements in the segment will execute. The last statement in a segment must be a predicate, or a loop control, a break, a goto, a method exit.

9 Control flow graph for previous slide

10 Control flow graphs Depict which program segments may be followed by others A segment is a node in the CFG A conditional transfer of control is a branch represented by an edge An entry node (no inbound edges) represents the entry point to a method An exit node (no outbound edges) represents an exit point of a method

11 Control flow graphs An entry-exit path is a path from the entry node to the exit node Path expressions represent paths as sequences of nodes Loops are represented as segments within parentheses followed by an asterisk

12 Paths in our example

13 Statement coverage Achieved when all statements in a method have been executed at least once Segment coverage counts segments rather than statements Assume two segments P and Q. P has one statement, Q has nine. Exercising only one of the segments will give 10% or 90% statement coverage. Segment coverage will be 50% in both cases Path 17 in the previous slide achieves statement coverage

14 Statement coverage problems Predicate may be tested for only one value (misses many bugs) Loop bodies may only be iterated once Statement coverage can be achieved without branch coverage. Important cases may be missed char* c = NULL; if (x == y) c = &aString; *c = “oops”;

15 Statement coverage problems 100% statement coverage is only a minimum requirement

16 Branch coverage Achieved when every path from a node is executed at least once At least one true and one false evaluation for each predicate Can be achieved with D+1 paths in a control flow graph with D 2-way branching nodes and no loops Even less if there are loops Branch coverage in the example XLXC(YF)*YGL XC(YF)*YGH(ZK)*ZL

17 Branch coverage problems Short-circuit evaluation means that many predicates might not be evaluated A compound predicate is treated as a single statement. If n clauses, 2 n combinations, but only 2 are tested Only a subset of all entry-exit paths is tested (3 of 11 in the example) if (a == b) x++; if (x == y) x--;

18 Multiple-condition coverage Condition coverage: Evaluate each condition as true and as false at least once… Branch/condition coverage: ditto + each branch at least once Multiple condition coverage: all true-false combinations of simple conditions at least once… Not necessarily achievable due to lazy evaluation or mutually exclusive conditions

19 Object code coverage For safety critical applications! The executable is instrumented in order to measure the percentage of object code branches and segments covered by a test suite The structure of the object code may be very different than that of the source code No evidence that object code coverage reveals more bugs than source code coverage

20 Dealing with Loops A loop can be thought of as a path with several special cases!

21 Data Flow Coverage Considers how data gets accessed and modified in the system and how it can get corrupted Common access-related bugs Using an undefined or uninitialized variable Assigning a value to a variable more than once without an intermediate access Deallocating or reinitializing a variable before it is constructed, initialized, or used Deleting a collection object leaving its members unaccessible (garbage collection helps here)

22 Data Flow Model Two kinds of actions are considered: Define: changes the value of an instance variable Use: get the value of an instance variable without changing it C-use: Used in an expression P-use: Used inside a predicate A data flow path is a control flow path that starts with a D and ends with (but does not include) a D

23 An Example A segment A decision predicate involving max Definitions of max A definition of len main() /* find longest line */ { int len; extern int max; extern char save[]; max = 0; while ((len = getline ()) > 0) if (len >= max) { max = len; copy(); } if (max > 0) /* there was a line */ printf("%s", save); } A p-use of max A c-use of len

24 An OO example public void bar(int x) { MyClass fred = new MyClass(); if (x > 0) fred.setX(x); int y = fred.getX(); } Define action for fred Use action for fred DD-path DU-path

25 All-Defs Criterion For every variable v For every Define action d of v There exists at least one test case that follows a path from d to a Use action of v

26 All-Uses Criterion For every variable v For every Define action d of v For every Use action u of v There exists at least one test case that follows a path from d to u

27 All-P-Uses / Some-C-Uses For every variable v For every Define action d of v For every p-Use u of v There exists at least one test case that follows a path from d to u If there is no p-Use of v, there must exist at least one test case that follows a path from d to a c-Use of v

28 All-C-Uses / Some-P-Uses For every variable v For every Define action d of v For every c-Use u of v There exists at least one test case that follows a path from d to u If there is no c-Use of v, there must exist at least one test case that follows a path from d to a p-Use of v

29 Data flow analysis issues Aliasing of variables causes serious problems! Working things out by hand for anything but small methods is hopeless Compiler-based tools help in determining all DU paths

30 FAQ about Coverage Is 100% coverage the same as exhaustive testing? Are branch and path coverage the same? Can path coverage be achieved? Is every path in a flow graph testable? Is less than 100% coverage acceptable? Can I trust a test suite without measuring coverage? When can I stop testing?

31 Some answers… When you run out of time When continued testing reveals no new faults When you cannot think of any new test cases When you reach a point of diminishing returns When mandated coverage has been attained When all faults have been removed

32 A coverage counter-example void Depository::give_change(int price) { int n_100, n_25, n_10, n_5; if (deposit <= price) { change_due = 0; } else { change_due = deposit-price; n_100 = change_due / 100; change_due = change_due – n_100*100; n_25 = change_due / 25; change_due = change_due – n_25*25; n_10 = change_due / 10; change_due = change_due – n_10*10; n_100 = change_due / 10; // A cut-and paste bug }

33 Path Sensitization Path sensitization is the process of determining argument and instance variable values that will cause a particular path to be taken It is undecidable Must be solved heuristically Gets complicated as size increases

34 Using existing test cases Inspect the existing test suite for missing test values or feature combinations Identify a test case that follows part of the missing path. Change input to force the opposite path to be taken at the critical branch Begin with the conditions at the end of the missing path and trace backwards until you reach the entry node

35 Paths that cannot be executed Short-circuit evaluation Mutually exclusive conditions (x > 2) || (x < 10) Redundant predicates if (x == 0) do1; else do2; if (x != 0) do3; else do4; Dead code “This should never happen”