1 Software Unit Test Coverage And Test Adequacy Hong Zhu, Patrick A. V. Hall, John H.R. May Presented By: Arpita Gandhi

2 Survey recent research on software test criteria in the last two decades Put it into a uniform framework Address issues like Defining a test criterion Understanding different test adequacy criteria Whether they are worth the cost Future directions for this subject Goal

3 Importance today Critical feature in the software development cycle Identify correctness, completeness Quality assurance Verification and validation Reliability estimation

4 Previous Work-Test Adequacy A software test adequacy criterion is a predicate that defines what properties of a program must be exercised to constitute a thorough test. Proposed test criteria : Reliability- producing consistent results Validity – producing meaningful results Not practicably applicable Reliability and validity closely related and not independent.

5 Notions associated with a test adequacy criterion It is a stopping rule: Determines if sufficient testing is done that it can stop. E.g. execution of all statements for statement coverage It provides measurements of test quality: Degree of adequacy E.g. Percent of code coverage Current Work-Test Adequacy

6 Specify a software testing requirement Determine test cases to satisfy requirement Determine observations that should be made during testing Control the cost of testing Avoid redundant and unnecessary tests Help assess software dependability Build confidence in the integrity estimate Uses of Test Adequacy Criteria

7 Categories of Adequacy Criteria Based on: Source of information to specify testing requirements Specification based Program based Combined Interface based Underlying testing approach Structural Fault based Error based

8 Structural Testing Criteria Program-Based MutationPathBranchStatement Specification-Based Control Flow BasedData Flow Based Dependence coverage Structural Testing

9 Statement Coverage Execute every statement at least once The simplest, weakest type of testing int* p = NULL; if (condition) { p = &variable; *p = 1; } *p = 0; void function(const char* string1, const char* string2) { if (condition || strcmp(string1, string2) == 0)... } Useful link: Control flow – based testing

10 Control flow-based testing Branch Coverage Exercise all control statements in the program Subsumes statement coverage All combinations of control transfers may not be checked if (condition1 && (condition2 || function1( ))) statement1; else statement2;

11 Path Coverage Execute all paths from programs entry to exit Subsumes branch coverage Number of paths is exponential to the number of branches Many paths are impossible to exercise if (success) statement1; statement2; if (success) statement3; Control flow-based testing

12 Criteria Inclusion Hierarchy Branch Coverage Statement Coverage Path Coverage Control flow-based testing

13 Program-BasedSpecification-Based Control Flow BasedData Flow Based Dependence coverage Structural Testing Simple Definition-Use Coverage Testing for structured and dynamic data Inter-procedural testing Program-Based Data flow-based testing

14 Definition Occurance – Where a value is bound to the variable. (x=1) Use Occurance- Where the value is referred (a=x) C-use :if used compute a value or as an output (n=x+1) Global c-use: Value is assigned in a block other than the one in which it is being used. Local c-use P-use: if used to decide whether a predicate is true (x>0) Definition-Clear path w.r.t x: for all nodes in the path,there is no definition occurance of x Data flow-based testing

15 Simple definition use coverage If variable x has a global definition in node i, All definitions Criterion : test data covers some path which goes from i to a node or edge at which the value assigned to x in node i is used All Uses Criterion : test data covers at least one path which goes from i to each node and edge at which the value assigned to x in node i is used All-du-paths: test data covers all paths which go from i to each node and edge at which the value assigned to x in node i is used are exercised There may be infinitely many such paths Data flow-based testing

Data flow-based testing

17 Criteria Inclusion Hierarchy All-paths All-du-paths All-uses All-c/some-p-usesAll-p/some-c-uses All-definitions All-c-usesAll-p-uses Data flow-based testing

18 For structured and dynamic data Limitations of testing criteria seen so far: No distinction between atomic (integers) and structured data arrays and records) A[0l = A[il; A[i] = A[j]; A[jl = A[0l; There is a DU path from each statement to the next. The actual DU path should involve A [0] in the first and last statement. Proposed solution: Treat structured data elements as independent data May lead to infinite du paths to be tested. Data flow-based testing

19 Test data-dependence among procedure interfaces Types of data dependence Direct Definition occurs in procedure P, use in directly called procedure Q Indirect Considers multiple levels of calls and returns Definition occurs in P, use in G. Example: Class Q extends P, R extends Q Inter-procedural data-flow testing Data flow-based testing f(P) f(Q) f(R)

20 Fault-based testing criteria Error Seeding Estimate the number of faults that remain Measure quality of software testing r = # artificial faults detected # of artificial faults f = # of not seeded errors detected Estimated no. of inherent faults = (1/r)*f Applicable to any testing method Dependent on how faults are introduced

21 Program Mutation Testing Mutant: A program with a planted fault Execute mutants on each member of test set Compare results Mutation Adequacy Score =D/N D=No. of dead mutants N = No. of non equivalent mutants Fault-based testing criteria c = a – b; c = a + b; R1 R2 If (R1 = R2): mutant is alive otherwise it is killed.

22 Variants of Program Mutation Testing Weak Mutation Testing proposed to improve efficiency mutate and test components Firm Mutation Testing Select portion of program, subset of parameters and mutate them. Compare original and changed versions Less expensive than strong mutation testing,more efficient than weak mutation testing No basis to select area of program code, parameters Fault-based testing criteria

23 Criteria Inclusion Hierarchy Firm Mutation Testing Weak Mutation Testing Strong Mutation Testing Fault-based testing criteria

24 Perturbation Testing Tests the robustness of a program Predicted fault tolerance = # of faults detected total # of executions A perturbation function is applied to change the data state Example: int perturbation (int x) { int changedX; changedX = x + 50; return changedX; } Fault-based testing criteria

25 main() {int x; x = getVal(); if (x > 0) printf(X is positive); else printf(X is negative); } main() {int x; x = getVal(); x = perturbation(x); if (x > 0) printf(X is positive); else printf(X is negative); } Fault-based testing criteria Original program Fault injected program Perturbation Testing

26 Firm Mutation Testing Weak Mutation Testing Strong Mutation Testing Branch Coverage Statement Coverage Summary All-paths All-du-paths All-uses All-c/some-p-usesAll-p/some-c-uses All-definitions All-c-usesAll-p-uses