1. 14004 L8 - © D. Deugo, 2003 – 2008 Testing Procedures (You MUST bring Binder chapter 10)

2. 24004 L8 - © D. Deugo, 2003 – 2008 Class Testing? We don’t really test a class do we? –We can test each method of a class: »method scope reuses experience in procedural testing –And we can test sequences of methods of that class »we will be using control flow graphs –And we can consider inheritance by testing the flattened statechart of a class Why not just test instances in context? –In other words, why not just scenario-based testing? »See 5 reasons p.350, esp. 2 nd one –Bottom line: a contextual testing must precede contextual testing, but the question remains as to how much is worth investing in a contextual testing. There are different ways of performing class testing: –4 bullets p.353

3. 34004 L8 - © D. Deugo, 2003 – 2008 Control (or code) Coverage

4. 44004 L8 - © D. Deugo, 2003 – 2008 A First Look at Code Coverage Definition: (p.357) –A code coverage model calls out the parts of an implementation that must be exercised to satisfy an implementation-based test model. Coverage, as a metric, is the percentage of these parts exercised by a test suite. Branch coverage is said to subsume state coverage: –Exercising all branches necessarily exercises all statements –A subsumption hierarchy is an analytical ranking of coverage strategies. –No generalizable results about relative bug-finding effectiveness has been established… For Binder, a code coverage model is NOT a test model: –Test from responsibility-based models and then use coverage reports to analyze the adequacy of a test suite. Coverage models typically ignore deferred methods, class scope (focusing on method scope instead) and inheritance.

5. 54004 L8 - © D. Deugo, 2003 – 2008 Introducing CFGs Code in fig 10.2 p.363 Corresponding Control Flow Graph (CFG) in fig 10.3 p.364 –A segment is 1 or more lexically contiguous statements with no conditionally executed statements. That is, once a segment is entered, all statements in the segment will execute. –The last statement in a segment must be a predicate, or a loop control, a break, a goto, a method exit. Corresponding paths in 10.4 Dealing with If statements: fig 10.5 Dealing with switch statements: fig 10.6 –Watch out for fall-through behavior… In fig. 10.7, thinking in terms of Boolean expressions rather than individual predicates ignores lazy evaluation and does not considerably simplify the CFG.

6. 64004 L8 - © D. Deugo, 2003 – 2008 Another Example read(x);read(y) while x  y loop if x>y then x += x – y; else y += y – x; end if; end loop; gcd := x; y = 0; Do this CFG correspond to the code?

7. 74004 L8 - © D. Deugo, 2003 – 2008 Incompleteness Statement coverage typically leads to incompleteness if x < 0 then x := -x; else null; end if z := x; A negative x would result in the coverage of all statements. But not exercising x >= 0 would not cover all cases (implicit code in green italic). And, doing nothing for the case x >= may turn out to be wrong and need to be tested.

8. 84004 L8 - © D. Deugo, 2003 – 2008 Uncovering Hidden Edges if c1 and c2 then st; else sf; end if; if c1 then if c2 then st; else sf; end if; else sf; end if;

9. 94004 L8 - © D. Deugo, 2003 – 2008 Path Coverage Path Coverage Criterion: Select a test suite T such that, by executing P for each test in T, all paths leading from the initial to the final node of P’s control flow graph are traversed In practice, however, the number of path is too large, if not infinite It is key to determine “critical paths”

10. 104004 L8 - © D. Deugo, 2003 – 2008 Example if x  0 then y := 5; else z := z – x; end if; if z > 1 then z := z / x; else z := 0; end if; T1 = {, } Executes all edges but does not show risk of division by 0 T2 = {, } Would find the problem by exercising the remaining possible flows of control through the program fragment T1  T2 -> all paths covered

11. 114004 L8 - © D. Deugo, 2003 – 2008 FAQ about Coverage Is 100% coverage the same as exhaustive testing? Are branch and path coverage the same? Can path coverage be achieved? Is every path in a flow graph testable? Is less than 100% coverage acceptable? Can a trust a test suite without measuring coverage? When can I stop testing?

12. 124004 L8 - © D. Deugo, 2003 – 2008 Coverage Strategies (1) C1 coverage (i.e. line coverage) is not enough: –Typically does not exercise all true/false combinations for the predicates: »entails may miss errors in predicate –Common loop bugs are missed if loop is iterated over once C2 coverage (i.e. branch coverage): –Every path from a node is executed at least once –Unfortunately treats a compound predicate as a single statement (if n clauses, 2 n combinations, but only 2 are tested…) »This is a serious oversimplification! See example at top of p.375 –C2 can be achieved in D+1 paths with D 2-way branching nodes »For 10.7 the paths are at top of p.374 Multiple condition coverage –Condition coverage: Evaluate each condition as true and as false at least once… –Branch/condition coverage: ditto + each branch at least once –Multiple condition coverage: all true-false COMBINATIONS of simple conditions at least once… »Subsumes all of the above (and similar to All-variants of ch.6) –N.B.: not necessarily achievable (e.g. due to lazy evaluation) Object code coverage (!!): for safety critical applications!

13. 134004 L8 - © D. Deugo, 2003 – 2008 Coverage Strategies (2) McCabe’s basis-path test model: –C = e – n + 2 (cyclomatic complexity metric) –Supported by most coverage tools –Not proven useful, esp. in OO where 90% of methods typically have less than 10 lines of code… –Unreliable: see fig 10.9 p.379 »The metric is not correlated to the number of entry/exit paths… and collapses compound predicates into a single node. In fig 10.3, C is 5 (using X, Y and Z) and this is less than 1/3 of the expanded entry-exit paths »It is possible to select C paths and achieve NEITHER statement NOR branch coverage!!!! Bottom line: –CFGs are limited to small members and classes –Some authors test only the canonical form of a class –Coverage does not address inheritance

14. 144004 L8 - © D. Deugo, 2003 – 2008 Dealing with Loops A loop can be thought of as a path with several special cases! Fig 10.10 summarizes representation of different kinds of loops Fixed sized loops can be translated (unrolled) into a single segment You need at least to test 0, 1, and 2 iterations Fig 10.11: coverage can be hopeless for a method that contains more than 1 loop –breaking and nesting are highly problematic! Fig 10.12: tests for ONE loop control variable –Test: min, min +1, typical, max –1, max + exclusions Nested loops can be tested according to the strategy of table 10.3 (which proceeds from Procedure 4 p.383) –See table 10.3 p.384 Spaghetti loops with several entry or exit points should be rejected!!

15. 154004 L8 - © D. Deugo, 2003 – 2008 Data Flow Coverage

16. 164004 L8 - © D. Deugo, 2003 – 2008 Data Flow Coverage Considers how data gets modified in the system and how it can get corrupted Recasted in OO terms, at the method scope, Binder suggests a list of 7 typical errors (p.385) –It is unlikely the DUK method catches the last 2 bullets… Three kinds of actions are considered: –Define: changes the value of an instance variable –Use: C-use: really r-value P-use: used inside a predicate –Kill: “kills” instance variable Table 10.4 p.387 : all in/valid D/K/U pairs: really not conclusive!! Example p.386: J and K are interesting! This strategy is to complement control coverage!! (see bottom of p.388) Binder’s recommendation (see quote p.389): –all-uses criterion: at least one DU path be exercised for every definition-use pair. Aliasing of variables causes serious problems!! Working things out by hand for anything but small methods is hopeless… 

17. 174004 L8 - © D. Deugo, 2003 – 2008 Another Example A basic block A decision predicate involving max Definitions of max A definition of len main() /* find longest line */ { int len; extern int max; extern char save[]; max = 0; while ((len = getline ()) > 0) if (len >= max) { max = len; copy(); } if (max > 0) /* there was a line */ printf("%s", save); } A p-use of max A c-use of len

18. 184004 L8 - © D. Deugo, 2003 – 2008 Corresponding CFG max = 0 while((len = … if (len >=.. max = len if (max >.. v = … Definition DEF(v, n) v >= … P-use USE(v,n) … = …v … C-use USE(v,n) max = len … Interestingly enough, the notions of l-value and r-value would be better!!

19. 194004 L8 - © D. Deugo, 2003 – 2008 Factorial Example 1.void FACTORIAL (int N) 2.int RESULT = 1; 3.begin 4.for int I in 2.. N loop 5.RESULT = RESULT * I; 6.end loop; 7.return RESULT; 8.end; DU pairs 2 5 2 7 5 5 7

20. 204004 L8 - © D. Deugo, 2003 – 2008 Class Control Flow Coverage

21. 214004 L8 - © D. Deugo, 2003 – 2008 The Class Flow Graph “Test design at the class scope must produce method activation sequences to exercise intraclass (ie intermethod) control and data flow.”” The Class Flow Graph is constructed by joining the method flow graph with a state transition diagram for each method! –It shows how paths in each method may be followed by paths in other methods, thereby supporting intraclass path analysis. –It represents all possible intraclass control flow paths. –If no sequential constraint exists for method activation, then the exit node of each method is effectively connected to the entry node of EVERY other method, including its own. –Consider figs 10.13 through 10.17 –Can apply a coverage model to it. Alpha-omega paths are numerous, even if loops are limited to 0 and 1 iteration! Choose by inspection…

22. 224004 L8 - © D. Deugo, 2003 – 2008 Coverage and Path Sensitization Consider the code p.398 –Last line is wrong –Need a specific test case to find this! –Neither coverage, no all-DU will find this… Marick (p.399) suggests another example: –Missing a condition to check when to raise the landing gear »Preferably must be in flight ;-) Path sensitization is the process of determining argument and instance variable values that will cause a particular path to be taken: –It is undecidable… must be solved heureustically –Gets complicated as size increases… –Most of the work may have been done through UCMs –Know the 6 cases (pp.400-1) of infeasible paths Bottom line: know about equivalence partitioning and choosing test values using boundary value analysis!

23. 234004 L8 - © D. Deugo, 2003 – 2008 Boundary Value Analysis Only through domain analysis can you find the required equivalence classes and boundary values!! –Fig 10.18 domain fault model –Fig 10.19 for function at bottom of p.404 »Ignores stack… –On, off, in and out points defined p.405, open/closed boundaries p.406 »e.g., table 10.5 for our functions Applies to primitive data types. Not classes… –Relevant to instance variables and parameters… »The input domain of a method considers the flattened preconditions and flattened class invariant –Back to 7.16 and 7.17: you have to define the states of an instance if you want to apply boundary value analysis »Abstract state on/off/in points pp.408-9 E.g., for a stack in table 10.6 p.409 Basic strategy: one-by-one –One on point and one off point for each boundary –Rules pp.411-2 –Back to the domain matrix… fig 10.21

24. 244004 L8 - © D. Deugo, 2003 – 2008 Definitions An on point is a value that lies on the boundary An off point is a value not on a boundary An in point is a value that satisfies all boundary conditions and does not lie on a boundary An out point is a value that satisfies NO boundary condition and does not lie on any boundary

25. 254004 L8 - © D. Deugo, 2003 – 2008 Appendix 1 Formal Definitions for Data Flow Coverage

26. 264004 L8 - © D. Deugo, 2003 – 2008 Basic Idea Generates test data according to the way data is manipulated in the program Helps define intermediary criteria between all- edges testing (possibly too weak) and all-paths testing (often impossible) But needs effective tool support

27. 274004 L8 - © D. Deugo, 2003 – 2008 Basic Definitions (1) Node n  CFG(P) is a defining node of the variable v  V, written as DEF(v,n), iff the value of the variable v is defined in the statement corresponding to node n Node n  CFG(P) is a usage node of the variable v  V, written as USE(v,n), iff the value of the variable v is used in the statement corresponding to node n A usage node USE(v,n) is a predicate use (denoted as P- Use) iff the statement n is a predicate statement, otherwise USE(v,n) is a computation use (denoted as C-use) –Not the same as l-value and r-value unfortunately…

28. 284004 L8 - © D. Deugo, 2003 – 2008 Basic Definitions (2) A definition-use (sub)path with respect to a variable v (denoted du-path) is a path in PATHS(P) such that, for some v  V, there are define and usage nodes DEF(v, m) and USE(v, n) such that m and n are respectively the initial and final nodes of the path. A definition-clear (sub)path with respect to a variable v (denoted dc-path) is a definition-use path in PATH(P) with initial and final nodes DEF(v, m) and USE(v, n) such that no other node in the path is a defining node of v.

29. 294004 L8 - © D. Deugo, 2003 – 2008 Formal Definitions (1) The set T satisfies the all-Defs criterion for the program P iff for every variable v  V, T contains definition clear paths from every defining node of v to a use of v The set T satisfies the all-Uses criterion for the program P iff for every variable v  V, T contains definition clear paths from every defining node of v to every use of v, and to the successor node of each USE(v,n) The set T satisfies the all-P-Uses/Some C-Uses criterion for the program P iff for every variable v  V, T contains definition clear paths from every defining node of v to every predicate use of v, and if a definition of v has no P-Uses, there is a definition-clear path to at least one computation use.

30. 304004 L8 - © D. Deugo, 2003 – 2008 Formal Definitions (2) The set T satisfies the all-C-Uses/Some P-Uses criterion for the program P iff for every variable v  V, T contains definition-clear paths from every defining node of v to every computation use of v, and if a definition of v has no C-Uses, there is a definition-clear path to at least one predicate use. The set T satisfies the all-DU-Paths criterion for the program P iff for every variable v  V, T contains definition-clear paths from every defining node of v to every use of v, and to the successor node of each USE(v,n), and that these paths are either single loops traversals, or they are cycle free.

31. 314004 L8 - © D. Deugo, 2003 – 2008 Subsumption All paths All definition-use paths All uses All computational/ some predicate uses All computational uses All definitions All predicate/ some computational uses All predicate uses Edge (Branch) Statement

32. 324004 L8 - © D. Deugo, 2003 – 2008 Appendix 2 Another White-Box Approach: Mutant Testing

33. 334004 L8 - © D. Deugo, 2003 – 2008 Mutation Testing (!!) Basic idea: –Take a program and test data generated for that program –Create a number of similar programs (mutants), each differing from the original in one small way, i.e., each possessing a fault –E.g., replace addition operator by multiplication operator –The original data are then run through the mutants –If test data detect differences in mutants, then the mutants are said to be dead –If they do not, the test data are deemed inadequate and the test data need to be re-examined, possibly augmented to kill the live mutant

34. 344004 L8 - © D. Deugo, 2003 – 2008 Underlying Hypotheses Hypotheses: –Competent programmers: they write programs that are nearly correct –Coupling effect: Test data that distinguishes all programs differing from a correct one by only simple errors is so sensitive that it also implicitly distinguishes more complex errors Observations: –What about more complex errors, involving several statements? Do you really believe the coupling effect? –There is some empirical evidence of these hypotheses!!