1. Software Testing Sudipto Ghosh CS 406 Fall 99 November 16, 1999

2. 11/16/99CS 406 Testing2 Learning Objectives Data flow-based testing Test adequacy assessment

3. 11/16/99CS 406 Testing3 Problems with path testing With each predicate, there is a combinatorial explosion in the number of paths. Potentially infinite number of paths because of loops. Not all paths a feasible, i.e., there may not be inputs for which the path is executed. Suppose that somehow, we satisfied this criterion. Would we find all the faults?

4. 11/16/99CS 406 Testing4 More problems A path is tested only if it is present!! Consider the example: double logBase19 (double x) { return ln(x)/ln(19); } Missing condition is: if(x > 0)

5. 11/16/99CS 406 Testing5 More problems It is possible to test every path without detecting the fault in the product. Function to test the equality of 3 integers: (Fallacious) assumption: If the average of the 3 numbers is equal to the first, then they are equal. boolean areEqual(int x,int y,int z){ if ((x+y+z)/3 == x) return TRUE; else return FALSE; } Exercise: Think of test cases.

6. 11/16/99CS 406 Testing6 Reducing the number of paths to be covered Find something between branch and path coverage. Select a set of paths that ensure branch coverage and try some other paths that help reveal errors. Consider two paths same if they differ only in their sub-paths that are caused due to loops. Restrict test cases to linear code sequences Identify set of points L, from which control flow may jump; includes entry, exit, branch statements Paths begin and end at elements in L.

7. 11/16/99CS 406 Testing7 Data Flow-Based Testing Select paths to be executed based on data flow analysis Information about where variables are defined variables are used Idea is to make sure that the definitions of variables and their subsequent use is tested

8. 11/16/99CS 406 Testing8 Variables in statements Variables occur in statements as Definition – def Variables on the left side of a statement ( c ) c := 10; Variable is used in a computation – c-use Variables on the right-hand side of a statement ( c ) temp := c + 5; Variables used in a write statement Variable is used in a predicate – p-use if( c!= MAXLENGTH ) {

9. 11/16/99CS 406 Testing9 An example (x y ) 1.scanf(x, y); if(y < 0) 2. pow = 0 – y; 3.else pow = y; 4.z = 1.0; 5.while(pow != 0) 6. { z = z * x; pow = pow – 1; } 7.if ( y < 0 ) 8. z = 1.0/z; 9.printf(z);

10. 11/16/99CS 406 Testing10 Def/use graph of example 1 2 3 6 4 5 7 8 9 def = {x, y}: c-use = {} def = {pow}: c-use = {y} def = {pow}: c-use = {y} def = {z}: c-use = {} def = {}: c-use = {} def = {z, pow}: c-use = {x, z, pow} def = {z}: c-use = {z} def = {}: c-use = {z} y y y pow y

11. 11/16/99CS 406 Testing11 More on def/use graphs With each node, we associate the c-uses of a variable With each edge, we associate the p-uses of a variable

12. 11/16/99CS 406 Testing12 Def-clear paths Def-clear path with respect to a variable x : Path from node i to node j, such that there is no def of x in the nodes in the path from i to j. Nodes i and j may have a def. Find out examples from the previous graph. Path from node i to edge (j, k), such that there is no def of x in the nodes in the path. Find out examples from the previous graph.

13. 11/16/99CS 406 Testing13 Global c-use A c-use of a variable is considered global if: there is not def of x within the block preceding the c-use Example?

14. 11/16/99CS 406 Testing14 Global def A def is global if it can be used outside the block in which it is defined. A def of a variable x, in a node i, is a global def, if: it is the last def of x in the block (node) i a def-clear path exists from i to another node which has a global c-use of x Example?

15. 11/16/99CS 406 Testing15 Set def(i) Set of variables for which there is a global def in the node i. Examples:

16. 11/16/99CS 406 Testing16 Set c-use(i) Set of variables for which there is a global def in the node i. Examples:

17. 11/16/99CS 406 Testing17 Set p-use(i, j) For an edge (i, j), the set p-use(i, j), is the set of variables for which there is a p-use for the edge (i, j). Examples:

18. 11/16/99CS 406 Testing18 Set dcu(x, i) dcu(x, i) represents all those nodes in which the global c-use of x uses the value assigned by the def of x in i. Suppose a variable x is in def(i) of node i. Set of nodes, such that : Each node has x in its c-use There is a def-clear path from i to j Examples:

19. 11/16/99CS 406 Testing19 Set dpu(x, i) dpu( x, i) represents all those edges in which the p-use of x uses the value assigned by the def of x in i. Suppose a variable x is in def(i) of node i. Set of edges, such that : each edge has x in its p-use There is a def-clear path from i to (j, k)

20. 11/16/99CS 406 Testing20 Test case selection criteria Let G be the def-use graph for a program Let P be a set of all the complete paths of G that were executed during the test. Examples:

21. 11/16/99CS 406 Testing21 All-defs criterion Make sure that the definitions of all variables are tested. For every def of every variable, one of its uses (either p-use or c-use) must be included in a path. For every node i in G and every x in def(i), P includes a def-clear path w.r.t. x to some member of dcu( x, i)

22. 11/16/99CS 406 Testing22 All-p-uses criterion All p-uses of all definitions should be tested. For every x  def (i), P includes a def- clear path w.r.t. x from i to all members of dpu(x, i). Note that a c-use may not be tested Can require all p-uses, some c-uses criterion

23. 11/16/99CS 406 Testing23 All c-uses criterion All c-uses of all definitions should be tested. For every x  def (i), P includes a def- clear path w.r.t. x from i to all members of dcu(x, i). Note that a p-use may not be tested Can require all c-uses, some p-uses criterion

24. 11/16/99CS 406 Testing24 All-uses criterion All p-uses and all c-uses of a definition must be exercised. The set P must include, for every node i and every x  def(i) a def-clear path w.r.t. x from i to all elements of dcu( x, i) and to all elements of dpu( x, i).

25. 11/16/99CS 406 Testing25 Number of test cases Can be shown theoretically that the limit of the size of test set is up to quadratic in the number of two-way decision statements in the program. Empirical studies have shown that the actual number of test cases that satisfy a criterion is quite small and increases linearly with the number of two-way decisions.

26. 11/16/99CS 406 Testing26 Inclusion relationship between criteria all-paths (path coverage) all-uses all-defs all-c-uses/ some-p-uses all-p-uses/ some-c-uses all-p-uses all-edges (branch coverage) all-nodes (statement coverage)

27. 11/16/99CS 406 Testing27 Implication of inclusion Suppose C 1 includes C 2 (C 1  C 2 ), I.e test cases satisfying C 1 also satisfy C 2 Does it mean that C 1 is always better than C 2 ? Statistically, the set of test cases satisfying C 1 will be better than those satisfying C 2. Testing done using these criteria performs better than randomly selected test cases.

28. 11/16/99CS 406 Testing28 Test assessment Develop a test set T a collection of test inputs Question: How good is T? Test assessment is the measurement of the goodness of T. Test assessment is carried out based on one or more criteria.

29. 11/16/99CS 406 Testing29 Test adequacy assessment These criteria are known as test adequacy criteria. Test assessment is also known as test adequacy assessment.

30. 11/16/99CS 406 Testing30 Test adequacy assessment (contd) Test adequacy assessment provides the following information: A metric, also known as adequacy score, or coverage, usually between 0 and 1. A list of all the weaknesses found in T, which when removed, will raise the score to 1. The weaknesses depend on the criteria used for assessment.

31. 11/16/99CS 406 Testing31 Test adequacy assessment (contd) Once the coverage has been computed, and the weaknesses identified, one can improve T. Improvement of T is done by examining one or more weaknesses and constructing new test requirements designed to overcome the weaknesses.

32. 11/16/99CS 406 Testing32 Test adequacy assessment (contd) This is continued until all weaknesses are overcome, i.e., the adequacy criterion is satisfied (coverage = 1) In some instances, it may not be possible to satisfy the adequacy criteria for one or more of the following reasons: Lack of sufficient resources (people, time) Weaknesses that cannot be removed because they are infeasible The cost of removing the weakness is not justified

33. 11/16/99CS 406 Testing33 Test adequacy assessment (contd) While improving T by removing its weaknesses, one usually tests the program more thoroughly than it has been tested so far. This additional testing is likely to result in the discovery of remaining errors. Hence, we say that test assessment and improvement helps in the improvement of software reliability.

34. 11/16/99CS 406 Testing34 Test adequacy assessment - summary procedure 0. Develop T 1. Select an adequacy criterion C 6. DONE!! 2. Measure adequacy criterion of T w.r.t C 5. More testing is warranted? 3. Is T adequate? 4. Improve T Yes No