Download presentation
Presentation is loading. Please wait.
Published byJoan Stevens Modified over 8 years ago
2
Software Testing and Reliability Southern Methodist University CSE 7314
3
Functional and structural approaches to testing
4
A few words on combinatorics Based on the Cartesian product of sets, we can count the number of possible inputs that a program has, i.e. | I |
5
Example Assume a program has a single input, Customer ID (CID) May be any value in the domain {00000-99999} What is | I | ?
6
Example Now assume we add a second input, the Order ID (OID) This may be any value in the domain {00000-99999} as well Now what is | I |?
7
Example Finally, add a credit card number to the input This is a 12 digit number | I | has now reached 10**22 If we can execute 1 million tests per second, it will take 1016 seconds, or about 300 million years!!
8
Example Since we cannot know what data may exercise a given statement/path in general, we may attempt to resort to exhaustive testing This attempt is doomed to fail due to the combinatorial explosion
9
Engineering the testing process Any engineered product (and most other things) can be tested in one of two ways –Knowing the specified function that a product has been designed to perform, tests can be conducted that demonstrate each function is fully operational while at the same time searching for errors in each function
10
Engineering the testing process –Knowing the internal workings of a product, tests can be conducted to ensure that “all gears mesh” (internal operations performed according to specifications)
11
Structural testing Uses knowledge of the internal workings Also known as Clear box/glass box Code based Can be useful for finding interesting inputs Misses an entire class of faults, missing code
12
Behavioral Uses knowledge of the specific function that is to be performed Based solely on the specification without regard for the internals Also known as Black box More user oriented Misses an entire class of faults, extra code (surprises) except by accident
13
Passing criteria How do we know when 1. a single test has passed 2. when we are done testing
14
Passing criteria A single test passes when its output is correct –This requires a specific definition of correct and ties into the automated oracle problem
15
When are we done? Conway Criteria: No syntactic errors (it compiles) No compile errors or immediate execution failures There exists Some set of data for which the program gives the correct output A typical set of data produces the correct output
16
When are we done? Difficult sets of data produce the correct output. All possible data sets in the problem specification produce the correct output All possible data sets and likely erroneous input succeeds. All inputs produce the correct output
17
Nature of software defects Logic errors and incorrect assumptions are inversely proportional to the probability that a program path will be executed
18
Nature of software defects We often believe that a logical path is not likely to be executed when, in fact, it may be executed on a regular basis Typographical errors are random More of a case for WHITE box testing……
19
Overview of Coverage testing
20
Logical coverage What does statement coverage imply ?
21
Example 1 IF (( X > 1) AND ( Y == 1)) 2 Z = Z / X 3 END 4 IF ((X == 2) OR ( Z >1)) 5 Z = Z + 1 6 END What is required for statement coverage ?
22
Example To achieve statement coverage, one can choose x = __, y = __ Many possible errors are missed: 1 IF ((X > 1) OR (Y ==1)) 1 IF ((X >=1) AND (Y ==1)) 1 IF ((X > 1) AND (Y >=1))
23
Blindness Statement coverage is blind to several classes of error when choosing values to achieve coverage
24
Assignment blindness Due to assignment of a particular value to a variable, the error does not propagate
25
Equality blindness Due to an Equality check of a particular variable, the error does not propagate
26
Self blindness Conditional itself covers up the error
27
Assignment Blindness example 1 IF ( ) 2 X = 1 3 END 4 IF (X + Y > 1) //Should have been (Y > 0) 5.... 6 END
28
Assignment Blindness example In this example, values are chosen to make the conditional true, the statement X=1 is executed and the error in line 4 is not seen
29
Assignment Blindness example If every path forces the same assignment, then the 'error' doesn't really matter, (does it exist??) –For instance, if the conditional in statement 1 always evaluated to true
30
Equality Blindness Example 1 IF (A == 2) 2..... 3 IF (B > 1) // Should have been (A + B > 3)
31
Equality Blindness Example In this example, the value of 2 is chosen for A to force execution of the body The error in statement 3 is missed
32
Self Blindness 1 IF (X < 1) // Should have been (2X < 1) In this example, the value of 0 is chosen for X
33
Observation Statement coverage is the weakest form of coverage Many classes of errors can escape this testing Typical projects have less than 80% statement coverage!!
34
Branch coverage Each branch of a decision must be executed at least once This is stronger that statement coverage, but it is still weak Note: A decision is a logical combination of one or more conditions
35
Branch coverage example 1 IF (( X > 1) AND ( Y == 1)) 2 Z = Z / X 3 END 4 IF ((X == 2) OR ( Z >1)) 5 Z = Z + 1 6 END
36
Branch coverage example To achieve branch coverage, one option is to choose –x = __, y = __
37
Branch coverage example Many possible errors are still missed: 1 IF ((X > 1) OR ( Y ==1) 1 IF ((X >=1) AND (Y ==1)) 1 IF ((X > 1) AND (Y <=1))
38
Blindness in branch coverage Branch coverage is blind to several classes of error when choosing values to achieve coverage –Compound decisions (more than one condition) are weakly tested –Boundaries of conditions (within a decision) are not explicitly tested
39
Condition coverage Each condition within a decision must assume all possible values This is 'potentially' stronger than branch coverage, but not always It may, in fact, be weaker
40
Example of condition coverage 1 IF (( X > 1) AND ( Y == 1)) 2 Z = Z / X 3 END 4 IF ((X == 2) OR ( Z >1)) 5 Z = Z + 1 6 END
41
Example of condition coverage To achieve condition coverage for statement 1, one must choose values such that: X > 1 ~(Y == 1) ~(X > 1) Y == 1
42
Example of condition coverage Both of these vectors miss the execution of statement 2 A better choice may be: ~(X > 1) ~(Y == 1) X > 1 Y == 1
43
Multi-condition coverage Every combination of possible condition values within a decision must be chosen This is strictly stronger than branch or condition coverage –still has weaknesses!
44
Multi-condition coverage example 1 IF (( X > 1) AND ( Y == 1)) 2 Z = Z / X 3 END 4 IF ((X == 2) OR ( Z >1)) 5 Z = Z + 1 6 END
45
Multi-condition coverage example To achieve multi-condition coverage for statement 1, one must choose values such that: 1. X > 1 Y == 1 2. X > 1 ~(Y == 1) 3. ~(X > 1) Y == 1 4. ~(X > 1) ~(Y == 1)
46
Multi-condition coverage example To achieve multi-condition coverage for statement 2, one must choose values such that: 5. X == 2 Z > 1 6. X == 2 ~(Z > 1) 7. ~(X == 2) Z > 1 8. ~(X == 2) ~(Z > 1)
47
Coverage XYZCovers 1011,5 2144,8 1132,6 2013,7
48
Multi-condition coverage example There are 9 paths through this code and this test set only covers 4 of those
49
Multi-condition coverage example Consider this error Line 2 is incorrectly written as Z = Z –X Only the second vector will execute this line but for the values chosen, Z-X = Z/X
50
Multi-condition coverage example Also consider that no vector will execute both line 2 and line 5 In a more complex example, it is easy to imagine variables being set in line 2 that are then incorrectly used in line 5
51
Multi-condition coverage example For instance: 2. Z = Z/X; A = 0.... 5. Z = Z + 1; B = 1/A;
52
Path coverage Every possible path must be chosen Finding vectors to accomplish this is, in general, undecidable, and usually difficult
53
Path coverage This still misses boundary conditions If statement 1 was mis-typed as X >= 1, it is conceivable that the chosen test set does not contain a vector with X=1, missing this error
54
Basis path testing
55
White box technique Test case designer can derive a logical complexity measure of a procedural design Control flow represented by a control graph
56
Flow graph notation sequence if
57
Flow graph notation while
58
Flow graph notation case
59
Flowchart 1 2 3 4 5 6 7 8 9 10 11
60
Flow graph 1 2,3 6 7 8 9 10 11 4,5 R3 R2 R1 R4 region node edge
61
Compound logic Predicate nodes a b yx x IF a OR b then procedure x else procedure y ENDIF
62
Cyclomatic complexity Software metric that provides a quantitative measure of the logical complexity of a program Value computed defined the number of independent paths in the basis set of a program Upper bound for the number of tests that must be conducted to ensure all statements have been executed at least once
63
Independent path Any path through the program that introduces at least one new set of processing statements or a new condition Must move along at least one edge that has not been traversed before the path is defined
64
Example Path 1; 1-11 Path 2; 1-2-3-4-5-10-1-11 Path 3; 1-2-3-6-8-9-10-1-11 Path 4; 1-2-3-6-7-9-10-1-11 1-2-3-4-5-10-1-2-3-6-8-9-10-1-11 is not independent because no new edges traversed
65
Basis set Paths 1,2,3, and 4 constitute the basis set If tests can be designed to force execution of these paths, every statement in the program will have been guaranteed to be executed at least one time and every condition will have been executed on its true and false sides
66
Basis set Basis set is not unique; a number of different basis sets can be derived for a given procedural design
67
How do we know how many paths to look for? Compute the cyclomatic complexity A graph theoretic computation which yields a good software metric
68
Three ways of computing 1. The number of regions of the flow graph correspond to the cyclomatic complexity 2. V(G) = E – N + 2 –E = number of flow graph edges –N = number of flow graph nodes 3. V(G) = P + 1 –P = number of predicate nodes in G
69
V(G) for Flow graph 1 2,3 6 7 8 9 10 11 4,5 R3 R2 R1 R4 region node edge
70
Example 1. Flow graph has 4 regions 2. V(G) = 11 edges – 9 nodes + 2 = 4 3. V(G) = 3 predicate nodes + 1 = 4 V(G) provides upper bound on number of tests that must be designed and executed to guarantee coverage of all program statements
71
Deriving test cases Basis path testing can be applied to a procedural design or to source code
72
PDL for test case design PROCEDURE average; * This procedure computes the average of 100 or fewer numbers that lie between bounding values; it also computes the sum and the total number valid. INTERFACE RETURNS average, total.input, total.valid; INTERFACE ACCEPTS value, minimum, maximum; TYPE value[1:100] IS SCALAR ARRAY; TYPE average, total.input, total.valid; minimum, maximum, sum IS SCALAR; TYPE i IS INTEGER
73
i = 1; total.input = total.valid = 0; sum =0; DO WHILE value[i] <> -999 AND total.input < 100; increment total.input by 1; IF value[i] >= minimum AND value[i] <= maximum THEN increment total.valid by 1; sum = sum + value[i]; ELSE skip ENDIF increment i by 1; ENDDO IF total.valid > 0 THEN average = sum / total.valid; ELSE average = -999 ENDIF END average
74
Deriving test cases for “average” 1. Using the design or code as a foundation, draw a corresponding flow graph
75
i = 1; total.input = total.valid = 0; sum =0; DO WHILE value[i] <> -999 AND total.input < 100; increment total.input by 1; IF value[i] >= minimum AND value[i] <= maximum THEN increment total.valid by 1; sum = sum + value[i]; ELSE skip ENDIF increment i by 1; ENDDO IF total.valid > 0 THEN average = sum / total.valid; ELSE average = -999 ENDIF END average Identify the nodes in this PDL!!!
76
Draw the Flow graph
77
Deriving test cases for “average” 2. Determine the cyclomatic complexity of the resultant flow graph –V(G) can be determined without developing a flow graph by counting all conditional statements in the PDL (compound conditions count as two) and then adding 1
78
Computer V(G) using the three methods 2. (continued) –V(G) =
79
Deriving test cases for “average” 3. Determine a basis set of linearly independent paths –Value of V(G) provides the number of linearly independent paths Path 1; Path 2; Path 3; Path 4; Path 5; Path 6; List the basis set paths!!
80
Deriving test cases for “average” 4. Prepare test cases that will force execution of each path in the basis set –Data should be chosen to that conditions at the predicate nodes are appropriately set as each path is tested. Test cases that satisfy the basis set just described are;
81
Prepare a set of test cases
82
Summary Each test case is executed and compared to expected results Ensures all statements executed at least once Some independent paths cannot be tested in stand-alone fashion –Tested as part of another path test
83
Graph matrices
84
Testing basis paths can be mechanized by using graph matrices Size based on number of nodes in the graph Tabular representation of a flow graph
85
Graph matrix 1 3 4 2 5 a b c d e f g
86
a db cf ge 1 2 3 4 5 1234512345 node Connected to node
87
Link weight Provides additional information about control flow Connection either exists (1) or does not exist (0) Can also represent –Probabilities of execution –Processing time of a link –Memory or resources required
88
Connection matrix 1 11 11 11 1 2 3 4 5 1234512345 node Connected to node connections 1 – 1 = 0 2 – 1 = 1 3 + 1 = 4 Cyclomatic complexity
89
Data flow testing
90
This approach selects test paths according to the locations of definitions and uses of variables in the program Definition Use (DU) testing –Does not guarantee coverage of all branches of a program but does pretty good
91
Example proc x B1; do while C1 if C2 then if C4 then B4; else B5; endif; else if C3 then B2; else B3; endif; enddo; B6; end proc;
92
Now apply the DU testing strategy We must first know the definitions and uses of variables in each condition or block in the PDL Assume variable x is defined in the last statement of blocks B1, B2, B3, B4, and B5 and used in the first statement of blocks B2, B3, B4, B5, and B6
93
Now apply the DU testing strategy The DU testing strategy requires an execution of the shortest path from each of B i, 0<i<5, to each of B j, 1<j≤6 This testing also covers any use of variable x in conditions C1, C2, C3, and C4
94
Now apply the DU testing strategy There are 25 DU chains of variable x, but we need only five paths to cover these DU chains 5 paths are needed to cover the DU chain of x from B j, 1<j≤5, and other BU chains can be covered by making these five paths contain iterations of the loop
95
Loop testing
96
Loops are the cornerstone of many algorithms in software White box testing technique Four different classes of loops –Simple loops –Concatenated loops –Nested loops –Unstructured loops
97
Simple loops
98
Testing Simple loops Skip the loop entirely Only one pass through the loop Two passes through the loop m passes through the loop where m < n (n is maximum allowable passes through the loop) n-1, n, n+1 passes through the loop
99
Nested loops
100
Cannot simply extend the approach from simple loops –Can grow geometrically 1. Start at the innermost loop. Set all other loops to minimum values
101
Nested loops 2. Conduct simple loop tests for the innermost loop while holding the outer loops at their minimum iteration parameter (e.g. loop counter) values. Add other tests for out of range or excluded values
102
Nested loops 3. work outward, conducting tests for the next loop, but keeping all other outer loops at minimum values and other nested loops to “typical” values 4. continue until all loops have been tested
103
Concatenated loops
104
These loops can be tested using the approach defined for simple loops, if each of the loops is independent of the other However, if two loops are concatenated and the loop counter for loop 1 us used as the initial value for loop 2, then the loops are not independent. When this happens, use the nesting loops approach
105
Unstructured loops
106
Unstructured loop Redesign !!! Use structured programming constructs http://acweb.colum.edu/users/rc ourington/Progclass/struprog.ht ml
107
End of lecture
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.