1. White-Box Testing

2. Two Classes of Adequacy Criteria: Black Box and White Box
Black Box (aka Spec-Based, aka Functional)
Tests derived from functional requirements
Input/Output Driven
Internal nature of the software is not relevant to design the tests
White Box (aka Code-Based, aka Structural)
Tests derived from code structure - Code Based
Tests are evaluated in terms of coverage of the code structures
Many others in between

3. Some White Box Adequacy Criteria
Statement coverage
Decision coverage
Condition coverage
Path coverage
Dataflow coverage

4. Statement Coverage Adequacy
Statement coverage: find test cases that ensure coverage of every executable program statement
Try:
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

5. Statement Coverage Adequacy
Statement coverage: find test cases that ensure coverage of every executable program statement
Try:
i = -1, j = :
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

6. Statement Coverage Adequacy
Statement coverage: find test cases that ensure coverage of every executable program statement
Try:
i = -1, j = : 1,2,3,4,9
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

7. Statement Coverage Adequacy
Statement coverage: find test cases that ensure coverage of every executable program statement
Try:
i = -1, j = : 1,2,3,4,9
i = 10, j = 1,2 :
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

8. Statement Coverage Adequacy
Statement coverage: find test cases that ensure coverage of every executable program statement
Try:
i = -1, j = : 1,2,3,4,9
i = 10, j = 1,2 : 1,2,3,4,5,6,7,8,9
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

9. Greedy Algorithm for Achieving Statement Coverage
Input: program P
Output: test suite T
Instrument P to report statement coverage
Repeat
create test t, execute P with t
measure coverage of P by t
if t adds to cumulative coverage, add it to T
Until testing is statement-coverage-adequate
Is T necessarily
“minimal”?

10. Statement Coverage Issues
What about unreachable statements?

11. Statement Coverage Issues
What about unreachable statements?
Bar(int x)
…..
if (x==0)
print “Bar: divide by zero”
else
x = 1/x
endif
Foo(int x)
…..
if (x>0)
Bar(x)
endif

12. Statement Coverage Issues
What about unreachable statements?
What about hard-to-reach statements?

13. Statement Coverage Issues
What about unreachable statements?
What about hard-to-reach statements?
int *ptr = malloc(sizeof(int) * 5000);
if (ptr==NULL) {
garbage-collect();
printf(“Unrecoverable system error\n”);
exit(1); }

14. Statement Coverage Issues
What about unreachable statements?
What about hard-to-reach statements?
How do you choose test cases to use?
Manually
With automated test case generation tools
Start with functional, then augment until coverage is achieved.

15. Code Instrumentation
Code instrumentation involves inserting probe statements into code that report data about its execution.
How is it done and what issues does it involve?

16. Code Instrumentation
How would you instrument this using print statements?
read i
read j
sum = 0
while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
endwhile
9. print sum

17. Code Instrumentation
How would you instrument this using print statements?
read i
print “1 executed”
read j
print “2 executed”
sum = 0
print “3 executed”
print “4 executed”
while (i > 0) and (i < = 10) do
print “5 executed”
if (j >0)
sum = sum + j
print “6 executed”
endif
i = i + 1
print “7 executed”
print “8 executed”
endwhile
9. print sum
print “9 executed”

18. Code Instrumentation
Can you reduce the number of probe statements (or minimize them) and still get the same information? How?
read i
print “1 executed”
read j
print “2 executed”
sum = 0
print “3 executed”
print “4 executed”
while (i > 0) and (i < = 10) do
print “5 executed”
if (j >0)
sum = sum + j
print “6 executed”
endif
I = i + 1
print “7 executed”
print “8 executed”
endwhile
9. print sum
print “9 executed”

19. Code Instrumentation
Can you reduce the number of probe statements (or minimize them) and still get the same information? How?
What’s being assumed about program execution in this case?
read i
read j
sum = 0
print “1, 2, 3, 4 executed”
while (i > 0) and (i < = 10) do
print “5 executed”
if (j >0)
sum = sum + j
print “6 executed”
endif
i = i + 1
print “7, 8 executed”
endwhile
9. print sum
print “9 executed”

20. Other Issues Using function calls instead of prints (how?)
Effects on run-time and determinism?
Instrumenting source code vs binary code. Tradeoffs?
Profiling interpreted code in the interpreter (e.g., the JVM). Drawbacks?

21. Statement Coverage – A Weakness
What’s the weakness?
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

22. Decision Coverage Adequacy
Decision coverage: find test cases that ensure coverage of every outcome of each predicate (decision) in the program.
Try:
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

23. Decision Coverage Adequacy
Decision coverage: find test cases that ensure coverage of every outcome of each predicate (decision) in the program.
Try:
i = -1, j = 1
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

24. Decision Coverage Adequacy
Decision coverage: find test cases that ensure coverage of every outcome of each predicate (decision) in the program.
Try:
i = -1, j = : 4F
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

25. Decision Coverage Adequacy
Decision coverage: find test cases that ensure coverage of every outcome of each predicate (decision) in the program.
Try:
i = -1, j = : 4F
i = 10, j = 1, 2 :
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

26. Decision Coverage Adequacy
Decision coverage: find test cases that ensure coverage of every outcome of each predicate (decision) in the program.
Try:
i = -1, j = : 4F
i = 10, j = 1, 2 : 4T, 5T, 4F
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

27. Decision Coverage Adequacy
Decision coverage: find test cases that ensure coverage of every outcome of each predicate (decision) in the program.
Try:
i = -1, j = : 4F
i = 10, j = 1, 2 : 4T, 5T, 4F
i = 9, j = 1, 0 :
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

28. Decision Coverage Adequacy
Decision coverage: find test cases that ensure coverage of every outcome of each predicate (decision) in the program.
Try:
i = -1, j = : 4F
i = 10, j = 1, 2 : 4T, 5T, 4F
i = 9, j = 1, 0 : 4T, 5T, 5F, 4F
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

29. Greedy Algorithm for Achieving C-Coverage
Replace “C” with “decision”, “condition”, “path”, …..
Input: program P
Output: test suite T
Instrument P to report C-coverage
Repeat
create test t, execute P with t
measure coverage of P by t
if t adds to cumulative C-coverage, add it to T
Until testing is C-coverage-adequate

30. Decision Coverage Issues
What about unreachable decisions?
What about hard-to-reach decisions?
How do you choose test cases to use?
How to instrument code to measure decision coverage?
What to do about “switch” statements?

31. Decision Coverage – A Weakness
What’s the weakness?
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

32. Condition Coverage Adequacy
Condition coverage: find test cases that ensure coverage of every outcome of each predicate (decision) in the program, under each assignment of truth values to the individual conditions in that predicate.
Try:
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

33. Condition Coverage Adequacy
Condition coverage: find test cases that ensure coverage of every outcome of each predicate (decision) in the program, under each assignment of truth values to the individual conditions in that predicate.
Try:
i = -1, j = 1
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

34. Condition Coverage Adequacy
Condition coverage: find test cases that ensure coverage of every outcome of each predicate (decision) in the program, under each assignment of truth values to the individual conditions in that predicate.
Try:
i = -1, j = : 4(F,T)
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

35. Condition Coverage Adequacy
Condition coverage: find test cases that ensure coverage of every outcome of each predicate (decision) in the program, under each assignment of truth values to the individual conditions in that predicate.
Try:
i = -1, j = : 4(F,T)
i = 10, j = 1, 2
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

36. Condition Coverage Adequacy
Condition coverage: find test cases that ensure coverage of every outcome of each predicate (decision) in the program, under each assignment of truth values to the individual conditions in that predicate.
Try:
i = -1, j = : 4(F,T)
i = 10, j = 1, 2 : 4(T,T),5(T),4(T,F)
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

37. Condition Coverage Adequacy
Condition coverage: find test cases that ensure coverage of every outcome of each predicate (decision) in the program, under each assignment of truth values to the individual conditions in that predicate.
Try:
i = -1, j = : 4(F,T)
i = 10, j = 1, 2 : 4(T,T),5(T),4(T,F)
i = 9, j = 1, 0
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

38. Condition Coverage Adequacy
Condition coverage: find test cases that ensure coverage of every outcome of each predicate (decision) in the program, under each assignment of truth values to the individual conditions in that predicate.
Try:
i = -1, j = : 4(F,T)
i = 10, j = 1, 2 : 4(T,T),5(T),4(T,F)
i = 9, j = 1, : 4(T,T),5(T),5(F),4(T,F)
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

39. Condition Coverage Adequacy
Condition coverage: find test cases that ensure coverage of every outcome of each predicate (decision) in the program, under each assignment of truth values to the individual conditions in that predicate.
Try:
i = -1, j = : 4(F,T)
i = 10, j = 1, 2 : 4(T,T),5(T),4(T,F)
i = 9, j = 1, : 4(T,T),5(T),5(F),4(T,F)
4(F,F) is not possible in this case
1. read i
2. read j
3. sum = 0
4. while (i > 0) and (i < = 10) do
if (j >0)
sum = sum + j
endif
i = i + 1
read j
endwhile
9. print sum

40. Condition Coverage Issues
What about unreachable conditions?
What about hard-to-reach conditions?
How do you choose tests to use?
How to instrument code to measure condition coverage?
How many tests are needed for
(P || Q)
(P || Q || R)
(P || Q || R || S)

41. Discussion
Are code coverage criteria “partitioning strategies”? Why or why not? If yes, how so? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

42. Graph-Based Adequacy Criteria Control Flow Graphs (CFGs)
PROGRAM GCD
begin
1 read(x)
2 read(y)
3 while (x <> y) do
if (x > y) then
x = x – y
else
y = y – x
endif
endwhile
7 print x
end
Entry
read(x)
Exit
read(y)
while x <> y
if x > y
x = x - y
y = y - x
print x
endif
endwhile T F

43. Control Flow Graph Issues
Entry and exit nodes
Predicate nodes
Edge labels
Back edges
“Join” nodes
Basic blocks
Node labeling
How to handle “switch” statements?
How to represent interprocedural control flow?
Entry
read(x)
Exit
read(y)
while x <> y
if x > y
x = x - y
y = y - x
print x
endif
endwhile T F

44. Procedure for Constructing CFGs
Determine set of nodes
Create entry node E and exit node X
Add edge from E to node representing first statement
Add edges from all nodes associated with statements from which control exits the routine to X
For all nodes n1 and n2 other than E and X, add an edge from n1 to n2 if control may pass from the statement corresponding to n1 to the statement corresponding to n2
Label edges out of predicate nodes with the value to which the predicate must evaluate in order for control to flow that way

45. More CFG Issues
The prior algorithm is for humans to use; in practice tools build these, often during the process of parsing
Cost of building CFGs is linear in program size
Initially, CFGs used extensively for code optimization, now used for many things
See handouts page for more examples

46. CFG-based Coverage Criteria
Node coverage
Branch coverage
What are they similar to?
What about condition coverage?
Entry
read(x)
Exit
read(y)
while x <> y
if x > y
x = x - y
y = y - x
print x
endif
endwhile T F

47. Path Coverage Adequacy
Entry
read(x)
Exit
read(y)
while x <> y
if x > y
x = x - y
y = y - x
print x
endif
endwhile T F
Path coverage: find test cases that ensure coverage of every entry-to-exit path in the program.
Problems?

48. Restricted-Path Strategies
Cover all acyclic paths
Cover a set basis paths for the program.
Basis paths are entry-to-exit paths, and techniques for covering them typically restrict coverage to C = e-n+2 paths (C is McCabe’s complexity metric, e and n are the number of nodes in the CFG).

49. Comparing Criteria via Subsumption
Criterion A subsumes criterion B if, for any program P and test suite T for P, T being A-adequate for P implies that T is B-adequate for P.
Consider:
statement vs decision
decision vs condition
decision vs path
condition vs path
path
statement
decision
condition

50. Comparing Criteria Empirically (Hutchins et al, ICSE 94)
Strategy Mean Cases Faults Found
Random Testing %
Branch Testing %
All Uses %