1. Testing in the Small (aka Unit Testing, Class Testing)
1209

2. Unit Testing Focus on the smallest units of code:
Functions
Methods
Subroutines
Frequently done (at least partially) during code development by code developers
Often the target of testing frameworks such as JUnit

3. Each component is Tested in isolation from the rest of the system
Tested in a controlled environment
Uses appropriately chosen input data
Uses component-level design description as guide

4. During Unit Tests Data transformations across the module are tested
Data structures are tested to ensure data integrity

5. Unit Test Procedures Driver Module to be tested Results Test cases
Stub
Stub

6. Two fundamental approaches:
Black box
Based on specification
Inner structure of test object is not considered
White box
Inner structure of test object is the basis of test case selection

7. Usefulness
Effectiveness of black box is similar to white box, but the mistakes found are different. (Hetzel 1976, Myers 1978)
Examples of errors not detected by white box testing:
Lock up PC if keyboard used during disk IO (neglected to turn off interrupts in pre ’84 versions of DOS)
Monitor destroyed if both color and monochrome monitors used to play PC games

8. Black Box Unit Testing
1209

9. What is black-box testing?
Unit (code, module) seen as a black box
No access to the internal or logical structure
Determine if given input produces expected output.
Input
Output

10. Black Box Testing Test set is derived from requirements
Goal is to cover the input space
Lots of approaches to describing input space:
Equivalence Classes
Boundary Value Analysis
Decision Tables
State Transitions
Use Cases
. . .

11. Advantage and Disadvantage
Advantage: it does not require access to the internal logic of a component
Disadvantage: in most real-world applications, impossible to test all possible inputs
Need to define an efficient strategy to limit number of test cases

12. General Black Box Process
Analyze requirements
Select valid and invalid inputs
Determine expected outputs
Construct tests
Run tests
Compare actual outputs to expected outputs

13. Equivalence Relations:
Reflexive
Symmetric
Transitive

14. Equivalence Relations:
Reflexive
Symmetric
Transitive
What are these? G

15. Equivalence Relations:
Reflexive
x R(x,x)
Symmetric
x,y R(x,y)  R(y,x)
Transitive
x,y,z R(x,y)  R(y,z)  R(x,z)

16. Equivalence classes: Basic Strategy
Partition the input into equivalence classes
This is the tricky part
It’s an equivalence class if:
Every test using one element of the class tests the same thing that any other element of the class tests
If an error is found with one element, it should be found with any element
If an error is not found with some element, it is not found by any element
Test a subset from each class

17. Basic strategy: Example: fact(n): if n<0, or n>=200 error
if 0<=n<=20, exact value
if 20<n<200, approximate value within .1%
What classes can you see? G

18. Basic strategy: Example: fact(n): if n<0, or n>=200 error
if 0<=n<=20, exact value
if 20<n<200, approximate value within .1%
Obvious classes are n<0, 0<=n<=20, 20<n<200, and 200<=n.
Might be some subclasses here, also, but this is a good start.

19. Simple Example:
Suppose you are building an airline reservation system. A traveler can be a child, an adult, or a senior.
The price depends on the type of traveler.
The seat reservation does not depend on the type of traveler.
How many test cases can you identify for the reservation component and the billing component? G

20. Finding equivalence classes:
Identify restrictions for inputs and outputs in the specification

21. Finding equivalence classes:
Identify restrictions for inputs and outputs in the specification
If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN)

22. Finding equivalence classes:
Identify restrictions for inputs and outputs in the specification
If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN)
If a number of values is required, create one valid and two invalid classes (one invalid, two invalid)

23. Finding equivalence classes:
Identify restrictions for inputs and outputs in the specification
If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN)
If a number of values is required, create one valid and two invalid classes
If a set of values is specified where each may be treated differently, create a class for each element of the set and one more for elements outside the set

24. Finding equivalence classes:
Identify restrictions for inputs and outputs in the specification
If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN)
If a number of values is required, create one valid and two invalid classes
If a set of values is specified where each may be treated differently, create a class for each element of the set and one more for elements outside the set
If there is a condition, create two classes, one satisfying and one not satisfying the condition

25. Boundary Values:
Programs that fail at interior elements of a class usually fail at the boundaries too.
Test the boundaries.
if it should work for 1-99, test 0, 1, 99, 100.
if it works for A-Z, A, Z, [, a, and z
The hard part is identifying boundaries

26. Hints
If a domain is a restricted set, check the boundaries. e.g., D=[1,10], test 0, 1, 10, 11
It may be possible to test the boundaries of outputs, also.
For ordered sets, check the first and last elements
For complex data structures, the empty list, full lists, the zero array, and the null pointer should be tested
Extremely large data sets should be tested
Check for off-by-one errors

27. More Hints
Some boundaries are not obvious and may depend on the implementation (use gray box testing if needed)
Numeric limits (e.g., test 255 and 256 for 8-bit values)
Implementation limits (e.g., max array size)

28. Boundary Value Exercise
Determine the boundary values for US Postal Service ZIP codes
Determine the boundary values for a 15- character last name entry. G

29. Decision Tables Construct a table (to help organize the testing)
Identify each rule or condition in the system that depends on some input
For each input to one of these rules, list the combinations of inputs and the expected results

30. Decision Table Example
Theater ticket prices are discounted for senior citizens and students.
Test Case C1
Student C2
Senior
Result
Discount? 1 T 2 F 3 4

31. Pairwise Testing

32. Problem 1 From Lee Copeland, A Practitioner’s Guide to Software Test Design, Artech House Publishers, 2004.
A web site must operate correctly with different browsers: IE 5, IE 6, and IE 7; Mozilla 1.1; Opera 7; and FireFox 2, 3, and 4.
It must work using RealPlayer, MediaPlayer, or no plugin.
It needs to run under Windows ME, NT, 2000, XP, and Vista, and 7.0
It needs to accept pages from IIS, Apache and WebLogic running on Windows NT, 2000, and Linux servers
How many different configurations are there? G

33. Problem 1
A web site must operate correctly with different browsers: IE 5, IE 6, and IE 7; Mozilla 1.1; Opera 7; and FireFox 2, 3, and 4.
It must work using RealPlayer, MediaPlayer, or no plugin.
It needs to run under Windows ME, NT, 2000, XP, and Vista, and 7.0
It needs to accept pages from IIS, Apache and WebLogic running on Windows NT, 2000, and Linux servers
How many different configurations are there?
8 browsers x 3 plugins x 6 OS x 3 web servers x 3 server OSs = 1296 combinations

34. Problem 2 A bank is ready to test a data processing system
From Lee Copeland, A Practitioner’s Guide to Software Test Design, Artech House Publishers, 2004.
A bank is ready to test a data processing system
Customer types
Gold (i.e., normal)
Platinum (i.e., important)
Business
Non profits
Account types
Checking
Savings
Mortgages
Consumer loans
Commercial loans
States (with different rules): CA, NV, UT, ID, AZ, NM
How many different configurations are there? G

35. When given a large number of combinations: (options improve …)
Give up and don’t test

36. When given a large number of combinations: (options improve …)
Give up and don’t test
Test all combinations … miss targets, delay product launch, and go out of business

37. When given a large number of combinations: (options improve …)
Give up and don’t test
Test all combinations … miss targets, delay product launch, and go out of business
Choose one or two cases

38. When given a large number of combinations: (options improve …)
Give up and don’t test
Test all combinations … miss targets, delay product launch, and go out of business
Choose one or two cases
Choose a few that the programmers already ran

39. When given a large number of combinations: (options improve …)
Give up and don’t test
Test all combinations … miss targets, delay product launch, and go out of business
Choose one or two cases
Choose a few that the programmers already ran
Choose the tests that are easy to create

40. When given a large number of combinations: (options improve …)
Give up and don’t test
Test all combinations … miss targets, delay product launch, and go out of business
Choose one or two cases
Choose a few that the programmers already ran
Choose the tests that are easy to create
List all combinations and choose first few

41. When given a large number of combinations: (options improve …)
Give up and don’t test
Test all combinations … miss targets, delay product launch, and go out of business
Choose one or two cases
Choose a few that the programmers already ran
Choose the tests that are easy to create
List all combinations and choose first few
List all combinations and randomly choose some

42. When given a large number of combinations: (options improve …)
Give up and don’t test
Test all combinations … miss targets, delay product launch, and go out of business
Choose one or two cases
Choose a few that the programmers already ran
Choose the tests that are easy to create
List all combinations and choose first few
List all combinations and randomly choose some
“Magically” choose a small subset with high probability of revealing defects

43. Orthogonal Arrays A 2-D array with the property
All pairwise combinations occur in every pair of columns
Example: consider 3 variables (columns) with {a,b}, {1,2}, and (α,β) 1 2 3 a α b β 4

44. Orthogonal Array Example1 2 3 a α b β 4
Look at each pair of columns (1 and 2),( 1 and 3), and (2 and 3)
Does each of the 4 pairings appear in each?
(Yes, of course!)

45. Pairwise Testing Algorithm
Identify variables
Determine choices for each variable
Locate an orthogonal array
Map test cases to the orthogonal array
Construct tests

46. Example using Bank Identify variables
Customers (4)
Account types (5)
States (6)
Determine choices for each variable
Customer types (Gold, Platinum, Business, Non Profit)
Account types (Checking, Savings, Mortgages, Consumer loans, Commercial loans
States (CA, NV, UT, ID, AZ, NM)
Locate an orthogonal array
Map test cases to the orthogonal array
Construct tests

47. Locate an orthogonal array (look up on web)1 2 3 4 5 1 2 3 4 6

48. https://controls. engin. umich. edu/wiki/index

49. Map Test Cases (30) CA Check Gold Save Plat Mort Busi Cons NonP NV UTID AZ
Comm
Gold
Check
Plat
Save
Busi
Mort
NonP NM CA NV UT ID
Cons

50. Pairwise Summary
When the combination is large, test all pairs, not all combinations
Studies indicate that most defects are single or double-mode effects
Can be found in pairing
Few defects require more than two modes
Orthogonal arrays have the pairwise property and can be found or generated

51. State Transition Testing

52. State Transition Testing
Build STD of system or component
Cover the STD
Visit each state
Trigger each event
Exercises each transition
Exercise each path*
* Might not be possible

53. Groups: How many tests to visit each state?
Example
Payment made
Ticket Printed
Paid
Res Made
Ticketed
Cancel
Cancel/refund
Cust Order/start timer
Time Expires
Cancelled
Cust
Cancelled
Non Pay
Ticket Delivered
Cancel/refund
Used
Customer makes reservation and has
limited time to pay. May cancel at any
time.
Groups: How many tests to visit each state?

54. Example: Each State Paid Res Made Ticketed Cancelled Cancelled Cust
Payment made
Ticket Printed
Paid
Res Made
Ticketed
Cancel
Cancel/refund
Cust Order/start timer
Time Expires
Cancelled
Cust
Cancelled
Non Pay
Ticket Delivered
Cancel/refund
Used
3 tests needed:
From start to final
From start to cancelled non-pay
From start to Res Made to Cancelled Cust

55. Example: Each Event Paid Res Made Ticketed Cancelled Cancelled Cust
Payment made
Ticket Printed
Paid
Res Made
Ticketed
Cancel
Cancel/refund
Cust Order/start timer
Time Expires
Cancelled
Cust
Cancelled
Non Pay
Ticket Delivered
Cancel/refund
Used
3 tests needed:
From start to final
From start to cancelled non-pay
From start to Res Made to Cancelled Cust

56. Example: Each Transition
Payment made
Ticket Printed
Paid
Res Made
Ticketed
Cancel
Cancel/refund
Cust Order/start timer
Time Expires
Cancelled
Cust
Cancelled
Non Pay
Ticket Delivered
Cancel/refund
Used
5 tests needed

57. Use Case Testing

58. Use Case Testing Use the use cases to specify test cases
Use case specifies both normal, alternate, and exceptional operations
Use cases may not have sufficient detail
Beizer estimates that 30-40% of effort in testing transactions is generating the test data
Budget for this!

59. White-Box Testing
Pfleeger, S. Software Engineering Theory and Practice 2nd Edition. Prentice Hall, 2001.
Ghezzi, C. et al., Fundamentals of Software Engineering. Prentice Hall, 2002.
Pressman, R., Software Engineering A Practitioner’s Approach, Mc Graw Hill, 2005.
Hutchinson, Marnie, Software Testing Fundamentals, Wiley, 2003.
0310

60. White Box Testing Also known as:
Glass box testing
Structural testing
Test set is derived from structure of code
Code structure represented as a Control Flow Graph
Goal is to cover the CFG

61. Control Flow Graphs Programs are made of three kinds of statements:
Sequence (i.e., series of statements)
Condition (i.e., if statement)
Iteration (i.e., while, for, repeat statements)

62. Control Flow Graphs Programs are made of three kinds of statements:
Sequence (i.e., series of statements)
Condition (i.e., if statement)
Iteration (i.e., while, for, repeat statements)
CFG: visual representation of flow of control.
Node represents a sequence of statements with single entry and single exit
Edge represents transfer of control from one node to another

63. Control Flow Graph (CFG)
Join
Join
Sequence
If-then-else
If-then
Iterative

64. Control Flow Graph (CFG)
Join
Join
Sequence
If-then-else
If-then
Iterative
When drawing CFG, ensure that there is one exit: include the join node if needed

65. Example 1: CFG 1. read (result); 2. read (x,k)
3. while result < 0 then {
4. ptr  false
5. if x > k then
ptr  true
7. x  x + 1
8. result  result + 1 }
9. print result
Draw CFG

66. Example 1: CFG 1 2 3 4 9 5 6 8 1. read (result); 2. read (x,k)
3. while result < 0 then {
4. ptr  false
5. if x > k then
ptr  true
7. x  x + 1
8. result  result + 1 }
9. print result 3 4 9 5 6 7 8

67. Example 1: CFG 1 1,2 2 3 3 9 4 9 4,5 5 6 6 7,8 8 1. read (result);
2. read (x,k)
3. while result < 0 then {
4. ptr  false
5. if x > k then
ptr  true
7. x  x + 1
8. result  result + 1 }
9. print result 5 6 6
Join 7
7,8 8

68. In Class: Write CFGs Example 2 if (a < b) then while(a < n)
a  a + 1;
4. else
while(b < n)
b  b + 1;
Example 3
1. read (a,b);
2. if (a  0 && b  0) then {
c  a + b;
if c > 10 then
c  max
else c  min }
7. else print ‘Done’
Example 4
1. read (a,b);
2. if (a  0 || b  0) then {
c  a + b
while( c < 100)
c  a + b; }
6. c  a * b

69. Write a CFG 1 2 3 5 6 Join Example 2 1. if (a < b) then
while (a < n)
a  a + 1;
4. else
while (b < n)
b  b + 1;

70. Answers 1,2 3,4 5 6 7 Join Example 3 1. read (a,b);
2. if (a  0 && b  0) then {
c  a + b;
if c > 10 then
c  max
else c  min }
7. else print ‘Done’

71. Answers 1, 2 3 4 Join 5 6 Example 4 1. read (a,b);
2. if (a  0 || b  0) then {
c  a + b
while( c < 100)
c  a + b; }
6. c  a * b
1, 2 3
Join 4 5 6

72. Coverage
Statement
Branch
Condition
Path
Def-use
Others

73. Statement Coverage Every statement gets executed at least once
Every node in the CFG gets visited at least once

74. Example (from Hutchinson, Software Testing Fundamentals, Wiley, 2003)1 6 1 6 2 7 2 7 3 8 3 8 4 9 4 9 5 5

75. Examples: number of paths needed for Statement Coverage
Examples: number of paths needed for Statement Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003) 1 6 1 6 2 7 2 7 3 8 3 8 4 9 4 9 5 5

76. Examples: number of paths needed for Statement Coverage
Examples: number of paths needed for Statement Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003) 1 6 1 6 2 7 2 7 3 8 3 8 4 9 4 9 4 1 5 5

77. Branch coverage Every decision is made true and false
Every edge in a CFG of the program gets traversed at least once
Also known as
Decision coverage
All edge coverage
Basis path coverage

78. Examples: number of paths needed for Branch Coverage
Examples: number of paths needed for Branch Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003) 1 6 1 6 2 7 2 7 3 8 3 8 4 9 4 9 5 5

79. Examples: number of paths needed for Branch Coverage
Examples: number of paths needed for Branch Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003) 1 6 1 6 2 7 2 7 3 8 3 8 4 9 4 9 5 1 5 5

80. Condition coverage
Every complex condition is made true and false by every possible combination
E.G., (x and y)
x = true, y = true
x=false, y=true
x = true, y= false
x =false, y = false
There are lots of different types of condition coverage: Condition, multiple condition, condition/decision, modified condition/decision (MCDC), …I’m only covering the simplest.
Under most circumstances, achieving Condition achieves Branch.

81. Condition Coverage: CFGs
One way to determine the number of paths is to break the compound conditional into atomic conditionals
Suppose you were writing the CFG for the assembly language implementation of the control construct
If (A AND B) then C
Endif
(short circuit eval) (no short circuit eval)
LD A LD A ; in general, lots of
BZ :endif LAND B ; code for A and B
LD B BZ :endif
BZ :endif JSR C
JSR C :endif nop
:endif nop

82. AND Condition 1 2A 2B 4 3 Join 1. read (a,b);
2. if (a == 0 && b == 0) then {
c  a + b }
4. else c  a * b
paths:
1, 2A,2B,3, J
1, 2A, 2B, 4, J
1, 2A, 4, J 2A 2B 4 3
Join

83. OR Condition 1 2A 3 2B 4 5 Join 1. read (a,b);
2. if (a == 0 || b == 0) then }
c  a + b;
while( c < 100)
c  a + b;}
Paths:
1, 2A, 3, 4, 5, 4 … J
1, 2A, 3, 4, J
1, 2A, 2B, 3, 4, 5, 4, … J
1,2A, 2B, J 2A 3 2B 4 5
Join

84. Path A path is a sequence of statements A path is sequence of branches
A path is a sequence of edges

85. Examples: number of paths needed for Path Coverage
Examples: number of paths needed for Path Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003) 1 6 1 6 2 7 2 7 3 8 3 8 4 9 4 9 5 5

86. Examples: number of paths needed for Path Coverage
Examples: number of paths needed for Path Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003) 1 6 1 6 2 7 2 7 3 8 3 8 4 9 4 9 5 16 5 5

87. Path Coverage-1
Every distinct path through code is executed at least once
Example
1. read (x)
2. read (z)
3. if x  0 then begin
y  x * z;
x  z end
6. else print ‘Invalid’
7. if y > 1 then
8. print y
9. else print ‘Invalid’
Test Paths:
1, 2, 3, 4, 5, J1, 7, 8, J2
1, 2, 3, 4, 5, J1, 7, 9, J2
1, 2, 3, 6, J1, 7, 8, J2,
1, 2, 3, 6, J1, 7, 9, J2
1,2,3
4,5 6
Join1
P353 Pfleeger 2nd ed 7 8 9
Join2

88. Cyclomatic Complexity
Software metric for the logical complexity of a program.
Defines the number of independent paths in the basis set of a program
Provides the upper bound for the number of tests that must be conducted to ensure that all statements been have executed at least once
For Edges (E) and Nodes (N)
V(G) = E – N + 2

89. Examples: Complexity of CFGs
3,4
3,4 1 1 5 5 6 2
Join
Join 3
N=3
E=2
E-N+2=
2-3+2=
1=V(G)
N=4
E=4
E-N+2=
4-4+2=
2=V(G)
N=3
E=3
E-N+2=
3-3+2=
2=V(G)
N=1
E=0
E-N+2=
0-1+2=
1=V(G)

90. Example 1
V(G) = 11 – = 4
Independent paths:
1-11
…-11
…-11
2,3 6
4,5 8 7 9 10 11

91. In Class: Compute the cyclomatic complexity
1,2
Join
3,4 5 6 7
1, 2 3
Join 4 5 6
1,2 3 9
4,5 1 2 3
Join 5 6 6
Join
7,8

92. Example 1: CFG
1,2
N=7
E=8
E-N+2=
8-7+2=
3=V(G) 3 9
4,5 6
Join
7,8

93. Example 2 1 2 3
Join 5 6
N=6
E=8
E-N+2=
8-6+2=
4=V(G)

94. Answers
1,2
Join
3,4 5 6 7
N=7
E=8
E-N+2=
8-7+2=
3=V(G)

95. Answers
N=6
E=7
E-N+2=
7-6+2=
3=V(G)
1, 2 3
Join 4 5 6

96. Independent Path Coverage
Basis set does not yield minimal test set
Example
1. read (x)
2. read (z)
3. if x  0 then begin
y  x * z;
x  z end
6. else print ‘Invalid’
7. if y > 1 then
8. print y
9. else print ‘Invalid’
Cyclomatic complexity: 3
Test Paths:
1, 2, 3, 4, 5, J1, 7, 8, J2
1, 2, 3, 4, 5, J1, 7, 9, J2
1, 2, 3, 6, J1, 7, 8, J2,
1,2,3
4,5 6
Join1
P353 Pfleeger 2nd ed 7 8 9
Join2 96

97. Def-Use Coverage 1,2,3 Example 4,5 6 Join 7 8 9 Join
Def-use coverage: every path from every definition of every variable to every use of that definition is exercised in some test.
Example
1. read (x)
2. read (z)
3. if x  0 then begin
y  x * z;
x  z end
6. else print ‘Invalid’
7. if y > 1 then
8. print y
9. else print ‘Invalid’
1,2,3
Def: x, z
Use: x
Def: y, x
Use: x, z
4,5 6
Use: none
Join 7
Use: y 8 9
Use: none
Use: y
Test Path: 1, 2, 3, 4, 5, 7, 8, J
Join

98. Strength of Coverage Statement
Arrows point from weaker to stronger coverage.
Stronger coverage requires more test cases.
Branch
Def-Use
Condition
Path

99. What paths don’t tell you
Timing errors
Unanticipated error conditions
User interface inconsistency (or anything else)
Configuration errors
Capacity errors