1. Unit Testing
CS 4311
Hans Van Vliet, Software Engineering, Principles and Practice, 3rd edition, John Wiley & Sons, Chapter 13.

2. Hierarchy of Testing Testing Ad Hoc Program Testing System Testing
Acceptance
Testing
Unit Testing
Integration Testing
Function
Benchmark
Properties
Black Box
Black Box
Black Box
Black Box
Black Box
Black Box
Top Down
Pilot
Performance
Equivalence
Equivalence
Equivalence
Equivalence
Equivalence
Equivalence
Bottom Up
Boundary
Boundary
Boundary
Boundary
Boundary
Reliability
Alpha
Big Bang
Decision Table
Decision Table
Availability
Beta
Sandwich
State Transition
State Transition
State Transition
State Transition
State Transition
Security
Use Case
Use Case
Use Case
Use Case
Usability
Documentation
Domain Analysis
Domain Analysis
Domain Analysis
White Box
White Box
Portability
Control Flow
Data Flow
Capacity

3. Outline
Basic concepts
Black box testing
White box testing

4. Basic Concepts What is unit testing?
Test focusing on the smallest units of code, such as
Functions, procedures, subroutines, subprograms
Methods, classes
Component tested in isolation from the rest of the system and in a controlled environment:
Uses appropriately chosen input data
Uses component-level design description as guide
Often the target of testing frameworks such as JUnit

5. Basics Why unit testing? What to test? When and who?
Foundation for other testing like integration and system testing
What to test?
Data transformations across the unit are tested
Data structures are tested to ensure data integrity
When and who?
Frequently done (at least partially) during code development by code developers

6. Unit Test Procedures Driver Module to Be Tested Results Test cases
Stub
Stub

7. Two Different Techniques
Black box
Based on specification
Inner structure of test object is not considered
White box
Based on source code
Inner structure of test object is the basis of test case selection
Often complementary
Effectiveness of black box is similar to white box, but the mistakes found are different (Hetzel 1976, Myers 1978)
Use in combinations

8. Outline Basic concepts Black box testing White box testing
Equivalence classes …
White box testing

9. What Is Black Box Testing?
Unit (code, module) seen as a black box
No access to the internal or logical structure
Determine if given input produces expected output
Input
Output

10. Black Box Testing
Test set is derived from specifications or requirements
Goal is to cover the input space
Lots of approaches to describing input space:
Equivalence classes
Boundary value analysis
Decision tables
State transitions
Use cases
. . .

11. Advantage and Disadvantage
(Dis)Advantages
It does not require access to the internal logic of a component
However, in most real-world applications, impossible to test all possible inputs
Need to define an efficient strategy to limit number of test cases

12. General Process Analyze specifications or requirements
Select valid and invalid inputs (i.e., positive and negative tests)
Determine expected outputs
Construct tests
Run tests
Compare actual outputs to expected outputs

13. Outline Basic concepts Black box testing White box testing
Equivalence classes
Boundary values …
White box testing

14. Motivation
Assume you test a sign function, int sign(int x), whose result is defined as:
1 if x > 0
0 if x = 0
-1 otherwise (i.e., x < 0)
One way to reduce the number of test cases is:
to partition the input into three subsets
{ x | x > 0}, { x | x = 0}, { x | x < 0}
to pick one from each subset, e.g., 10, 0, and -10.

15. Basic Strategy of Equivalence Classes
Partition the input into equivalence classes
This is the tricky part.
It’s an equivalence class if:
Every test using one element of the class tests the same thing that any other element of the class tests
If an error is found with one element, it should be found with any element
If an error is not found with some element, it is not found by any element
Test a subset from each class

16. Exercise Consider a factorial function, fact(n):
if n < 0 or n >= 200, error
if 0  n  20, exact value
if 20 < n < 200, approximate value within 0.1%.
Q: What equivalence classes can you see?

17. Simple Example Suppose you are building an airline reservation system.
A traveler can be a child, an adult, or a senior.
The price depends on the type of traveler.
The seat reservation does not depend on the type of traveler.
Q: How many test cases can you identify for the reservation component and the billing component?

18. Heuristics for Finding Equivalence Classes
Identify restrictions for inputs and outputs in the specification.

19. Heuristics for Finding Equivalence Classes
Identify restrictions for inputs and outputs in the specification.
If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN).

20. Heuristics for Finding Equivalence Classes
Identify restrictions for inputs and outputs in the specification.
If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN).
If a number of values is required, create one valid and one or more invalid classes.

21. Heuristics for Finding Equivalence Classes
Identify restrictions for inputs and outputs in the specification.
If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN).
If a number of values is required, create one valid and one or more invalid classes.
If a set of values is specified where each may be treated differently, create a class for each element of the set and one more for elements outside the set.

22. Heuristics for Finding Equivalence Classes
Identify restrictions for inputs and outputs in the specification.
If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN).
If a number of values is required, create one valid and one or more invalid classes.
If a set of values is specified where each may be treated differently, create a class for each element of the set and one more for elements outside the set.
If there is a condition, create two classes, one satisfying and one not satisfying the condition.

23. Exercise
Define test cases for a program that reserves a golf tee time.
The standard green fee is $65 on weekdays (Monday-Friday) and $80 on weekend (Saturday and Sunday).
However, an El Paso resident pays a reduced green fee of $45 and $60 on weekdays and weekend, respectively.
A senior (of age 60+) pays only $40 and $50 on weekdays and weekend, respectively.
A junior (of age <17) pays only $20 and $30 on weekdays and weekend, respectively.
Q: How many equivalence classes? Define them.
Q: Define one test case per equivalence class.

24. Outline Basic concepts Black box testing White box testing
Equivalence classes
Boundary values
Decision tables …
White box testing

25. Boundary Values Observations Test the boundaries
Programs that fail at interior elements of an equivalence class usually fail at the boundaries too.
Programmers often make errors on boundary conditions (e.g., branch/loop condition x <= 10 instead of x < 10).
Test the boundaries
if it should work for 1-99, test 0, 1, 99, 100.
if it works for A-Z, A, Z, [, a, and z
The hard part is identifying boundaries.

26. Hints
If a domain is a restricted set, check the boundaries. e.g., D=[1,10], test 0, 1, 10, 11
It may be possible to test the boundaries of outputs, also.
For ordered sets, check the first and last elements.
For complex data structures, the empty list, full lists, the zero array, and the null pointer should be tested.
Extremely large data sets should be tested.
Check for off-by-one errors.

27. More Hints
Some boundaries are not obvious and may depend on the implementation (use gray box testing if needed)
Numeric limits (e.g., test 255 and 256 for 8-bit values)
Implementation limits (e.g., max array size)

28. Example
Consider a simple voltage stabilizer with an input range of volts hertz.

29. Example
Consider a simple voltage stabilizer with an input range of volts hertz.
Voltage boundaries
189 volts (below the lowest boundary)
190 volts (lowest boundary)
240 volts (highest boundary)
241 volts (above the highest boundary)
Cycles (hertz) boundaries
49 hertz (below)
50 hertz (lowest)
60 hertz (highest)
61 hertz (above)

30. One extremes with others fixed.
Example
Consider a simple voltage stabilizer with an input range of volts hertz.
Voltage boundaries
189 volts (below the lowest boundary)
190 volts (lowest boundary)
240 volts (highest boundary)
241 volts (above the highest boundary)
Cycles (hertz) boundaries
49 hertz (below)
50 hertz (lowest)
60 hertz (highest)
61 hertz (above)
How to combine?
Exhaustive,
Pairwise (later),
One extremes with others fixed.

31. Exercise
Determine the boundary values for US Postal Service ZIP codes (5 digits such as or 5 digits hyphen 4 digits such as ).
Determine the boundary values for a 15-character last name entry.
Hints: Two kinds of boundaries---size boundary and value boundary.

32. Outline Basic concepts Black box testing White box testing
Equivalence classes
Boundary values
Decision tables
Pairwise testing
State transitions
Use cases
White box testing

33. Decision Tables Construct a table (to help organize the testing)
Identify each rule or condition in the system that depends on some input
For each input to one of these rules, list the combinations of inputs and the expected results

34. Example
Theater ticket prices are discounted for senior citizens and students.
Test Case C1
Student? C2
Senior?
Result
Discount? 1 T 2 F 3 4

35. Exercise
Construct a decision table for a program that reserves a golf tee time.
The standard green fee is $65 on weekdays (Monday-Friday) and $80 on weekend (Saturday and Sunday).
However, an El Paso resident pays a reduced green fee of $45 and $60 on weekdays and weekend, respectively.
A senior (of age 60+) pays only $40 and $50 on weekdays and weekend, respectively.
A junior (of age <17) pays only $20 and $30 on weekdays and weekend, respectively.

36. Pairwise Testing: Motivation
Consider a website that must:
operate correctly with different browsers: IE 5, IE 6, and IE 7; Mozilla 1.1; Opera 7; FireFox 2, 3, and 4, and Chrome.
work using RealPlayer, MediaPlayer, or no plug-in.
run under Windows ME, NT, 2000, XP, and Vista, 7.0, and 8.0
accept pages from IIS, Apache and WebLogic running on Windows NT, 2000, and Linux servers.
How many different configurations are there?

37. Motivation Consider a website that must:
operate correctly with different browsers: IE 5, IE 6, and IE 7; Mozilla 1.1; Opera 7; FireFox 2, 3, and 4, and Chrome.
work using RealPlayer, MediaPlayer, or no plug-in.
run under Windows ME, NT, 2000, XP, and Vista, 7.0, and 8.0
accept pages from IIS, Apache and WebLogic running on Windows NT, 2000, and Linux servers.
How many different configurations are there?
A: 9 browsers x 3 plug-ins x 7 OS’s x 3 web servers x 3 server OS’s = 1701 combinations

38. Motivation Many such examples (i.e., finite sets of discrete values).
A bank is ready to test a data processing system
Customer types: Gold, Platinum, Business, Non profits
Account types: Checking, Savings, Mortgages, Consumer loans, Commercial loans
States (with different rules): CA, NV, UT, ID, AZ, NM
How many different configurations are there?

39. Approaches? When given a large number of combinations:
Give up and don’t test
Test all combinations: miss targets, delay product launch, and go out of business
Choose one or two cases
Choose a few that the programmers already ran
Choose the tests that are easy to create
List all combinations and choose first few
List all combinations and randomly choose some
“Magically” choose a small subset with high probability of revealing defects

40. Pairwise Testing
Combinatorial testing technique in which every pair of input parameters of software is tested.
Reasonable cost-benefit compromise
Much faster than exhaustive testing
More effective than less exhaustive methods that fail to exercise all possible pairs of input parameters
Why? Majority of software failures are caused by a single input parameter or a combination of two input parameters.
Each pair of input parameter values should be captured at least by one test case.
However, finding the least number of test cases is an NP-complete problem.

41. Example
Consider software that takes three input parameters, say x, y, and z.
If each parameter can have three different values, then there will be 27 different pairs: (x1, y1), (x1, y2), …, (y3, z3).
A test case (x1, y3, z2), for example, captures three of these 27 pairs: (x1, y3), (x1, z2), and (y3, z3).
By selecting test cases judiciously, all pairs of input parameters can be exercised with a minimum number of test cases; e.g., a set of 9 test cases can capture all 27 pairs of three parameters, each with three different values.

42. State Transitions
Use state transitions (e.g., UML state machine diagrams) to derive test inputs
Cover a state transition diagram, e.g.,
Visit each state at least once
Trigger each event at least once
Exercise each transition at least once
Exercise each path at least once*
* Might not be possible.

43. Example and Exercise
Customer makes a reservation and has a limited time to pay. May cancel at any time.
printed
Paid
paid
Ticketed
ordered/
start timer
Reserved
delivered
canceled
canceled/
refund
canceled/
refund
timer
expired
Cancelled
Used
Unpaid
Groups: How many tests to visit each state, trigger each event, exercise each transition, and exercise each path?

44. Use Cases Use the use cases to specify test cases
Use case specifies both normal, alternate, and exceptional operations
However, use cases may not have sufficient details, esp. for unit testing

45. Outline Basic concepts Black box testing White box testing
Control flow graph (CFG)
Statement coverage …

46. White Box Testing Test set is derived from structure of (source) code
Also known as:
Glass box testing
Structural testing
Often use a graph called a control flow graph (CFG)
To represent code structure
To cover it (CFG), e.g., all statements
Input
Output

47. Control Flow Graphs Programs are made of three kinds of statements:
Sequence (i.e., series of statements)
Condition (i.e., if statement)
Iteration (i.e., while, for, repeat statements)

48. Control Flow Graphs Programs are made of three kinds of statements:
Sequence (i.e., series of statements)
Condition (i.e., if statement)
Iteration (i.e., while, for, repeat statements)
CFG: visual representation of flow of control
Node represents a sequence of statements with single entry and single exit
Edge represents transfer of control from one node to another

49. Control Flow Graph (CFG)
Sequence
If-then-else
If-then
Iterative

50. Control Flow Graph (CFG)
Join
Join
Sequence
If-then-else
If-then
Iterative
When drawing CFG, ensure that there is one exit: include the join node if needed

51. Example 1. read (result); 2. read (x,k) 3. while result < 0 {5 6 7 9 8 2 1 4
1. read (result);
2. read (x,k)
3. while result < 0 {
4. ptr  false
5. if x > k then
ptr  true
7. x  x + 1
8. result  result + 1 }
9. print result

52. Example 1. read (result); 2. read (x,k) 3. while result < 0 {5 6 7 9 8 2 1 4
1,2
1. read (result);
2. read (x,k)
3. while result < 0 {
4. ptr  false
5. if x > k then
ptr  true
7. x  x + 1
8. result  result + 1 }
9. print result 3 9
4,5 6
Join
7,8

53. Exercise: Draw CFGs Example 1 if (a < b) then while (a < n)
a  a + 1;
4. else
while (b < n)
b  b + 1;
Example 2
1. read (a, b);
2. if (a  0 && b  0) then {
c  a + b;
if c > 10 then
c  max;
else c  min; }
7. else print ‘Done’;
Example 3
1. read (a, b);
2. if (a  0 || b  0) then {
c  a + b;
while( c < 100)
c  a + b; }
6. c  a * b;

54. Outline Basic concepts Black box testing White box testing
Control flow graph (CFG)
Statement coverage
Branch coverage
Condition coverage
Path coverage …

55. Coverage Coverage-based testing Control-flow coverage
Adequacy of testing in terms of the coverage of the product to be tested, e.g., the percentage of statements executed or the percentage of functional requirements tested.
Control-flow coverage
Statement
Branch
Condition
Path
Data-flow coverage
Def-use

56. Statement Coverage Every statement gets executed at least once
Every node in the CFG gets visited at least once, also known as all-nodes coverage

57. Example Q: Number of paths needed for statement coverage? 1 6 1 6 2 73 8 3 8 4 9 4 9 5 5

58. Branch Coverage Every decision is made true and false.
Every edge in a CFG of the program gets traversed at least once.
Also known as
Decision coverage
All edges coverage
Basis path coverage
Branch is finer than statement.
C1 is finer than C2 if
T1  C1  (T2  C2  T2  T1)
true
false

59. Example Q: Number of paths needed for branch coverage? 1 6 1 6 2 7 2 73 8 3 8 4 9 4 9 5 5

60. Condition Coverage
Make each Boolean sub-expression (called a condition) of a decision evaluated both to true and false
Example:
if (x > 0 || y > 0) { … } else { … }
C1: x > 0, x  0
C2: y > 0, y  0
Q: How many tests?

61. Many Variances of Condition Coverage
Condition coverage is not finer than branch coverage.
There are pathological cases where you can achieve condition and not branch. E.g., if (x > 0 || y > 0) { … }
(x = 10, y = 0), (x = 0, y = 10): condition but not branch
Under most circumstances, achieving condition achieves branch
Many variances, e.g.,
Multiple condition
Condition/decision
Modified condition/decision (MC/DC)

62. CFG for Condition Coverage
One way to determine the number of paths is to break the compound conditional into atomic conditionals
Suppose you were writing the CFG for the assembly language implementation of the control construct
if (A AND B) then C
endif
(short circuit eval) (no short circuit eval)
LD A LD A ; in general, lots of
BZ :endif LAND B ; code for A and B
LD B BZ :endif
BZ :endif JSR C
JSR C :endif nop
:endif nop

63. AND Condition 1 2A 2B 4 3 Join 1. read (a, b);
2. if (a == 0 && b == 0) then {
c  a + b; }
4. else c  a * b;
paths:
1, 2A,2B,3, J
1, 2A, 2B, 4, J
1, 2A, 4, J 2A
true
false 2B 4
true
false 3
Join

64. OR Condition 1 2A 3 2B 4 5 Join 1. read (a,b);
2. if (a == 0 || b == 0) then {
c  a + b;
while( c < 100)
c  a + b;}
Paths:
1, 2A, 3, 4, 5, 4 … J
1, 2A, 3, 4, J
1, 2A, 2B, 3, 4, 5, 4, … J
1, 2A, 2B, J 2A
true
false 3 2B
true
false 4 5
Join

65. Outline Basic concepts Black box testing White box testing
Control flow graph (CFG)
Statement coverage
Branch coverage
Condition coverage
Path coverage
Def-use coverage

66. Path Coverage
Every distinct path through code is executed at least once.
All-paths coverage similar to exhaustive testing
A path is a sequence of:
Statements
Branches
Edges

67. Example 1,2,3 4,5 6 Test Paths: 1, 2, 3, 4, 5, J1, 7, 8, J2 Join1
1. read (x)
2. read (z)
3. if x  0 then {
y  x * z;
x  z; }
6. else print ‘Invalid’;
7. if z > 1 then
8. print z;
9. else print ‘Invalid’;
Test Paths:
1, 2, 3, 4, 5, J1, 7, 8, J2
1, 2, 3, 4, 5, J1, 7, 9, J2
1, 2, 3, 6, J1, 7, 8, J2
1, 2, 3, 6, J1, 7, 9, J2
1,2,3
4,5
Join1 6 7 8
Join2 9
P353 Pfleeger 2nd ed 67

68. Linearly Independent Paths
It is not feasible to calculate the total number of paths.
It is feasible to calculate the number of linearly independent paths:
A complete path which, disregarding back tracking (such as loops), has an unique set of decisions in a program.
Any path through the program that introduces at least one new edge that is not included in any other linearly independent paths.
The number of linearly independent paths is the number of end-to-end paths required to touch every path segment at least once.

69. McCabe’s Cyclomatic Complexity
Software metric for the logical complexity of a program
Defines the number of independent paths in the basis set of a program (basis set: maximal linearly independent set of paths).
Provides the upper bound for the number of tests that must be conducted to ensure that all statements have been executed at least once.
For edges (E) and nodes (N), cyclomatic complexity number V is:
V = E – N + 2

70. Example: Complexity of CFG
3,4
3,4 1 1 5 5 6 2
Join
Join 3
N = 1
E = 0
V = E - N + 2 =
= 1
N = 3
E = 2
V = E - N + 2
= 2 – 3 + 2
= 1
N = 4
E = 4
V = E - N + 2 =
= 2
N = 3
E = 3
V = E - N + 2 =
= 2

71. Example 1 2,3 6 4,5 8 7 9 10 11 V = 11 – 9 + 2 = 4 Independent paths:
1-11
…-11
…-11

72. Exercise: Cyclomatic Complexity3
4,5 6
Join 9
7,8
1,2
1,2
Join
3,4 5 6 7
1, 2 3
Join 4 5 6 1 2 3
Join 5 6

73. Linearly Independent Path Coverage
Cyclomatic-number criterion: construct a test set such that all linearly independent paths are covered.
1. read (x)
2. read (z)
3. if x  0 then {
y  x * z;
x  z; }
6. else print ‘Invalid’;
7. if z > 1 then
8. print z;
9. else print ‘Invalid’;
Cyclomatic complexity: 3 (= 9 – 8 + 2)
Test Paths:
1, 2, 3, 4, 5, J1, 7, 8, J2
1, 2, 3, 4, 5, J1, 7, 9, J2
1, 2, 3, 6, J1, 7, 8, J2,
1,2,3
4,5
Join1 6 7 8
Join2 9
P353 Pfleeger 2nd ed 73

74. Outline Basic concepts Black box testing White box testing
Control flow graph (CFG)
Statement coverage
Branch coverage
Condition coverage
Path coverage
Def-use coverage

75. Data Flow Coverage
Consider how variables are treated along the various paths, a.k.a. data flow analysis, e.g., definitions and uses.
Many variations
All-defs-uses
All-defs
All-uses
All-C-uses/Some-P-uses
All-P-uses/Some-C-uses
All-P-uses
* P-uses are predicate uses, e.g., in conditions; all others are C-uses.

76. Def-Use Coverage
Def-use coverage: every path from every definition of every variable to every use of that definition is exercised in some test.
1. read (x);
2. read (z);
3. if x  0 then {
y  x * z;
x  z; }
6. else print ‘Invalid’;
7. if y > 1 then
8. print y;
9. else print ‘Invalid’;
Test Path: 1, 2, 3, 4, 5, 7, 8, J
D: x, z
U: x
1,2,3
U: x, z
D: y, x
4,5 6
U: none
Join 7
U: y 8
U: y 9
U: none
Join

77. Strength of Coverage Statement
Arrows point from weaker to stronger coverage.
Stronger coverage requires more test cases.
Branch
Def-Use
Condition
Path

78. Limitation of Paths What they don’t tell you: Timing errors
Unanticipated error conditions
User interface inconsistency (or anything else)
Configuration errors
Capacity errors