1. White-box Testing Black-box Testing Extensions
Unit Testing Methods
White-box Testing
Black-box Testing
Extensions

2. Software Testing Methods
Testing Goals
Reveal failures/faults/errors
Locate failures/faults/errors
Show system correctness
Within the limits of optimistic inaccuracy
Improve confidence that the system performs as specified (verification)
Improve confidence that the system performs as desired (validation)
CONTENT BLOCKED Due to copyright infringement.
Program testing can be used to show the presence of bugs, but never to show their absence [Dijkstra]

3. Unit Testing Techniques
Unit Testing checks that an individual program unit (subprogram, object class, package, module) behaves correctly.
Static Testing - testing a unit without executing the unit code
Dynamic Testing - testing a unit by executing a program unit using test data
CONTENT BLOCKED Due to copyright infringement.

4. Black-Box Testing Black box testing
Purpose: tests the overall functional behavior
Drawback: may not reveal cause of the defect
Method: Treat the module as a classic input/output module; when you provide it with certain inputs it should provide certain outputs
CONTENT BLOCKED Due to copyright infringement.
requirements
output
input
events

5. Black-box Testing (naïve approach)
Getting started:
Review documentation for expected output
Example: “A Fibonacci sequence starts like this:
0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89”
CONTENT BLOCKED Due to copyright infringement.
Unit Test Procedure:
Ask the user to provide a number from 0 to 11 inclusive
Call the Fib function
Compare result value to the corresponding value in the above list
You go try it this way

6. Specification-Based Testing
Black-box Testing is more formally known as specification-based testing
Specification-based testing
Test cases designed, selected, and run based on specifications
Use specifications to derive test cases
Requirements
Design
Function signature
Based on some kind of input domain
The approach is not naïve, but considers the range of data inputs
Typical values
Boundary values
Special cases
Invalid input values (negative testing)
CONTENT BLOCKED Due to copyright infringement.

7. Equivalence Partitioning
Equivalence partitioning is an approach to black box testing that divides the input domain of a program into classes of data from which test cases can be derived.
Math review: What is an equivalence class?
CONTENT BLOCKED Due to copyright infringement.
Equivalence classes are based on equivalence relations:
For all x, y, z in set T, the following hold:
Reflexivity: x ~ x
Symmetric:x ~ yiffy ~x
Transitivity: if x ~ y and y ~ z, then x ~ z
Examples: “=“ “has the same astrological sign”
Non-examples: “>” “is taller than”
How does this help?
Define equivalence as “these data are expected to produce the same result”
We only need to test with 1 representative data element from that class
May need to remind students of the concept of an equivalence relation from discrete math (reflexive, symmetric, transitive)

8. Equivalence partitioning
Example: Suppose a program computes the value of the function What are the equivalence classes?
X < = -2 valid
-2 < X < 1 invalid
X >= 1 valid
Test cases would be selected from each equivalence class.
Cons:
Lacks causality: 2 inputs may expect to produce the same output for entirely different reasons
No deterministic way to define the classes; often difficult for non-mathematical operations or multi-dimensional inputs
Pros:
Often one can intuitively identify representative data, so there really is no reason not to try this.
Can greatly reduce the number of test cases you need to run
CONTENT BLOCKED Due to copyright infringement.

9. Boundary Value Analysis
Boundary Value Analysis - black box technique where test cases are designed to test the boundary of an input domain.
Studies have shown that more errors occur on the "boundary" of an input domain than in the "center".
Commonly referred to as a type of “edge case”
Boundary value analysis complements and can be used in conjunction with equivalence partitioning.
Example revisited:
After using equivalence partitioning to generate equivalence classes, boundary value analysis would dictate that the boundary values of the three ranges be included in the test data. That is, we might choose the following test cases (for a 32 bit system):
X <= , -100, -2.1, -2
-2 < X < , -1, 0, 0.9
X >= 1 1, 1.1,100, 231-1
CONTENT BLOCKED Due to copyright infringement.
Edge cases are a broader term in that they imply not just data-driven boundary cases but unusual interactions with the unit under test

10. Example: Possible Boundaries
Basis: Array length
Empty,1, 2, small, or large number of elements
Basis: Position of minimum score
Smallest element first, second, in middle, in last
Basis: Number of minima
Unique minimum, a few minima, all minima
Input domain: float[] Basis: array length
CONTENT BLOCKED Due to copyright infringement.
empty
one
two
small
large
Input domain: float[] Basis: position of minimum
first
somewhere in middle
last
second
Input domain: float[] Basis: number of minima
1 minimum
All data equal
2 minima

11. Structural (White-Box) Testing
Structural testing
Test cases designed, selected, and run based on structure of the source code
Scale: tests the nitty-gritty (line-by-line)
Drawbacks: need access to the source
Use source code to derive test cases
Build a graph model of the system
Control flow
Data flow
Choose test cases that guarantee different types of coverage
Node coverage
Edge coverage
Loop coverage
Condition coverage
Path coverage
CONTENT BLOCKED Due to copyright infringement.
... our goal is to ensure that all
statements and conditions have
been executed at least once ...

12. Exhaustive Testing CONTENT BLOCKED Due to copyright infringement.
loop < 20 X 14
There are 10 possible paths! If we execute one
test per millisecond, it would take 3,170 years to
test this program!!

13. Selective Testing Selected path
CONTENT BLOCKED Due to copyright infringement.
Selected path
loop < 20 X

14. Example 1
float findAverage(float[] scores) { 2
float total = 0, min = MAX_FLOAT, min2 = MAX_FLOAT; 3
for (inti = 0 ; i<scores.length ; i++) { 4
if (scores[i] < min) { 5
min2 = min; 6
min = scores[i]; 7 } 8
else if (scores[i] < min2) 9
min2 = scores[i]; 10
total += scores[i]; 11 12
if (min != MAX_FLOAT && min2 != MAX_FLOAT) 13
output(min, min2); 14
return total / scores.length; 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

15. Node Coverage
Select test cases such that every node in the graph is visited
Also called statement coverage
Guarantees that every statement in the source code is executed at least once
Select minimal number of test cases & statements
Test case: {<1,2,3,4,5,6,7,10,11,3,4,8,9,10,11,3,12,13,14,15>} 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

16. Edge Coverage
Select test cases such that every edge in the graph is visited
Also called branch coverage
Guarantees that every branch in the source code is executed at least once
More thorough than node coverage
More likely to reveal logical errors
Test cases: {<1,2,3,4,5,6,7,10,11,3,12,13,14,15>,
<1,2,3,4,8,9,10,11,3,4,8,10,11,3,12,14,15>} 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

17. Other Coverage Criteria
Loop coverage
Select test cases such that every loop boundary and interior is tested
Boundary: 0 iterations
Interior: 1 iteration and> 1 iterations
This was missing from our tests, and could have uncovered bugs!
Watch out for nested loops
Less precise than edge coverage
Condition coverage
Select test cases such that all conditions are tested
if (min != MAX_FLOAT && min2 != MAX_FLOAT)
More precise than edge coverage
Our edge coverage test doesn’t say why 12  14 is traversed
Loop coverage would have found cases where length is 1 or 2, then we wouldn’t have found 2 mins
Condition coverage would have ensured the logic of the 2nd condition in particular on line 12 wasn’t faulty

18. Other Coverage Criteria
Path coverage
Select test cases such that every pathis traversed
Loops are a problem
Consider example on earlier slide – what is scores.length?
Suppose it was 5, how many test cases would you need?
0, 1, average, max iterations
Why are paths important?
Because previous instructions may cause side effects to runtime state that require you to verify later instructions in light of those side effects
Most thorough……but is it feasible?
Not really, so we try instead to compute the set of linearly independent paths within the graph
Called the basis paths; more to come in Metrics (McCabe)
If scores.length == 5, consider how many test cases:
- Simple approach: 2 – you just go around the loop 5 times (twice for the post-loop conditional)
But, path implies the exact sequence all the way through, so:
There are 3 paths through each loop body, so you have to choose one each time through 5 times:
6 * 6 * 6 * 6 * 6 – (3 * 2 for the conditional at the end) == 7776, and that is before you consider edge cases.

19. Unit Testing: OO Perspective
Can apply per method
But this is not considered a best practice as methods represent conceptual behaviors and as such may not be used in isolation.
Unit test an entire scenario implemented as a collection of objects exchanging messages
Think UML sequence diagram
Use-case driven unit tests often happen at the component level
OO white box testing involves verifying the object traverses a defined set of states
In response to stimuli (messages) - reactive systems
Verify object is used in the manner designed
State machines a common modeling approach for this
Polymorphism creates unique challenges
Encapsulation is violated
Different languages support different semantics

20. Unit Testing: Process Perspective
Unit testing is popular (again)
Promoted by XP/Agile methodologies
Provides continuous feedback (“code a little, test a little”)
“Testing is good so we will do it all the time”
Has surpassed inspections in popularity
Data still says inspections are better
Unit tests struggle to provide white box capability
Process Terminology
Test class - set of input data that is identified for testing
Test case - specific data chosen for testing a test class.
Test suite - set of test cases that serve a particular goal
Exhaustive testing - test all logical paths in a module

21. TDD: An Alternative View
Test-Driven Development (TDD)
Write your tests first!
A promoted practice for XP
“code a little, test a little” becomes “test a little, code a little”
Derives from your user story acceptance tests
CONTENT BLOCKED Due to copyright infringement.

22. Summary: Unit Testing White-box Black-box Unit-level testing process
Structural evaluation
Coverage difficult – set a target!
Black-box
Treats implementation of function as “unknown”
Test for valid outputs given a range of inputs
Science based on domain/range of the inputs
Unit-level testing process
Unit testing now considered a very agile way of coding.
TDD a great way to ensure you verify& validate as you go.
Many developers struggle to write complete unit tests.
Many developers struggle to maintain their unit tests.