1. Testing Approaches

2. The “Test Case Problem”: A More Formal Treatment
P is a program, D is the domain of inputs, R is the range of outputs, S is the specification for P
Choose test suite T subset D s.t. for each fault in P, there exists t member T such that P(t) ≠ S(t)
For every P and D such a test suite exists, but there is no algorithm for computing it. P D R S

3. Approach 1: Equivalence Partitioning
Input data and output results often fall into different classes
Each class is an equivalence partition
Program behaves in an equivalent way for each class member
If one member fails, all fail
If one member passes, all pass
Tests are chosen from each partition

4. Equivalence Partitioning
Divide input domain into classes of data (based on input classes and/or relationship to output classes)
A test can cover all “equivalent” data elements
Partitions consist of valid and invalid sets
Sample of class types for different input types:
Range of values: one valid, two invalid partitions
Specific value: one valid, one or two invalid partitions
Set member: one valid and one invalid partition
Boolean value: one valid and one invalid class
Other elements: use common sense and analysis

5. Example: Search Routine Spec. with Pre- and Post-conditions
procedure Search (Key : ELEM ; T: ELEM_ARRAY;
Found : ret BOOLEAN; L: ret ELEM_INDEX) ;
Pre-condition
-- the array has at least one element
T’FIRST <= T’LAST
Post-condition
-- the element is found and is referenced by L
( Found and T (L) = Key) or
-- the element is not in the array
( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))

6. Example: Partitioning for a Search Routine Specification
procedure Search (Key : ELEM ; T: ELEM_ARRAY;
Found : ret BOOLEAN; L: ret ELEM_INDEX) ;
Pre-condition
-- the array has at least one element
T’FIRST <= T’LAST
Post-condition
-- the element is found and is referenced by L
( Found and T (L) = Key) or
-- the element is not in the array
( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
Search result attributes
Array attributes
Key element attributes

7. Example: Partitioning for a Search Routine Specification
procedure Search (Key : ELEM ; T: ELEM_ARRAY;
Found : ret BOOLEAN; L: ret ELEM_INDEX) ;
Pre-condition
-- the array has at least one element
T’FIRST <= T’LAST
Post-condition
-- the element is found and is referenced by L
( Found and T (L) = Key) or
-- the element is not in the array
( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
Search result attributes
Key element in array
Key element not in array
Array attributes
Has a single value
Has more than one value
Key element attributes
Position in array

8. Example of Partitioning for a Search Routine Specification
Classes

9. Example of Partitioning for a Search Routine Specification
Classes
Values
Spec problem?

10. Does it Really Work?
Observation (Weyuker and Jeng, “Analyzing Partition Testing Strategies, TSE 17(7), July 1991)
“Partition testing can be better, worse, or the same as random testing, depending on how the partitioning is done”
Good partition: group of inputs are likely to all cause failures when exposed to same/similar circumstances

11. Approach 2: Test Adequacy Criteria
We need a strategy for choosing inputs that provides sufficient confidence and helps us determine when we have done enough.
Partition testing approaches aim to provide this
Adequacy criterion: A partitioning criterion that helps us select test data and lets us judge the sufficiency of a set of test data for a piece of software.

12. Two Classes of Adequacy Criteria: Black Box and White Box
Black Box (aka Spec-Based, aka Functional)
Tests derived from functional requirements
Input/Output Driven
Internal nature of the software is not relevant to design the tests
White Box (aka Code-Based, aka Structural)
Tests derived from code structure - Code Based
Tests are evaluated in terms of coverage of the code structures
Many others in between

13. Black vs White-Box: Which is Better?
Spec: evenORodd returns 0 if number is even, -1 if it is odd.
Code:
int evenORodd (int number) {
int result;
result= number mod 2;
return (result); }
evenORodd works for even numbers, not for odd numbers
BB detects fault (returns 1 for odd numbers instead of –1)
WB with 100% coverage does not detect fault

14. Black vs White-Box: Which is Better?
Spec: prettyPrint prints an integer input to it
Code:
prettyPrint (int number) {
if (number > 1000)
printf(“%d thousand”, number/100);
else
printf(“%d”, number); }
prettyPrint only works for numbers < 1000
BB may not detect the fault (all inputs are considered integers)
WB with 100% statement coverage would detect the fault

15. Black Box vs White Box Testing
Missing
Features
Additional Features
Faulty Modules
Final Product
Intended
(specified)
Product OK

16. Some Test Adequacy Criteria
Requirements coverage
Statement coverage
Branch coverage
Condition coverage
Path coverage
Boundary value coverage
Dataflow coverage