1. Specification-Based Testing and the Category-Partition Method

2. Equivalence Partitioning (Review)
Divide input domain into classes of data (based on input classes and/or relationship to output classes)
A test can cover all “equivalent” data elements
Partitions consist of valid and invalid sets
Sample of class types
Range of values: one valid, two invalid partitions
Specific value: one valid, one or two invalid partitions
Set member: one valid and one invalid partition
Boolean value: one valid and one invalid class
Other elements: use common sense and analysis

3. Search Routine Specification with Pre- and Post-conditions
procedure Search (Key : ELEM ; T: ELEM_ARRAY;
Found : ret BOOLEAN; L: ret ELEM_INDEX) ;
Pre-condition
-- the array has at least one element
T’FIRST <= T’LAST
Post-condition
-- the element is found and is referenced by L
( Found and T (L) = Key) or
-- the element is not in the array
( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))

4. Example of Partitioning for a Search Routine Specification
procedure Search (Key : ELEM ; T: ELEM_ARRAY;
Found : ret BOOLEAN; L: ret ELEM_INDEX) ;
Pre-condition
-- the array has at least one element
T’FIRST <= T’LAST
Post-condition
-- the element is found and is referenced by L
( Found and T (L) = Key) or
-- the element is not in the array
( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
Element presence
Key element in array
Key element not in array
Array attributes
Has a single value
Has more than one value
Element position in array
First
Middle
Last

5. Example of Partitioning for a Search Routine Specification
Classes

6. Example of Partitioning for a Search Routine Specification
Classes
Values

7. A Systematic Process for Partitioning: The Category Partition Method
Thomas J. Ostrand and Marc J. Balcer, “The Category-Partition Method for Specifying and Generating Functional Tests”, Communications of the ACM, Volume 31 Number 6, June 1988, pages 676 – 686

8. The Category Partition Method
Identify functional units that can be tested independently, then for each such unit:
Identify categories (parameters and environmental factors affecting execution)
Partition categories into choices
Identify constraints
Generate test frames
Iterate on 3-5 until frame set is the right size
Generate test cases from test frames
Create test script to automate test cases

9. CPM Example NAME grep - search a file for a pattern SYNOPSIS
grep <pattern><filename>
DESCRIPTION
The grep utility searches a file for a pattern and prints all lines that contain that pattern to standard output. A line that contains multiple occurrences of the pattern is printed only once. A pattern is any sequence of characters. To include a blank in the pattern, the entire pattern must be enclosed in quotes (“). To include a quotation mark in the pattern, two quotes in a row must be used (“”).

10. The Category Partition Method
Identify functional units that can be tested independently, then for each such unit:
Identify categories (parameters and environmental factors affecting execution)
Partition categories into choices
Identify constraints
Generate test frames
Iterate on 3-5 until frame set is the right size
Generate test cases from test frames
Create test script to automate test cases

11. CPM Example: Steps 1 and 2
1. Grep is a single functional unit that can be tested in isolation
2. Identify categories (parameters and environmental factors)
Related to pattern
Pattern size
Presence of enclosing quotes
Presence of blanks
Presence of quotes within the pattern
Related to filename
Presence of a file corresponding to the name
Related to file
Number of occurrences of the pattern in the file
Number of occurrences of the pattern in one line

12. The Category Partition Method
Identify functional units that can be tested independently, then for each such unit:
Identify categories (parameters and environmental factors affecting execution)
Partition categories into choices
Identify constraints
Generate test frames
Iterate on 3-5 until frame set is the right size
Generate test cases from test frames
Create test script to automate test cases

13. CPM Example: Step 3
3. Generate choices for the categories (Note: choices are more general than values, and are disjoint)
Related to Pattern
Pattern size: Empty, Single character, Many characters
Presence of enclosing quotes: Not enclosed, Enclosed, Incorrect
Presence of blanks: None, One, Many
Presence of quotes within the pattern: None, One, Many
Related to Filename
Presence of file named: File not present, File present,
Filename omitted
Related to File
Number of occurrences of pattern in file: None, One, Many
Number of occurrences of pattern in line: One, Many

14. CPM Example: Step 3 Tips for Selecting Choices
Aim for boundaries
Expose all error messages
Exercise all default values
Explore allowable character set
Overflow input buffers
Exercise repeated actions
Generate invalid outputs
Apply inputs with a variety of starting conditions (values, permissions, filenames, locations, …)
Force very large or very small computation results
Fill the system to capacity
Force resource to be unavailable
Provide corrupt file contents

15. CPM Example: Step 3 Continued
3. Generate an initial set of test frames by generating all combinations of choices. A test frame is a selection of choices, one for each category.
Related to Pattern
Pattern size: Empty, Single character, Many characters
Presence of enclosing quotes: Not enclosed, Enclosed, Incorrect
Presence of blanks: None, One, Many
Presence of quotes within the pattern: None, One, Many
Related to Filename
Presence of file named: File not present, File present, Filename omitted
Related to File
Number of occurrences of pattern in file: None, One, Many
Number of occurrences of pattern in one line: One, Many
Combinatorially this generates 3 x 3 x 3 x 3 x 3 x 3 x 2 = 1458 frames!

16. The Category Partition Method
Identify functional units that can be tested independently, then for each such unit:
Identify categories (parameters and environmental factors affecting execution)
Partition categories into choices
Identify constraints
Generate test frames
Iterate on 3-5 until frame set is the right size
Generate test cases from test frames
Create test script to automate test cases

17. CPM Example: Step 4
4. Use constraints to reduce number of test frames, and remove frames that make no sense
Constraints can specify removal of combinations that have no meaning
Empty file
More than one occurrence of the pattern in the file
Constraints can specify that incorrect inputs & values should be utilized only once, not in all combinations
File not present
Pattern

18. The Category Partition Method
Identify functional units that can be tested independently, then for each such unit:
Identify categories (parameters and environmental factors affecting execution)
Partition categories into choices
Identify constraints
Generate test frames
Iterate on 3-5 until frame set is the right size
Generate test cases from test frames
Create test script to automate test cases

19. CPM Example: Steps 5 and 6 5. Generate test frames given constraints
6. Iterate on steps 3-5 until frame set is acceptable (in terms of size, and in terms of containing meaningful tests)

20. The Category Partition Method
Identify functional units that can be tested independently, then for each such unit:
Identify categories (parameters and environmental factors affecting execution)
Partition categories into choices
Identify constraints
Generate test frames
Iterate on 3-5 until frame set is the right size
Generate test cases from test frames
Create test script to automate test cases

21. CPM Example: Step 7
7. Generate test cases from test frames by providing specific values
Test spec ‘i’: pattern is quoted, with one blank, no quotes in pattern, length many characters, filename exists, pattern appears once in file
Test case ‘i’: grep “var foo” code.c …
… var foo

22. The Category Partition Method
Identify functional units that can be tested independently, then for each such unit:
Identify categories (parameters and environmental factors affecting execution)
Partition categories into choices
Identify constraints
Generate test frames
Iterate on 3-5 until frame set is the right size
Generate test cases from test frames
Create test script to automate test cases

23. CPM Example: Step 8 8. Generate test scripts to automate test cases
grep “var foo” code.c
grep foo code.c
grep textnotthere code.c
grep “embedded “” quote” code.c
grep “embedded “” quote” code2.c
…..
code.c
code2.c …
… var foo …
… line with embedded “ quote
Question: What to do about outputs and validation?

24. A TSL Specification (Unconstrained)
Parameters:
Pattern size:
empty
single character
many characters
Presence of blanks:
none
one
many
Presence of file named:
file not present
file present
filename omitted …
Environments:
Number of occurrences of pattern in file:

25. A TSL Specification (Constrained)
Parameters:
Pattern size:
empty [property Empty]
single character [property NonEmpty]
many characters [property NonEmpty]
Presence of blanks:
none [if NonEmpty]
one [if NonEmpty and Quoted]
many [if NonEmpty and Quoted]
Presence of file named:
file not present [error]
file present
filename omitted [error] …
Environments:
Number of occurrences of pattern in file:
none [if NonEmpty] [single]
one [if NonEmpty] [property Match]
many [if NonEmpty] [property Match]

26. A TSL Test Frame Test case 28: (Key = 3.2.3.1.1.2.1)
Pattern size : many characters
Presence of enclosing quotes : enclosed
Presence of blanks : many
Presence of quotes within the pattern : none
Presence of file named : file present
Number of occurrences of pattern : one
Number of occurrences of pattern in one line : one

27. Example of Partitioning for a Search Routine Specification
procedure Search (Key : ELEM ; T: ELEM_ARRAY;
Found : ret BOOLEAN; L: ret ELEM_INDEX) ;
Pre-condition
-- the array has at least one element
T’FIRST <= T’LAST
Post-condition
-- the element is found and is referenced by L
( Found and T (L) = Key) or
-- the element is not in the array
( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
Element presence
Key element in array
Key element not in array
Array attributes
Has a single value
Has more than one value
Element position
First
Middle
Last

28. TSL Spec for the Search Routine Specification
Element in array?
Yes
No [single]
Array size
Single value
More than one value [property MultiValue]
Element position
First
Middle [if MultiValue]
Last [if MultiValue]

29. TSL Spec and Tests for the Search Routine Specification
Element in array?
Yes
No [single]
1: Element in array: No
2. Element in array: Yes
Array size: Single value
Element position: First
3. Element in array: Yes
Array size: More than one
4. Element in array: Yes
Element position: Middle
5. Element in array: Yes
Element position: Last
Array size
Single value
More than one value [property MultiValue]
Element position
First
Middle [if MultiValue]
Last [if MultiValue]

30. Example 2: Flight Status
Goto delta.com and click on “FLIGHT STATUS” Specify categories, choices, and constraints to use when testing this simple web form.

31. For Next Class
Download the TSL tool (tsl.tar.gz) from the Handouts page to the CSE machine
Untar, unzip the file (see man pages on the “gunzip” and “tar” commands if you aren’t familiar with them)
Read the README and tsl-tool-spec.txt files
Build the tsl tool as instructed
Make sure you can run the tsl tool on the example provided in the tsl materials*
Next class: in-class TSL assignment
*If, when you try to run the “tsl” excutable, you get a message something like “tsl: Command not found”, you probably need to adjust your “path”, see next page

32. Setting Your Path
On Linux/Unix systems, an environment variable, “$path”, tells the system where to look for executables. This variable is set when you log in, by code in your startup file. By default on CSE this is named “.cshrc” (unless you use a different shell than the default, in which case you probably know how to take care of this, or can use google to find out how). To run an executable that is in your current directory, your path must contain “.”, which denotes “current directory”. Here’s how to add this to your path.
Go to your home directory
Save the current .cshrc file for safety with “cp .cshrc .cshrc.save”
Edit the .cshrc file with any editor
Find the line where the variable “$PATH” is set. For instance, it might look something like: “set path = ($path /home/…../bin)”
To add “.” to the foregoing path, add the following on the line just after it: “set path = ($path .)”
Exit the editor
Type “source .cshrc” to cause the system to re-run the .cshrc
Now go to the directory where the tsl executable is, and you should be able to invoke it
If something goes wrong, go back to your home directory and copy .cshrc.save back into .cshrc, then go seek outside help.
Note: the next time you login, the variable will already be set, and you don’t need to repeat any of the foregoing steps.