Specification-Based Testing and the Category-Partition Method

Equivalence Partitioning (Review) Divide input domain into classes of data (based on input classes and/or relationship to output classes) A test can cover all “equivalent” data elements Partitions consist of valid and invalid sets Sample of class types Range of values: one valid, two invalid partitions Specific value: one valid, one or two invalid partitions Set member: one valid and one invalid partition Boolean value: one valid and one invalid class Other elements: use common sense and analysis

Search Routine Specification with Pre- and Post-conditions procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : ret BOOLEAN; L: ret ELEM_INDEX) ; Pre-condition -- the array has at least one element T’FIRST <= T’LAST Post-condition -- the element is found and is referenced by L ( Found and T (L) = Key) or -- the element is not in the array ( not Found and not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))

Example of Partitioning for a Search Routine Specification procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : ret BOOLEAN; L: ret ELEM_INDEX) ; Pre-condition -- the array has at least one element T’FIRST <= T’LAST Post-condition -- the element is found and is referenced by L ( Found and T (L) = Key) or -- the element is not in the array ( not Found and not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key )) Element presence Key element in array Key element not in array Array attributes Has a single value Has more than one value Element position in array First Middle Last

Example of Partitioning for a Search Routine Specification Classes

Example of Partitioning for a Search Routine Specification Classes Values

A Systematic Process for Partitioning: The Category Partition Method Thomas J. Ostrand and Marc J. Balcer, “The Category-Partition Method for Specifying and Generating Functional Tests”, Communications of the ACM, Volume 31 Number 6, June 1988, pages 676 – 686

The Category Partition Method Identify functional units that can be tested independently, then for each such unit: Identify categories (parameters and environmental factors affecting execution) Partition categories into choices Identify constraints Generate test frames Iterate on 3-5 until frame set is the right size Generate test cases from test frames Create test script to automate test cases

CPM Example NAME grep - search a file for a pattern SYNOPSIS grep <pattern><filename> DESCRIPTION The grep utility searches a file for a pattern and prints all lines that contain that pattern to standard output. A line that contains multiple occurrences of the pattern is printed only once. A pattern is any sequence of characters. To include a blank in the pattern, the entire pattern must be enclosed in quotes (“). To include a quotation mark in the pattern, two quotes in a row must be used (“”).

The Category Partition Method Identify functional units that can be tested independently, then for each such unit: Identify categories (parameters and environmental factors affecting execution) Partition categories into choices Identify constraints Generate test frames Iterate on 3-5 until frame set is the right size Generate test cases from test frames Create test script to automate test cases

CPM Example: Steps 1 and 2 1. Grep is a single functional unit that can be tested in isolation 2. Identify categories (parameters and environmental factors) Related to pattern Pattern size Presence of enclosing quotes Presence of blanks Presence of quotes within the pattern Related to filename Presence of a file corresponding to the name Related to file Number of occurrences of the pattern in the file Number of occurrences of the pattern in one line

The Category Partition Method Identify functional units that can be tested independently, then for each such unit: Identify categories (parameters and environmental factors affecting execution) Partition categories into choices Identify constraints Generate test frames Iterate on 3-5 until frame set is the right size Generate test cases from test frames Create test script to automate test cases

CPM Example: Step 3 3. Generate choices for the categories (Note: choices are more general than values, and are disjoint) Related to Pattern Pattern size: Empty, Single character, Many characters Presence of enclosing quotes: Not enclosed, Enclosed, Incorrect Presence of blanks: None, One, Many Presence of quotes within the pattern: None, One, Many Related to Filename Presence of file named: File not present, File present, Filename omitted Related to File Number of occurrences of pattern in file: None, One, Many Number of occurrences of pattern in line: One, Many

CPM Example: Step 3 Tips for Selecting Choices Aim for boundaries Expose all error messages Exercise all default values Explore allowable character set Overflow input buffers Exercise repeated actions Generate invalid outputs Apply inputs with a variety of starting conditions (values, permissions, filenames, locations, …) Force very large or very small computation results Fill the system to capacity Force resource to be unavailable Provide corrupt file contents

CPM Example: Step 3 Continued 3. Generate an initial set of test frames by generating all combinations of choices. A test frame is a selection of choices, one for each category. Related to Pattern Pattern size: Empty, Single character, Many characters Presence of enclosing quotes: Not enclosed, Enclosed, Incorrect Presence of blanks: None, One, Many Presence of quotes within the pattern: None, One, Many Related to Filename Presence of file named: File not present, File present, Filename omitted Related to File Number of occurrences of pattern in file: None, One, Many Number of occurrences of pattern in one line: One, Many Combinatorially this generates 3 x 3 x 3 x 3 x 3 x 3 x 2 = 1458 frames!

The Category Partition Method Identify functional units that can be tested independently, then for each such unit: Identify categories (parameters and environmental factors affecting execution) Partition categories into choices Identify constraints Generate test frames Iterate on 3-5 until frame set is the right size Generate test cases from test frames Create test script to automate test cases

CPM Example: Step 4 4. Use constraints to reduce number of test frames, and remove frames that make no sense Constraints can specify removal of combinations that have no meaning Empty file More than one occurrence of the pattern in the file Constraints can specify that incorrect inputs & values should be utilized only once, not in all combinations File not present Pattern

The Category Partition Method Identify functional units that can be tested independently, then for each such unit: Identify categories (parameters and environmental factors affecting execution) Partition categories into choices Identify constraints Generate test frames Iterate on 3-5 until frame set is the right size Generate test cases from test frames Create test script to automate test cases

CPM Example: Steps 5 and 6 5. Generate test frames given constraints 6. Iterate on steps 3-5 until frame set is acceptable (in terms of size, and in terms of containing meaningful tests)

The Category Partition Method Identify functional units that can be tested independently, then for each such unit: Identify categories (parameters and environmental factors affecting execution) Partition categories into choices Identify constraints Generate test frames Iterate on 3-5 until frame set is the right size Generate test cases from test frames Create test script to automate test cases

CPM Example: Step 7 7. Generate test cases from test frames by providing specific values Test spec ‘i’: pattern is quoted, with one blank, no quotes in pattern, length many characters, filename exists, pattern appears once in file Test case ‘i’: grep “var foo” code.c … … var foo

The Category Partition Method Identify functional units that can be tested independently, then for each such unit: Identify categories (parameters and environmental factors affecting execution) Partition categories into choices Identify constraints Generate test frames Iterate on 3-5 until frame set is the right size Generate test cases from test frames Create test script to automate test cases

CPM Example: Step 8 8. Generate test scripts to automate test cases grep “var foo” code.c grep foo code.c grep textnotthere code.c grep “embedded “” quote” code.c grep “embedded “” quote” code2.c ….. code.c code2.c … … var foo … … line with embedded “ quote Question: What to do about outputs and validation?

A TSL Specification (Unconstrained) Parameters: Pattern size: empty single character many characters Presence of blanks: none one many Presence of file named: file not present file present filename omitted … Environments: Number of occurrences of pattern in file:

A TSL Specification (Constrained) Parameters: Pattern size: empty [property Empty] single character [property NonEmpty] many characters [property NonEmpty] Presence of blanks: none [if NonEmpty] one [if NonEmpty and Quoted] many [if NonEmpty and Quoted] Presence of file named: file not present [error] file present filename omitted [error] … Environments: Number of occurrences of pattern in file: none [if NonEmpty] [single] one [if NonEmpty] [property Match] many [if NonEmpty] [property Match]

A TSL Test Frame Test case 28: (Key = 3.2.3.1.1.2.1) Pattern size : many characters Presence of enclosing quotes : enclosed Presence of blanks : many Presence of quotes within the pattern : none Presence of file named : file present Number of occurrences of pattern : one Number of occurrences of pattern in one line : one

Example of Partitioning for a Search Routine Specification procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : ret BOOLEAN; L: ret ELEM_INDEX) ; Pre-condition -- the array has at least one element T’FIRST <= T’LAST Post-condition -- the element is found and is referenced by L ( Found and T (L) = Key) or -- the element is not in the array ( not Found and not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key )) Element presence Key element in array Key element not in array Array attributes Has a single value Has more than one value Element position First Middle Last

TSL Spec for the Search Routine Specification Element in array? Yes No [single] Array size Single value More than one value [property MultiValue] Element position First Middle [if MultiValue] Last [if MultiValue]

TSL Spec and Tests for the Search Routine Specification Element in array? Yes No [single] 1: Element in array: No 2. Element in array: Yes Array size: Single value Element position: First 3. Element in array: Yes Array size: More than one 4. Element in array: Yes Element position: Middle 5. Element in array: Yes Element position: Last Array size Single value More than one value [property MultiValue] Element position First Middle [if MultiValue] Last [if MultiValue]

Example 2: Flight Status Goto delta.com and click on “FLIGHT STATUS” Specify categories, choices, and constraints to use when testing this simple web form.

For Next Class Download the TSL tool (tsl.tar.gz) from the Handouts page to the CSE machine Untar, unzip the file (see man pages on the “gunzip” and “tar” commands if you aren’t familiar with them) Read the README and tsl-tool-spec.txt files Build the tsl tool as instructed Make sure you can run the tsl tool on the example provided in the tsl materials* Next class: in-class TSL assignment *If, when you try to run the “tsl” excutable, you get a message something like “tsl: Command not found”, you probably need to adjust your “path”, see next page

Setting Your Path On Linux/Unix systems, an environment variable, “$path”, tells the system where to look for executables. This variable is set when you log in, by code in your startup file. By default on CSE this is named “.cshrc” (unless you use a different shell than the default, in which case you probably know how to take care of this, or can use google to find out how). To run an executable that is in your current directory, your path must contain “.”, which denotes “current directory”. Here’s how to add this to your path. Go to your home directory Save the current .cshrc file for safety with “cp .cshrc .cshrc.save” Edit the .cshrc file with any editor Find the line where the variable “$PATH” is set. For instance, it might look something like: “set path = ($path /home/…../bin)” To add “.” to the foregoing path, add the following on the line just after it: “set path = ($path .)” Exit the editor Type “source .cshrc” to cause the system to re-run the .cshrc Now go to the directory where the tsl executable is, and you should be able to invoke it If something goes wrong, go back to your home directory and copy .cshrc.save back into .cshrc, then go seek outside help. Note: the next time you login, the variable will already be set, and you don’t need to repeat any of the foregoing steps.