1. Introduction to Software Testing

2. Introduction 2 Four Structures for Modeling Software Graphs Logic Input Space Syntax Use cases Specs Design Source Applied to DNF Specs FSMs Source Input Models Integ Source

3. Input Space Partitioning 3 All testing is choosing elements from the input space of the software being tested. Criteria can be viewed as defining way to divide the input space according to the test requirements. input domain input domain  Possible values that the input parameters can have.  The input domain is partitioned into regions that are assumed to contain equally useful values from a testing perspective, and values are selected from each region.  The tester doesn't need to understand the implementation.  Everything is based on a description of the inputs.

4. Input Space Partitioning 4 Block Block  Consider an abstract partition q over some domain D. The partition q defines a set of equivalence classes, which we simply call blocks, Bq. The blocks are pair wise disjoint, that is: and together the blocks cover the domain D, that is: b i  b j = ,  i  j, b i, b j  B q  b = D b  Bq

5. Input Space Partitioning 5 Example: Partitioning of input domain D into three blocks: A common way to apply input space partitioning is to start by considering the domain of each parameter separately.

6. Input Space Partitioning 6 Input space modeling Input space modeling  Sometimes the parameters are considered in conjunction, usually by taking the semantics of the program into account. characteristics characteristics  Each partition is usually based on some characteristic C of the program, the program’s inputs, or the program’s environment. Some possible characteristic examples:  Input X is null  Order of file F (sorted, inverse sorted, arbitrary)  Min separation distance of two aircraft

7. Input Space Partitioning 7 partition partition  Each characteristic C allows the tester to define a partition. A partition must satisfy two properties:  The partition must cover the entire domain (completeness).  The blocks must not overlap (disjoint).  Example: "Order of file F" could be used to create the following partitioning:  b1 = Sorted in ascending order  b2 = Sorted in descending order  b3 = Arbitrary order  But this is not a valid partitioning, specifically if the file is of length 0 or 1, then the file will belong in all three blocks.

8. Input Space Partitioning 8 Each characteristic should addresses only one property:  Splitting into two characteristics, namely sorted ascending and sorted descending, solves the problem:  File F sorted ascending b1 = True b2 = False  File F sorted descending b1 = True b2 = False

9. Input Domain Modeling 9 If the partitions are not complete or disjoint, that means the partitions have not been considered carefully enough They should be reviewed carefully, like any design attempt. Different alternatives should be considered. We model the input domain in five steps …

10. Input Domain Modeling 10 Step 1 : Identify testable functions: Individual methods have one testable function. In a class, each method often has the same characteristics. Programs have more complicated characteristics: modeling documents such as UML use cases can be used to design characteristics. Systems of integrated hardware and software components can use devices, operating systems, hardware platforms, browsers, etc.

11. Input Domain Modeling 11 Step 2 : Find all the parameters: Often fairly straightforward, even mechanical Important to be complete Methods : Parameters and state (non-local) variables used. Components : Parameters to methods and state variables. System : All inputs, including files and databases.

12. Input Domain Modeling 12 Step 3 : Model the input domain: The domain is scoped by the parameters. The structure is defined in terms of characteristics. Each characteristic is partitioned into sets of blocks. Each block represents a set of values. This is the most creative design step in applying ISP.

13. Input Domain Modeling 13 Step 4 : Apply a test criterion to choose combinations of values: A test input has a value for each parameter One block for each characteristic Choosing all combinations is usually infeasible Coverage criteria allow subsets to be chosen Step 5 : Refine combinations of blocks into test inputs: Choose appropriate values from each block

14. Input Domain Modeling 14 Two Approaches to Input Domain Modeling: 1. Interface-based approach: Develops characteristics directly from individual input parameters. Simplest application Can be partially automated in some situations. 2. Functionality-based approach: Develops characteristics from a behavioral view of the program under test. Harder to develop: requires more design effort. May result in better tests, or fewer tests that are as effective.

15. Interface-Based Input Domain Modeling 15 Mechanically consider each parameter in isolation. This is an easy modeling technique and relies mostly on syntax. Some domain and semantic information won’t be used  Could lead to an incomplete IDM. Ignores relationships among parameters:  Some parts of the functionality may depend on combinations of specific values of several interface parameters.

16. Functionality-Based Input Domain Modeling 16 Identify characteristics that correspond to the intended functionality of the system under test, rather than using the actual interface. Requires more design effort from tester. Can incorporate domain and semantic knowledge. Can use relationships among parameters. Modeling can be based on requirements, not implementation. The same parameter may appear in multiple characteristics, so it’s harder to translate values to test cases.

17. Identifying characteristics 17 Identifying characteristics in an interface-based approach is simple:  a mechanical translation from the parameters to characteristics. Developing a functionality-based IDM is more challenging. Candidates for characteristics : Preconditions and Postconditions, Relationships among variables, Relationship of variables with special values (zero, null, blank, …).

18. Identifying characteristics 18 Should not use program source: characteristics should be based on the input domain. domain knowledge use domain knowledge  use specifications or other documentation instead of program code (implementation) to develop characteristics. Program source should be used with graph or logic criteria. Better to have more characteristics with few blocks: Fewer mistakes and fewer tests.

19. Identifying characteristics 19 Example: Interface vs. Functionality-Based: public boolean findElement (List list, Object element) // Effects: if list or element is null throw NullPointerException // else return true if element is in the list, false otherwise Interface-Based Approach Two parameters : list, element Characteristics : list is null (block1 = true, block2 = false) list is empty (block1 = true, block2 = false) Functionality-Based Approach Two parameters : list, element Characteristics : number of occurrences of element in list (block1 : 0, block2 : 1, block3 : >1) element occurs first in list (block1 : true, block2 : false) element occurs last in list (block1 : true, block2 : false)

20. Choosing Blocks and Values 20 After choosing characteristics, the test engineer partitions the domains of the characteristics into sets of values called blocks:  how partitions should be identified?  how representative values should be selected from each block? More blocks means more tests. The partitioning often flows directly from the definition of characteristics and both steps are sometimes done together. Should evaluate them separately: sometimes fewer characteristics can be used with more blocks and vice versa.

21. Choosing Blocks and Values 21 Strategies for identifying values : Include valid, invalid and special values Sub-partition some blocks:]  A range of valid values can often be partitioned into subpartitions. Explore boundaries of domains Include values that represent “normal use” Try to balance the number of blocks in each characteristic Check for completeness and disjointness:  Check that the union of all blocks of a characteristic completely covers the input space of that characteristic.  Check that no value belongs to more than one block.

22. Choosing Blocks and Values 22 TriTyp Example : TriTyp have one testable function and three integer inputs ( the lengths of three sides of a triangle).  Interface-Based IDM: A maximum of 3*3*3 = 27 tests Some triangles are valid, some are invalid Refining the characterization can lead to more tests … First Characterization of TriTyp’s Inputs Characteristicb1b1 b2b2 b3b3 q 1 = “Relation of Side 1 to 0”greater than 0equal to 0less than 0 q 2 = “Relation of Side 2 to 0”greater than 0equal to 0less than 0 q 3 = “Relation of Side 3 to 0”greater than 0equal to 0less than 0

23. Choosing Blocks and Values 23  Interface-Based IDM (cont.): A maximum of 4*4*4 = 64 tests This is only complete because the inputs are integers (0.. 1) Second Characterization of TriTyp’s Inputs Characteristicb1b1 b2b2 b3b3 b4b4 q 1 = “Refinement of q 1 ”greater than 1equal to 1equal to 0less than 0 q 2 = “Refinement of q 2 ”greater than 1equal to 1equal to 0less than 0 q 3 = “Refinement of q 3 ”greater than 1equal to 1equal to 0less than 0

24. Choosing Blocks and Values 24  Interface-Based IDM (cont.): The above partitioning is interface based and only uses syntactic information about the program (it has three integer inputs). A functionality-based approach can use the semantic information of the traditional geometric classification of triangles. Possible values for partition q 1 Characteristicb1b1 b2b2 b3b3 b4b4 Side110 2 Test boundary conditions

25. Choosing Blocks and Values 25  Functionality-Based IDM: First two characterizations are based on syntax– parameters and their type A semantic level characterization could use the fact that the three integers represent a triangle Geometric Characterization of TriTyp’s Inputs Characteristicb1b1 b2b2 b3b3 b4b4 q 1 = “Geometric Classification”scaleneisoscelesequilateralinvalid

26. Choosing Blocks and Values 26  Functionality-Based IDM (cont.): equilateral is also isosceles ! We need to refine the example to make characteristics valid Characteristicb1b1 b2b2 b3b3 b4b4 q 1 = “Geometric Classification”scaleneisosceles, not equilateral equilateralinvalid Correct Geometric Characterization of TriTyp’s Inputs

27. Choosing Blocks and Values 27  Functionality-Based IDM (cont.): Values for this partitioning can be chosen as: Possible values for geometric partition q 1 Characteristicb1b1 b2b2 b3b3 b4b4 Triangle(4, 5, 6)(3, 3, 4)(3, 3, 3)(3, 4, 8)

28. Choosing Blocks and Values 28  Functionality-Based IDM (cont.): A different approach would be to break the geometric characterization into four separate characteristics Use constraints to ensure that:  Equilateral = True implies Isosceles = True  Valid = False implies Scalene = Isosceles = Equilateral = False Four Characteristics for TriTyp Characteristicb1b1 b2b2 q 1 = “Scalene”TrueFalse q 2 = “Isosceles”TrueFalse q 3 = “Equilateral”TrueFalse q 4 = “Valid”TrueFalse

29. Using More than One Input Domain Model 29 Some programs may have dozens or even hundreds of parameters. Create several small IDMs: A divide-and-conquer approach Different parts of the software can be tested with different amounts of rigor. For example, some IDMs may include a lot of invalid values. It is okay if the different IDMs overlap. The same variable may appear in more than one IDM.

30. Checking the Input Domain Model 30 In terms of characteristics, the test engineer should ask whether there is any information about how the function behaves that is not incorporated in some characteristics. The tester should also explicitly check each characteristic for the completeness and disjointness properties. If multiple IDMs are used, completeness should be relative to the portion of the input domain that is modeled in each IDM.

31. Combining Strategies Criteria 31 “How should we consider multiple partitions at the same time?” “What combination of blocks should we choose values from?” All Combinations Coverage (ACoC) All Combinations Coverage (ACoC)  All combinations of blocks from all characteristics must be used. This is almost certainly more testing than is necessary, and will usually be economically impractical as well. The first, fundamental assumption is that different choices of values from the same block are equivalent from a testing perspective.

32. Combining Strategies Criteria 32 Several combination strategies exist. The first combination strategy criterion is fairly straightforward and simply requires that we try each choice at least once. Each Choice Coverage (ECC): Each Choice Coverage (ECC):  One value from each block for each characteristic must be used in at least one test case. Assume the program under test has Q parameters q1, q2,..., qQ, and each parameter qi has Bi blocks. Then a test suite that satisfies ECC will have at least values.

33. Combining Strategies Criteria 33 ECC leaves a lot of flexibility to the tester in terms of how to combine the test values, so it can be called a relatively “weak” criterion. A natural next step is to require explicit combinations of values, called pair-wise. Pair-Wise Coverage (PWC): Pair-Wise Coverage (PWC):  A value from each block for each characteristic must be combined with a value from every block for each other characteristic. A test suite that satisfies PWC will pair each value with each other value or have at least values.

34. Combining Strategies Criteria 34 A natural extension to PWC is to require t values instead of pairs. T-Wise Coverage (TWC): T-Wise Coverage (TWC):  A value from each block for each group of t characteristics must be combined. Both PWC and TWC combine values “blindly,” without regard for which values are being combined. Base Choice Coverage (BCC): Base Choice Coverage (BCC):  A base choice block is chosen for each characteristic, and a base test is formed by using the base choice for each characteristic.  Subsequent tests are chosen by holding all but one base choice constant and using each non-base choice in each other characteristic.

35. Combining Strategies Criteria 35 A test suite that satisfies BCC will have one base test, plus one test for each remaining block for each partition. This is a total of : The base choice can be the simplest, the smallest, the first in some ordering, or the most likely from an end- user point of view. Sometimes the tester may have trouble choosing a single base choice and may decide that multiple base choices are needed.

36. Combining Strategies Criteria 36 Multiple Base Choices (MBCC): Multiple Base Choices (MBCC):  At least one, and possibly more, base choice blocks are chosen for each characteristic, and base tests are formed by using each base choice for each characteristic at least once.  Subsequent tests are chosen by holding all but one base choice constant for each base test and using each non- base choice in each other characteristic. Assuming mi base choices for each characteristic and a total of M base tests, MBCC requires test

37. Combining Strategies Criteria 37 The subsumption relationships among the input space partitioning combination strategy criteria: