1. Test Drivers and Stubs More Unit Testing Test Drivers and Stubs CEN 5076 Class 11 – 11/14

2. CEN 5076 Class 1 - 11/1742 More Unit Testing Fault-based adequacy criteria measure the quality of a test set according to its effectiveness or ability to detect faults. 1.Error seeding 2.Program mutation testing 3.Specification mutation testing 4.Perturbation Testing [Zhu et al. ’97]

3. CEN 5076 Class 1 - 11/1743 More Unit Testing 1. Error Seeding A technique originally proposed to estimate the number of faults that remain in software. Artificial faults are introduced into the program under test in some suitable random fashion unknown to the tester. Assumption: artificial faults are representative of the inherent faults in the program in terms of difficulty of detection.

4. CEN 5076 Class 1 - 11/1744 More Unit Testing 1. Error seeding cont Let r be the ratio of the # of artificial faults found to the number of total artificial faults. Number of inherent faults in the program is statistically predicted with maximum likelihood to be f/r, where f is the number of inherent faults found by testing. Technique can also be used to measure the quality of software testing.

5. CEN 5076 Class 1 - 11/1745 More Unit Testing 1. Error seeding cont Advantages: –Not restricted to measuring test quality fro dynamic testing, i.e., applicable to any testing method that aims at finding errors or faults in the s/w. Disadvantage: –The accuracy of the measure is dependent on how faults are introduced. –Artificial faults are manually planted, but it difficult to implement error seeding in practice.

6. CEN 5076 Class 1 - 11/1746 More Unit Testing 2. Program Mutation Testing Measures the degree to which a program has been tested i.e., measures test case adequacy. Given a program p and a set of test cases t: Step 1: construct a set of native programs (mutants) that differ from the original program in some fashion Step 2: Each mutant is executed on each member in the test set t, stopping either when an element of t is found on which p and the mutant program produce different responses (mutant killed) or when t is exhausted (mutant alive)

7. CEN 5076 Class 1 - 11/1747 More Unit Testing 2. Program Mutation Testing cont Mutants are created by syntatic changes in the program. A mutant remains alive for one of the following: 1.The test data are inadequate. 2.The mutant is equivalent to the original program. Based on two basic assumption: 1.Competent programmer assumption. 2.Coupling effect assumption – assumes that simple and complex errors are coupled.

8. CEN 5076 Class 1 - 11/1748 More Unit Testing 2. Program Mutation Testing cont Advantages: –Allows a great deal of automation. –Provides an interactive test environment that allows the tester to locate and remove errors. –Includes other testing methods as special cases. Disadvantages: –Requires a large amount of computation resources (time and space) to test large-scale software. Mutants for n-line program is of order n 2.

9. CEN 5076 Class 1 - 11/1749 More Unit Testing 3. Specification-Mutation Testing Attempts to detect faults in the implementation that are derived from misinterpreting the specification or the faults in the spec. Steps: 1.Faults are planted in the spec. 2.The program that implements the original program is executed and the results of the execution are checked against the original spec. and those with planted faults. 3.Adequacy based on the number of faults detected.

10. CEN 5076 Class 1 - 11/17410 More Unit Testing 3. Perturbation Testing Analyzes test effectiveness by considering the faults in an “error” space. [Zeil ‘83] Tries to identify the functional differences between the program under test and the hypothetical correct program. Adequacy of a test set is decided by its ability to limit the error space defined in terms of a set of functions. Theory of approach is based on vector spaces.

11. CEN 5076 Class 1 - 11/17411 More Unit Testing Error-based adequacy criteria & domain analysis Error-based testing methods require test cases to check programs on certain error-prone points. Domain analysis and domain testing is to partition the I/O behavior space into equivalent subdomains w.r.t. the behavior of the software. 1.Domain Testing 2.Partition analysis 3.Equivalence partitioning 4.Boundary value analysis

12. CEN 5076 Class 1 - 11/17412 More Unit Testing 1.Domain testing Idea is to consider the path predicates in the program and see which values from the input space will cause the path to be executed. Chop the domain into a number of regions and all data chosen from one region will cause the same path to be executed. The testing strategy then involves examining each of the borders of each region in detail and selecting test data near the border.

13. CEN 5076 Class 1 - 11/17413 More Unit Testing 2. Partition analysis Related to domain testing, takes into account the spec as well as the program. One of the few techniques that integrates formally spec-based and impl-based testing. Relies on performing a symbolic execution of the spec and program. Symbolic execution or symbolic evaluation is performed by letting variables take on symbolic values and then executing the program by hand.

14. CEN 5076 Class 1 - 11/17414 More Unit Testing 2. Partition analysis cont Sym. exe. generates an equation in terms of the input variables which, if satisfied, will cause the symbolically executed path to be executed. Path domains and path computations are calculated for the spec and impl. The next step is to create a procedure partition i.e., overlay the spec partition on the impl partitions. If there is 1-1 match between a spec partition and an impl partition then the data in both partitions can be treated equally.

15. CEN 5076 Class 1 - 11/17415 More Unit Testing 3. Equivalence partitioning Technique is based on the spec only. Divide up the input domain into equivalence partitions or classes of data which, according to the spec are treated identically. Basis of technique is that any datum chosen from an equivalence class is as valid as any other since it should be processed in a similar fashion. Criteria used to determine equivalence classes: coverage, disjointedness, representation.

16. CEN 5076 Class 1 - 11/17416 More Unit Testing Example: Valid inputs to test the getNumberDaysInMonth() method Equivalence classValue for month inputValue for year input Months with 31 days, non-leap yrs. 7 (July)1901 Months with 31 days, leap yrs.7 (July)1904 Months with 30 days, non-leap yrs. 6 (June)1901 Months with 30 days, leap yrs.6 (June)1904 Months with 28 or 29 days, non-leap yrs. 2 (February)1901 Months with 28 or 29 days, leap yrs. 2 (February)1904

17. CEN 5076 Class 1 - 11/17417 More Unit Testing 4. Boundary value analysis Is used in conjunction with equivalence partitioning in that it focuses on a source of faults - the boundaries of an equivalence partition. Test data derived by examining the edges of an equivalence partition. White et al. ’80 proposed the N x 1 domain- testing strategy that requires N test cases to be selected on the borders in the N-dimensional space and one test cases just off the border. There is a N x N that is stricter than N x 1.