Software Testing and Validation SWE 434 Mutation testing

Dynamic Testing Selection of test data is broadly based on the following techniques: Functional program testing Control flow testing Data flow testing Questions: How to assess the effectiveness of the test suite ? How many bugs are remaining ?

Mutation testing Create a number of mutants, i.e., faulty versions of program Each mutant contains one fault Fault created by using mutant operators Run test on the mutants (random or selected) When a test case reveals a fault, save test case and remove mutant from the set, i.e., it is killed Continue until all mutants are killed Results in a set of test cases with high quality Need for automation

Mutation Testing Modify a program by introducing a single small change to the code A modified program is called mutant A mutant is said to be killed when the execution of test case cause it to fail. The mutant is considered to be dead Given a mutant m  M for a ground string program P and a test t, t is said to kill m if and only if the output of t on P is different from the output of t on m.

Mutation Testing Mutants int getMax(int x, int y) { int getMax(int x, int y) { int max; int max; if (x >y) if (x >=y) max = x; max = x; else else max = y; max = y; return max; return max; } } int getMax(int x, int y) { int max; if (x >y) max = x; else return max; }

Examples of Mutant Operators In program In mutant Variable replacement z=x*y+1; x=x*y+1; z=x*x+1; Relational operator replacement if (x<y) if(x>y) if(x<=y) Off-by-1 z=x*(y+1)+1; z=(x+1)*y+1; Replacement by 0 z=0*y+1; z=0; Arithmetic operator replacement z=x*y-1; z=x+y-1;

Mutation Testing A mutant is an equivalent tot the given program if it always produce the same output as the original program A mutant is called killable or stubborn, if the existing set of test cases is insufficient to kill it A mutation score for a set of test cases is the percentage of non-equivalent mutants killed by the test suite The test suite is said to be mutation-adequate if its mutation score is 100%

How mutants survive A mutant may be equivalent to the original program Maybe changing (x < 0) to (x <= 0) didn’t change the output at all! The seeded “fault” is not really a “fault”. Determining whether a mutant is equivalent may be easy or hard; in the worst case it is undecideable Or the test suite could be inadequate If the mutant could have been killed, but was not, it indicates a weakness in the test suite But adding a test case for just this mutant is a bad idea. We care about the real bugs, not the fakes! (c) 2007 Mauro Pezzè & Michal Young

Mutation Testing: an example Consider the following program P: main(argc,argv) int argc, r, i; char *argv[]; { r = 1; for i = 2 to 3 do if (atoi(argv[i-1]) > atoi(argv[r-1])) r = i; printf(“Value of the rank is %d \n”, r); exit(0); } Test Case 1: input: 1 2 3 output: Value of the rank is 3 Test Case 2: input: 1 2 1 output: Values of the rank is 2 Test Case 3: input: 3 1 2 output: Value of the rank is 1

Mutation Testing: an example Test cases for the program P: Test Case 1: input: 1 2 3 output: Value of the rank is 3 Test Case 2: input: 1 2 1 output: Values of the rank is 2 Test Case 3: input: 3 1 2 output: Value of the rank is 1

Mutation Testing: an example Mutant 1: main(argc,argv) int argc, r, i; char *argv[]; { r = 1; for i = 1 to 3 do if (atoi(argv[i-1]) > atoi(argv[r-1])) r = i; printf(“Value of the rank is %d \n”, r); exit(0); } Test Case 1: input: 1 2 3 output: Value of the rank is ? Test Case 2: input: 1 2 1 output: Values of the rank is ? Test Case 3: input: 3 1 2

Mutation Testing: an example Mutant 2: main(argc,argv) int argc, r, i; char *argv[]; { r = 1; for i = 2 to 3 do if (i-1) > atoi(argv[r-1])) r = i; printf(“Value of the rank is %d \n”, r); exit(0); } Test Case 1: input: 1 2 3 output: Value of the rank is ? Test Case 2: input: 1 2 1 output: Values of the rank is ? Test Case 3: input: 3 1 2

Mutation Testing: an example Mutant 3: main(argc,argv) int argc, r, i; char *argv[]; { r = 1; for i = 2 to 3 do if (atoi(argv[i-1]) >= atoi(argv[r-1])) r = i; printf(“Value of the rank is %d \n”, r); exit(0); } Test Case 1: input: 1 2 3 output: Value of the rank is ? Test Case 2: input: 1 2 1 output: Values of the rank is ? Test Case 3: input: 3 1 2

Mutation Testing: an example main(argc,argv) int argc, r, i; char *argv[]; { r = 1; for i = 2 to 3 do if (atoi(argv[r-1]) > atoi(argv[r-1])) r = i; printf(“Value of the rank is %d \n”, r); exit(0); } Test Case 1: input: 1 2 3 output: Value of the rank is ? Test Case 2: input: 1 2 1 output: Values of the rank is ? Test Case 3: input: 3 1 2

Mutation Testing: an example Summary: After Execution of modified programs against the test suite, you will get the results: Mutants 1 & 3: Programs will pass the test suite, i.e., mutants 1 & 3 are not killable Mutant 2: Program will fail test cases 2 Mutant 4: Program will fail test case 1 and test cases 2 Mutation score is 50%, assuming mutants 1 & 3 non-equivalent

Mutation Testing: an example Improvements: The score is found to be low because we assumed mutants 1 & 3 are nonequivalent We need to show that mutants 1 and 3 are equivalent mutants or those are killable To show that those are killable, we need to add new test cases to kill these two mutants

Mutation Testing: an example Improvements: First, let us analyze mutant 1 in order to derive a “killer” test. The difference between P and mutant 1 is the starting point Mutant 1 starts with i = 1, whereas P starts with i = 2. There is no impact on the result r. Therefore, we conclude that mutant 1 is an equivalent mutant Second, if we add a fourth test case as follows: Test Case 4: input: 2 2 1 Program P will produce the output “Value of the rank is 1” and mutant 3 will produce the output “Value of the rank is 2” Thus, this test data kills mutant 3, which give us a mutation score 100%

Mutation Testing: an example Recap: Test Case 1: input: 1 2 3 output: Value of the rank is 3 Test Case 2: input: 1 2 1 output: Values of the rank is 2 Test Case 3: input: 3 1 2 output: Value of the rank is 1 Test Case 4: input: 2 2 1 which give us a mutation score 100% !!!

The Mutation Process Process Mutation Tests New Test Data If process is not error-free, fix it Create Mutants Test Process Mutation Tests New Test Data Problem with Tests? Test Mutants Any Live Mutants? Yes Any Mutations that are caught by tests are killed Test Complete No

Mutation testing Fault Forecasting

Fault Forecasting How to determine number of remaining bugs? Definition: Fault forecasting is conducted by performing an evaluation of the system behavior with respect to fault occurrence or activation Fault Injection technique: The idea is to inject (seed) some faults in the program and calculate the remaining bugs based on detecting the seeded faults [Mills 1972]. Assuming that the probability of detecting the seeded and nonseeded faults are the same.

Fault Forecasting Nr Ns Nd ns nd

Fault Forecasting Example

Mutation testing Tools

Tools for mutation testing As with any other type of test adequacy assessment, mutation based assessment must be done with the help of a tool. There are few mutation testing tools available freely. Two such tools are Proteum for C from Professor Maldonado and muJava for Java from Professor Jeff Offutt. We are not aware of any commercially available tool for mutation testing. See the textbook (Aditya) for a more complete listing of mutation tools. © Aditya P. Mathur 2009

Q & A