1. Software Testing and Validation SWE 434
Mutation testing

2. Dynamic Testing
Selection of test data is broadly based on the following techniques:
Functional program testing
Control flow testing
Data flow testing
Questions:
How to assess the effectiveness of the test suite ?
How many bugs are remaining ?

3. Mutation testing
Create a number of mutants, i.e., faulty versions of program
Each mutant contains one fault
Fault created by using mutant operators
Run test on the mutants (random or selected)
When a test case reveals a fault, save test case and remove mutant from the set, i.e., it is killed
Continue until all mutants are killed
Results in a set of test cases with high quality
Need for automation

4. Mutation Testing
Modify a program by introducing a single small change to the code
A modified program is called mutant
A mutant is said to be killed when the execution of test case cause it to fail. The mutant is considered to be dead
Given a mutant m  M for a ground string program P and a test t, t is said to kill m if and only if the output of t on P is different from the output of t on m.

5. Mutation Testing Mutants
int getMax(int x, int y) { int getMax(int x, int y) {
int max; int max;
if (x >y) if (x >=y)
max = x; max = x;
else else
max = y; max = y;
return max; return max;
} }
int getMax(int x, int y) {
int max;
if (x >y)
max = x;
else
return max; }

6. Examples of Mutant Operators
In program
In mutant
Variable replacement
z=x*y+1;
x=x*y+1;
z=x*x+1;
Relational operator replacement
if (x<y)
if(x>y)
if(x<=y)
Off-by-1
z=x*(y+1)+1;
z=(x+1)*y+1;
Replacement by 0
z=0*y+1;
z=0;
Arithmetic operator replacement
z=x*y-1;
z=x+y-1;

7. Mutation Testing
A mutant is an equivalent tot the given program if it always produce the same output as the original program
A mutant is called killable or stubborn, if the existing set of test cases is insufficient to kill it
A mutation score for a set of test cases is the percentage of non-equivalent mutants killed by the test suite
The test suite is said to be mutation-adequate if its mutation score is 100%

8. How mutants survive A mutant may be equivalent to the original program
Maybe changing (x < 0) to (x <= 0) didn’t change the output at all! The seeded “fault” is not really a “fault”.
Determining whether a mutant is equivalent may be easy or hard; in the worst case it is undecideable
Or the test suite could be inadequate
If the mutant could have been killed, but was not, it indicates a weakness in the test suite
But adding a test case for just this mutant is a bad idea. We care about the real bugs, not the fakes!
(c) 2007 Mauro Pezzè & Michal Young

9. Mutation Testing: an example
Consider the following program P:
main(argc,argv)
int argc, r, i;
char *argv[];
{ r = 1;
for i = 2 to 3 do
if (atoi(argv[i-1]) > atoi(argv[r-1])) r = i;
printf(“Value of the rank is %d \n”, r);
exit(0); }
Test Case 1:
input: 1 2 3
output: Value of the rank is 3
Test Case 2:
input: 1 2 1
output: Values of the rank is 2
Test Case 3:
input: 3 1 2
output: Value of the rank is 1

10. Mutation Testing: an example
Test cases for the program P:
Test Case 1:
input: 1 2 3
output: Value of the rank is 3
Test Case 2:
input: 1 2 1
output: Values of the rank is 2
Test Case 3:
input: 3 1 2
output: Value of the rank is 1

11. Mutation Testing: an example
Mutant 1:
main(argc,argv)
int argc, r, i;
char *argv[];
{ r = 1;
for i = 1 to 3 do
if (atoi(argv[i-1]) > atoi(argv[r-1])) r = i;
printf(“Value of the rank is %d \n”, r);
exit(0); }
Test Case 1:
input: 1 2 3
output: Value of the rank is ?
Test Case 2:
input: 1 2 1
output: Values of the rank is ?
Test Case 3:
input: 3 1 2

12. Mutation Testing: an example
Mutant 2:
main(argc,argv)
int argc, r, i;
char *argv[];
{ r = 1;
for i = 2 to 3 do
if (i-1) > atoi(argv[r-1])) r = i;
printf(“Value of the rank is %d \n”, r);
exit(0); }
Test Case 1:
input: 1 2 3
output: Value of the rank is ?
Test Case 2:
input: 1 2 1
output: Values of the rank is ?
Test Case 3:
input: 3 1 2

13. Mutation Testing: an example
Mutant 3:
main(argc,argv)
int argc, r, i;
char *argv[];
{ r = 1;
for i = 2 to 3 do
if (atoi(argv[i-1]) >= atoi(argv[r-1])) r = i;
printf(“Value of the rank is %d \n”, r);
exit(0); }
Test Case 1:
input: 1 2 3
output: Value of the rank is ?
Test Case 2:
input: 1 2 1
output: Values of the rank is ?
Test Case 3:
input: 3 1 2

14. Mutation Testing: an example
main(argc,argv)
int argc, r, i;
char *argv[];
{ r = 1;
for i = 2 to 3 do
if (atoi(argv[r-1]) > atoi(argv[r-1])) r = i;
printf(“Value of the rank is %d \n”, r);
exit(0); }
Test Case 1:
input: 1 2 3
output: Value of the rank is ?
Test Case 2:
input: 1 2 1
output: Values of the rank is ?
Test Case 3:
input: 3 1 2

15. Mutation Testing: an example
Summary:
After Execution of modified programs against the test suite, you will get the results:
Mutants 1 & 3: Programs will pass the test suite, i.e., mutants 1 & 3 are not killable
Mutant 2: Program will fail test cases 2
Mutant 4: Program will fail test case 1 and test cases 2
Mutation score is 50%, assuming mutants 1 & 3 non-equivalent

16. Mutation Testing: an example
Improvements:
The score is found to be low because we assumed mutants 1 & 3 are nonequivalent
We need to show that mutants 1 and 3 are equivalent mutants or those are killable
To show that those are killable, we need to add new test cases to kill these two mutants

17. Mutation Testing: an example
Improvements:
First, let us analyze mutant 1 in order to derive a “killer” test. The difference between P and mutant 1 is the starting point
Mutant 1 starts with i = 1, whereas P starts with i = 2. There is no impact on the result r. Therefore, we conclude that mutant 1 is an equivalent mutant
Second, if we add a fourth test case as follows:
Test Case 4:
input: 2 2 1
Program P will produce the output “Value of the rank is 1” and mutant 3 will produce the output “Value of the rank is 2”
Thus, this test data kills mutant 3, which give us a mutation score 100%

18. Mutation Testing: an example
Recap:
Test Case 1:
input: 1 2 3
output: Value of the rank is 3
Test Case 2:
input: 1 2 1
output: Values of the rank is 2
Test Case 3:
input: 3 1 2
output: Value of the rank is 1
Test Case 4:
input: 2 2 1
which give us a mutation score 100% !!!

19. The Mutation Process Process Mutation Tests New Test Data
If process is not error-free, fix it
Create Mutants
Test Process
Mutation
Tests
New Test Data
Problem with Tests?
Test Mutants
Any Live Mutants?
Yes
Any Mutations that are caught by tests are killed
Test Complete No

20. Mutation testing
Fault Forecasting

21. Fault Forecasting How to determine number of remaining bugs?
Definition:
Fault forecasting is conducted by performing an evaluation of the system behavior with respect to fault occurrence or activation
Fault Injection technique:
The idea is to inject (seed) some faults in the program and calculate the remaining bugs based on detecting the seeded faults [Mills 1972]. Assuming that the probability of detecting the seeded and nonseeded faults are the same.

22. Fault Forecasting Nr Ns Nd ns nd

23. Fault Forecasting Example

24. Mutation testing
Tools

25. Tools for mutation testing
As with any other type of test adequacy assessment, mutation based assessment must be done with the help of a tool.
There are few mutation testing tools available freely. Two such tools are Proteum for C from Professor Maldonado and muJava for Java from Professor Jeff Offutt. We are not aware of any commercially available tool for mutation testing. See the textbook (Aditya) for a more complete listing of mutation tools.
© Aditya P. Mathur 2009

26. Q & A