1. Software Testing
Syntax-based Testing

2. Four Structures for Modeling Software
Syntax Coverage
Four Structures for Modeling Software
Graphs
Logic
Input Space
Syntax
Use cases
Specs
Design
Source
Applied to
DNF
Specs
FSMs
Source
Applied to
Input
Models
Integ
Source
Applied to

3. Applying Syntax-based Testing to Programs
Syntax-based criteria originated with programs and have been used most with programs
BNF criteria are most commonly used to test compilers
Mutation testing criteria are most commonly used for unit testing and integration testing of classes

4. Instantiating Grammar-Based Testing
Program-based
Integration
Model-Based
Input-Based
String mutation
Program mutation
Valid strings
Mutants are not tests
Must kill mutants
Compiler testing
Valid and invalid strings
Grammar
Test how classes interact
Valid strings
Mutants are not tests
Must kill mutants
Includes OO
String mutation
FSMs
Model checking
Valid strings
Traces are tests
String mutation
Input validation testing
XML and others
Valid strings
Grammar
Input validation
testing
XML and others
Invalid strings
No ground strings
Mutants are tests
String mutation
5.2

5. Program-based Grammars
The original and most widely known application of syntax-based testing is to modify programs
Operators modify a ground string (program under test) to create mutant programs
Mutant programs must compile correctly (valid strings)

6. Program-based Grammars
Mutants are not tests, but used to find tests
Once mutants are defined, tests must be found to cause mutants to fail when executed
This is called “killing mutants”

7. Ground string: A string in the grammar
Mutation Testing
Ground string: A string in the grammar
The term “ground” is used as a reference to algebraic ground terms
Mutation Operator : A rule that specifies syntactic variations of strings generated from a grammar
Mutant : The result of one application of a mutation operator

8. Mutant operators: Examples
In P
In mutant
Variable replacement
z=x*y+1;
x=x*y+1;
z=x*x+1;
Relational operator replacement
if (x<y)
if(x>y)
if(x<=y)
Off-by-1
z=x*(y+1)+1;
z=(x+1)*y+1;
Replacement by 0
z=0*y+1;
z=0;
Arithmetic operator replacement
z=x*y-1;
z=x+y-1;

9. What is program mutation?
P’ is known as a mutant of P.
There might be a test t in T such that P(t)≠P’(t). In this case we say that t distinguishes P’ from P. Or, that t has killed P’.
There might be not be any test t in T such that P(t)≠P’(t). In this case we say that T is unable to distinguish P and P’. Hence P’ is considered live in the test process.

10. Killing Mutants
Given a mutant m  M for a ground string program P and a test t, t is said to kill m if and only if the output of t on P is different from the output of t on m.
Testers can keep adding tests until all mutants have been killed
Dead mutant : A test case has killed it
Stillborn mutant : Syntactically illegal
Trivial mutant : Almost every test can kill it
Equivalent mutant : No test can kill it (equivalent to original program)

11. Error detection using mutation
Consider the following function foo that is required to return the sum of two integers x and y.
int foo(int x, y){
return (x-y); }

12. Error detection using mutation
Now suppose that foo has been tested using a test set T that contains two tests:
T={ t1: <x=1, y=0>, t2: <x=-1, y=0>}
First note that foo behaves perfectly fine on each test in, i.e. foo returns the expected value for each test case in T. Also, T is adequate with respect to all control and data flow based test adequacy criteria.

13. Error detection using mutation
Let us evaluate the adequacy of T using mutation. Suppose that the following three mutants are generated from foo.
int foo(int x, y){
return (x+y); }
M1:
return (x-0);
M2:
return (0+y);
M3:
Note that M1 is obtained by replacing the - operator by a + operator, M2 by replacing y by 0, and M3 by replacing x by 0.

14. Error detection using mutation
Next we execute each mutant against tests in T until the mutant is distinguished or we have exhausted all tests. Here is what we get.
T={ t1: <x=1, y=0>, t2: <x=-1, y=0>}
Test (t)
foo(t)
M1(t)
M2(t)
M3(t) t1 1 t2 -1
Live
Killed

15. Questions About Mutation
Should more than one operator be applied at the same time ?
Should a mutated string contain one mutated element or several?
Almost certainly not – multiple mutations can interfere with each other
Extensive experience with program-based mutation indicates not

16. Mutation testing Consider the following program P main(argc,argv)
Handouts
Consider the following program P
main(argc,argv)
int argc, r, i;
char *argv[];
{ r = 1;
for i = 2 to 3 do
if (atoi(argv[i]) > atoi(argv[r])) r = i;
printf(“Value of the rank is %d \n”, r);
exit(0); }

17. Mutation testing Test Case 1: input: 1 2 3
Handouts
Test Case 1:
input: 1 2 3
output: Value of the rank is 3
Test Case 2:
input: 1 2 1
output: Values of the rank is 2
Test Case 3:
input: 3 1 2
output: Value of the rank is 1

18. Mutation testing Mutant 1: Change line 5 to for i = 1 to 3 do
Handouts
Mutant 1: Change line 5 to for i = 1 to 3 do
Mutant 2: Change line 6 to if (i > atoi(argv[r])) r = i;
Mutant 3: Change line 6 to if (atoi(argv[i]) >= atoi(argv[r])) r = i;
Mutant 4: Change line 6 to if (atoi(argv[r]) > atoi(argv[r])) r = i;
Execute modified programs against the test suite, you will get the results:
Mutants 1 & 3: Programs will pass the test suite, i.e., mutants 1 & 3 are not killable
Mutant 2: Program will fail test cases 2
Mutant 1: Program will fail test case 1 and test cases 2
Mutation score is 50%, assuming mutants 1 & 3 non-equivalent

19. Logic Mutation Testing Process
Systematically create program variations (mutants) each containing a single typical logic fault
if (a && b)  if (a || b)
For each mutant, find an input that distinguishes the original program from the mutant program (killing)
- a && b and a || b evaluate to different truth values

20. Program-based Grammars
Original Method
int Min (int A, int B) {
int minVal;
minVal = A;
if (B < A)
minVal = B; }
return (minVal);
} // end Min

21. With Embedded Mutants ∆ 1 minVal = B; ∆ 2 if (B > A)
int Min (int A, int B) {
int minVal;
minVal = A;
∆ 1 minVal = B;
if (B < A)
∆ 2 if (B > A)
∆ 3 if (B < minVal)
minVal = B;
∆ Bomb ();
∆ minVal = A;
∆ minVal = failOnZero (B); }
return (minVal);
} // end Min
Replace one variable with another
Changes operator
Immediate runtime failure … if reached
Immediate runtime failure if B==0 else does nothing

22. Syntax-Based Coverage Criteria
1) Strongly Killing Mutants:
Given a mutant m  M for a program P and a test t, t is said to strongly kill m if and only if the output of t on P is different from the output of t on m
2) Weakly Killing Mutants:
Given a mutant m  M that modifies a location l in a program P, and a test t, t is said to weakly kill m if and only if the state of the execution of P on t is different from the state of the execution of m immediately on t after l

23. Mutant 1 in the Min( ) example is:
Weak Mutation Example
Mutant 1 in the Min( ) example is:
minVal = A;
∆ 1 minVal = B;
if (B < A)
minVal = B;
The complete test specification to kill mutant 1:
(A = 5, B = 3) will weakly kill mutant 1, but not strongly

24. Equivalent Mutation Example
Mutant 3 in the Min() example is equivalent:
minVal = A;
if (B < A)
∆ 3 if (B < minVal)
The infection condition is “(B < A) != (B < minVal)”
However, the previous statement was “minVal = A”
Substituing, we get: “(B < A) != (B < A)”
Thus no input can kill this mutant

25. Strong Versus Weak Mutation
(X = -6) will kill mutant 4 under weak mutation
1 boolean isEven (int X)
2 {
if (X < 0)
X = 0 - X;
∆ X = 0;
if (float) (X/2) == ((float) X) / 2.0
return (true);
else
return (false); }

26. Testing Programs with Mutation
Automated steps
Prog
Input test method
Create mutants
Run equivalence detector
Generate test cases
Run T on P no
Run mutants:
schema-based
weak
selective
Define threshold
Threshold reached ?
Eliminate ineffective TCs
P (T) correct ?
yes
Fix P no

27. Mutation Operators for Java
Each arithmetic expression (and subexpression) is modified by the functions abs(), negAbs(), and failOnZero().
1. ABS –– Absolute Value Insertion:
Each occurrence of one of the arithmetic operators +,－,*,／, and % is replaced by each of the other operators. In addition, each is replaced by the special mutation operators leftOp, and rightOp.
2. AOR –– Arithmetic Operator Replacement:
Each occurrence of one of the relational operators (<, ≤, >, ≥, =, ≠) is replaced by each of the other operators and by falseOp and trueOp.
3. ROR –– Relational Operator Replacement:

28. Mutation Operators for Java (2)
Each occurrence of one of the logical operators (and - &&, or - || , and with no conditional evaluation - &, or with no conditional evaluation - |, not equivalent - ^) is replaced by each of the other operators; in addition, each is replaced by falseOp, trueOp, leftOp, and rightOp.
4. COR –– Conditional Operator Replacement:
5. SOR –– Shift Operator Replacement:
Each occurrence of one of the shift operators <<, >>, and >>> is replaced by each of the other operators. In addition, each is replaced by the special mutation operator leftOp.
Each occurrence of one of the logical operators (bitwise and - &, bitwise or
- |, exclusive or - ^) is replaced by each of the other operators; in addition, each is replaced by leftOp and rightOp.
6. LOR –– Logical Operator Replacement:

29. Mutation Operators for Java (3)
Each occurrence of one of the assignment operators (+=, -=, *=, /=, %=, &=, |=, ^=, <<=, >>=, >>>=) is replaced by each of the other operators.
7. ASR –– Assignment Operator Replacement:
8. UOI –– Unary Operator Insertion:
Each unary operator (arithmetic +, arithmetic -, conditional !, logical ~) is inserted in front of each expression of the correct type.
Each unary operator (arithmetic +, arithmetic -, conditional !, logical~) is deleted.
9. UOD –– Unary Operator Deletion:

30. Mutation Operators for Java (4)
Each variable reference is replaced by every other variable of the appropriate type that is declared in the current scope.
10. SVR –– Scalar Variable Replacement:

31. Binary search example int search ( int key, int [] elemArray){
int bottom = 0;
int top = elemArray.length - 1;
int mid;
int result = -1;
while ( bottom <= top ) {
mid = (top + bottom) / 2;
if (elemArray [mid] == key){
result = mid;
return result; } // if part
else {
if (elemArray [mid] < key)
bottom = mid + 1;
else
top = mid - 1; }
} //while loop
return result;}