1. 1 Automating the Generation of Mutation Tests Mike Papadakis and Nicos Malevris Department of Informatics Athens University of Economics and Business

2. Test Data Generation Approaches Symbolic execution Select set of paths Produce a system of algebraic constraints Solve and generate test cases Challenges (Symbolic execution) Infeasible paths Complex-unhandled expressions Availability of source code 2

3. Test Data Generation Approaches Search based approaches Definition of program input Dynamic program execution Fitness function o (efficiency and effectiveness) Challenges Handling of dynamic program inputs Require a high number of executions Handling of specific cases (e.g. flag problem) Effective fitness function 3

4. Test Data Generation Approaches Dynamic Symbolic Execution Simultaneously perform actual and symbolic execution o Simplify the process based on concrete program execution Produce a system of algebraic constraints Solve and generate test cases Challenges Infeasible paths Scalability issues 4

5. 5 In order to kill a mutant, tests must Stage 1: Reach the mutant Stage 2: infect the program state Stage 3: propagate the infected state Killing Mutants

6. 6 Reach the mutant (Reachability condition) Infect program state (necessity condition) Program execution must cause a discrepancy in the program state E.g. Orig: a > b, Mut = a b a > b a b o a == b where Orig = false, Mut = true Propagate (sufficiency condition) Heuristic approaches o explore path space, measure of mutants impact Joint satisfaction (Reach && Infect && Propagate) Killing Mutants

7. 7 Killing Mutants (path selection) Symbolic Execution Reach o Use paths from program input to mutant Infect o Use necessity constraints to infect program state o Original Statement Mutated Statement Propagate o Explore the path space (from mutant to program output) Path selection Strategy

8. 8 Enhanced Test Model (Enhanced Control Flow Graph) Represent mutants in the model Guide the selection of paths and the produced constraints based on the Shortest path strategy Killing Mutants (path selection)

9. 9 Shortest Path Strategy, the k-value An efficient path selection strategy. (eliminates the effects of infeasible paths) Select shortest paths from program input (S) to the mutant statement (m) Heuristic approach to weak mutation Strong mutation can be tackled incrementally

10. 10 Killing Mutants Dynamic approaches Dynamic approaches Gain the required information from actual program execution Require high number of program execution Mutation Testing Need to unify the runtime information of the original and mutant programs Need to include mutant neccesity and sufficiency conditions into program structure Need for efficiency (high cost)

11. 11 Mutant Schemata Mutant schemata-Parameterized mutants Technique for efficiently producing mutants Embeds all mutants into one meta-mutant program Use of global parameter to specify the mutants Example original meta-mutant a > b + c RelationalGT(a, ArithmeticPlus(M(b), M(c))) M(x) mutates variable x, e.g. abs(x) Arithmetic(x, y) mutates arithmetic operator, e.g. x - y Relational(x, y) mutates relational operator, e.g. x y

12. 12 Mutant Schemata (approach I) Weak mutation Include the mutant evaluation inside schematic functions o Original expression Mutant expression o Original statement Mutant statement After evaluation continue with the original execution (schematic function returns the original expression) Able to execute all mutants with one execution run Embeds all mutants killable condition into program structure (mutant evaluation branches)

13. Mutant is killed Mutant is Alive Mutant Schemata Proposed approach (Mutants to branches) Mutants M[1]…M[n] on node N. Mutant Evaluation Branches 13

14. 14 Mutant Schemata (approach II) Strong mutation In Same lines as Weak o Include the mutant evaluation inside schematic functions After evaluation continue with the mutant execution (schematic function returns the mutant expression) Incremental approach, from weak to strong Requires multiple execution runs Record original and mutants execution paths Evaluate strongly killed mutants

15. 15 Killing Mutants (DSE) Use the shortest path heuristic Negate the condition that will lead closer to mutant node (reach the mutant) Weakly kill the mutant Fulfill the mutant necessity condition Make the mutant evaluation Branch true Negate the produced conditions after the mutant Start from the mutant program Explores the mutant program path space

16. 16 Target Mutant DSE-Example Scenario

17. 17 Target Mutant DSE-Example Scenario

18. 18 Target Mutant DSE-Example Scenario

19. 19 Target Mutant DSE-Example Scenario

20. 20 Target Mutant Mutant is alive Infect Mutant DSE-Example Scenario

21. 21 Target Mutant Mutant is alive Propagate DSE-Example Scenario

22. 22 Target Mutant Mutant is alive DSE-Example Scenario

23. 23 Target Mutant Mutant Killed DSE-Example Scenario

24. 24 DSE-Necessity condition Example: a + b > k a + abs(b) > k RelationalGT(ArithmeticPlus(a, Abs(b)), k) Abs(x) if (x <0 ) //mutant necessity condition a + b > k a + abs(b) > k //Statement level condition Test: a=15, b=2, k=0 Conditions: b0 && a+b>k && … Negates: b0 b < 0 Produces: a=15, b=-10, k=0

25. 25 DSE-Necessity condition Example: a + b > k a + abs(b) > k RelationalGT(ArithmeticPlus(a, Abs(b)), k) Abs(x) if (x <0 ) //mutant necessity condition a + b > k a + abs(b) > k //Statement level condition Test: a=15, b=-10, k=0 Conditions: b k && a+abs(b)>k && … Negates: a+abs(b)>k b k && a+abs(b)k Infeasible Negates: a+b>k b<0 && a+bk Produces: a=-15, b=-10, k=0

26. 26 DSE-Necessity condition Example: a + b > k a + abs(b) > k RelationalGT(ArithmeticPlus(a, Abs(b)), k) Abs(x) if (x <0 ) //mutant necessity condition a + b > k a + abs(b) > k //Statement level condition Test: a=-15, b=-10, k=0 Conditions: b<0 && a+bk && a+abs(b)k && … Negates: a+abs(b)>k b k Produces: a=5, b=-10, k=0 Mutant infected: Orig: 5-10>0 (false), Mut: 5+10>0 (true)

27. 27 Killing Mutants (Search Based) Weak-Strong mutation Measure the closeness of reaching a mutant Measure branch distance of mutant branches o Closeness of weakly killing the targeted mutant (mutant necessity condition) Use simplified necessity fitness for improved performance Sufficiency condition can be approximated by exploring the path space or based on mutants impact

28. 28 Fitness function Approach level Closeness of executing a targeted mutant Calculated by the number of control dependent nodes missed Branch Distance Closeness of flipping a specific branch Mutation Distance Closeness of weakly killing the targeted mutant

29. 29 Mutant Fitness (example) OperatorOriginal expressionMutant Fitness Relationala > b a >= b: abs(a-b) a < b: k a <= b: 0 a != b: abs(a-b+k) a == b: abs(a-b) true: abs(a-b) false: abs(a-b+k) Arithmetica + b a - b:k a * b:k a / b:k a % b:k a:k b:k Absolutea abs(a):abs(a+k)-abs (a):abs(a) 0:abs(a) Logical a && b a||b:min[Tfit(a)+Ffit(b), Ffit(a)+Tfit(b)] a:Tfit(a)+Ffit(b) b:Ffit(a)+Tfit(b) true:min [Ffit(a), Ffit(b)] false:Tfit(a)+Tfit(b) a || b a&&b:min[Tfit(a)+Ffit( b), Ffit(a)+Tfit(b)] a:Ffit(a)+Tfit(b) b:Tfit(a)+Ffit(b) true:Ffit(a)+Ffit(b) false:min[Tfit(a), Tfit(b)]

30. 30 Path explosion problem Original program 8 program paths Mutant program 8 program paths per mutant Schematic program Mutant evaluation decisions 8 * 2 number of mutants paths Introduction of fake paths Same of them may be useful for higher order mutation Mutants

31. 31 Handling of path explosion Path selection strategy, (symbolic execution) Use weights on the Enhanced Control Flow Graph Shortest paths include only the original paths Dynamic Symbolic Execution Use the mutant parameters as local values Dynamic production of constraints eliminates the fake paths. Paths are produced dynamically Search based approaches Approach level contains only the control depended nodes o Ignores the path space

32. 32 Case Studies Dynamic Symbolic Execution (Strong mutation) Program units: Tritype, Remainder Siemens Programs: Schedule, Tcas, Replace ABS, AOR and ROR operators Search based (Weak mutation results) Hill climbing approach (AVM) Maximum 10 attempts per mutant Program units: Tritype, Triangle, Remainder, Calendar ABS, AOR, ROR and LCR operators

33. DSE Study-Results

38. Program ½ IterationsAll Iterations Produced Tests Killed Mutants Solver Calls No. Mut. Executions Killed Mutants Solver Calls No. Mut. Executions Tritype21410125002161588514 90 Remainder23427494352355498741268 Replace51414740194005203023880471 8927 Tcas1361733372913737054720 422 Schedule9471429394949303 301 DSE Study-Results Strong mutation results Replace 86.5%, Tcas 100%, Schedule 91% (compared with the accompanied test suite), Remainder 97.5 and Tritype 96.8% (all killable mutants)

39. 39 Search Based Study-Results

40. 40 Search Based Study-Results

41. 41 Search Based Study-Results

42. 42 Search Based Study-Results

43. 43 Search Based Study-Results Test Objects No. of Killed MutantsMutation Score Time (Sec) RandomMetallaxisRandomMetallaxis Tritype16824566.4%96.8%42 Triangle24627589.5%100%70 Remainder23824298.3%100%402 Cal23725293.3%99.2%44 Weak mutation results

44. Conclusion Mutation based test case generation Lack of attempts First steps using dynamic, state of the art techniques Dynamic approaches can be adopted to perform mutation Search based approaches o Mutation distance leads to improved effectiveness Dynamic Symbolic Execution o Quite effective 44

45. Conclusion Test data generation approaches and tools can be effectively extended to mutation Practical Technique (schemata) Future directions Efficient handling of infeasible paths Handling of the flag problem Efficient handling of equivalent mutants Hybrid approaches 45

46. Thank you for your attention… Questions ? Contact Mike Papadakis mpapad@aueb.gr Nicos Malevris ngm@aueb.gr 46

47. 47 References Mike Papadakis and Nicos Malevris. "Automatic Mutation Test Case Generation Via Dynamic Symbolic Execution", in 21st International Symposium on Software Reliability Engineering (ISSRE'10), San Jose, California, USA, November 2010. Mike Papadakis and Nicos Malevris. Metallaxis an Automated Framework for Weak Mutation", Technical Report, http://pages.cs.aueb.gr/~mpapad/TR/MetallaxisTR.pdf. Mike Papadakis, Nicos Malevris and Maria Kallia. "Towards Automating the Generation of Mutation Tests", in Proceedings of the 5th International Workshop on Automation of Software Test (AST'10), Cape Town, South Africa, May 2010, pp. 111-118. Mike Papadakis and Nicos Malevris. An Effective Path Selection Strategy for Mutation Testing", in Proceedings of the 16th Asia-Pacific Software Engineering Conference (APSEC'09), Penang, Malaysia, December 2009, pp. 422-429.