Mutation Testing Meets Approximate Computing Milos Gligoric1, Sarfraz Khurshid1, Sasa Misailovic2, August Shi2 ICSE NIER 2017 Buenos Aires, Argentina May 24, 2017 “My name is August Shi. I am from the University of Illinois at Urbana-Champaign. I will be presenting our work Mutation Testing Meets Approximate Computing. This is joint work with Milos and Sarfraz, who are from the University of Texas at Austin, and with Sasa, who is also from the University of Illinois at Urbana-Champaign. Our work involves both mutation testing and approximate computing, so I will start with a bit of background on both.” 1 2 CCF-1409423, CCF-1421503, CCF-1566363, CCF-1629431, CCF-1319688, CNS-1239498

Mutation Testing Goal: Evaluate quality of test suites How: Apply transformations (mutation operators) on the code and run tests to see if they can detect the changes Example: x = x + 1 “Mutation testing is a technique used for many years, particularly in software testing research, to evaluate quality of test suites” Say transformations = mutation operators

Mutation Testing Goal: Evaluate quality of test suites How: Apply transformations (mutation operators) on the code and run tests to see if they can detect the changes Example: x = x + 2 Problems: Evaluation of quality limited by mutation operators Too slow “Mutation testing is a traditional technique used in research to evaluate quality of test suites” Say transformations = mutation operators

Approximate Computing Goal: Improve performance of code How: Apply transformations that may lead to (slightly) inaccurate results Example: for (i = 0; i < n; i = i + 1) “Approximate computing is a new, emerging area of research, especially in the programming systems community, that does something similar, but has a different goal” Point out how transforming code is a common factor in the “how” part between the two

Approximate Computing Goal: Improve performance of code How: Apply transformations that may lead to (slightly) inaccurate results Example: for (i = 0; i < n; i = i + 2) Problems: Not sure where in exact code to apply approximations Unclear how to check quality of tests on already approximate code “Approximate computing is a new, up-and-coming area of research that does something similar, but has a different goal” Goal of performance, e.g., speed (tie in with later part) Point out how transforming code is a common factor in the “how” part between the two

How can Mutation Testing and Approximate Computing improve one another? “More specifically, can we use one to mitigate the problems of the other?”

Improving One Another Approximate computing to provide new mutation operators for evaluating quality of tests Approximate computing to improve speed of mutation testing Mutation testing to point out opportunities for applying approximations on exact code Mutation testing to evaluate quality of tests on (already) approximate code

Example Code: Commons-Math // MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 0; i < data.length; i++) { 0 values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 0; while (iterator.hasNext()) { out[i++] = iterator.next(); return out; // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); } Test Passes “I would like to demonstrate some of these ideas using an example…”

Mutant: Constant Replacement // MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 0; i < data.length; i++) { values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 1; while (iterator.hasNext()) { out[i++] = iterator.next(); return out; // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); } Test Fails Mutant Killed Replace 0 with 1

Mutant: Constant Replacement // MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 1; i < data.length; i++) { 0 values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 0; while (iterator.hasNext()) { out[i++] = iterator.next(); return out; // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); } Test Passes Mutant Survived Replace 0 with 1

Approximate: Loop Perforation // MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 0; i < data.length; i+=2) values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 0; while (iterator.hasNext()) { out[i++] = iterator.next(); return out; // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); } Test Fails Normally, this transformation would cause the test to fail In such a case, the approximation is invalid. So a developer has two options: the developer can do a different approximation, or if the developer has determined the approximation is good and it’s the test that’s too rigid, the developer can modify the test to accept the approximation Skip every other iteration 68% of runtime is in this loop

Approximate: Loop Perforation Modify assertion Approximate: Loop Perforation // MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 0; i < data.length; i+=2) values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 0; while (iterator.hasNext()) { out[i++] = iterator.next(); return out; // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArraySubset(values, MathArrays.unique(x), 0); } Test Passes For the sake of our example, let’s say the developer chooses to modify the test assertion to allow for this approximation Let’s say there is this new assertion that checks if elements are a subset of another (though assertion does not actually exist) Because in approximate computing, we need to have the test pass Approximation is Acceptable Skip every other iteration 68% of runtime is in this loop

Comparison of Transformation Results Mutation Testing Approximate Computing Failing Test While we see that both mutation testing and approximate computing do transformations, they expect different things from the results of tests due to transformations There seems to be a conflict between expectations of if the test passes or fails after transformations based on which mindset a developer is in Passing Test

Comparison of Transformation Results Mutation Testing Approximate Computing Failing Test When we start considering how mutation testing and approximate computing can improve one another, this contrast in expectations is not so clear cut Passing Test

Approx. Transformation as Operator // MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 0; i < data.length; i++) values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 0; while (iterator.hasNext()) { out[i++] = iterator.next(); return out; // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); } One of our ideas for approximate computing to help mutation testing…

Approx. Transformation as Operator // MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 0; i < data.length; i+=2) values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 0; while (iterator.hasNext()) { out[i++] = iterator.next(); return out; // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); } Test Fails Mutant Killed Replace i++ with i+=2 Perforates loop

Questions for Approx. Operators Do killed approximate mutants indicate different strengths? Do surviving approximate mutants indicate new weaknesses in the test suite? How do mutants generated by approximate computing differ from traditional mutants? Are approximate mutants faster than traditional mutants? What are characteristics of mutants generated by approximate transformations for mutation testing?

Mutants Find Approx. Opportunities // MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 0; i < data.length; i++) values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 0; while (iterator.hasNext()) { out[i++] = iterator.next(); return out; // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); }

Mutants Find Approx. Opportunities // MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 1; i < data.length; i++) values.add(data[i]); } int count = values.size(); double[] out = new double[count]; Iterator<Double> iterator = values.descendingIterator(); int i = 0; while (iterator.hasNext()) { out[i++] = iterator.next(); return out; // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2}; double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); } Test Passes For example, the 0 is redundant in the first location, so perhaps there is some leeway for approximation here Approximable? Replace 0 with 1

Questions for Approx. Opportunities How can we classify surviving mutants? Are they good for approximate computing? What approximations are applicable for which surviving mutants? How can we tailor mutants for the purpose of finding approximate computing opportunities? Tailored mutants have benefit of being automatic, to test the waters before more effort for actual, better approximation

Improving One Another More in paper! Approximate computing to provide new mutation operators for evaluating quality of tests Approximate computing to improve speed of mutation testing Mutation testing to point out opportunities for applying approximations on exact code Mutation testing to evaluate quality of tests on (already) approximate code More in paper!

Conclusions Approximate computing can provide new mutation operators Mutation testing can show opportunities for approximate computing on exact code There is so much more we can do (More directions in the paper) Many more directions for exploring the combination of mutation testing and approximate computing to improve one another August Shi: awshi2@illinois.edu

BACKUP

Mutating Approximate Code? // MathArrays.java static double[] unique(double[] data) { TreeSet<Double> values = new TreeSet<>(); for (int i = 1; i < data.length; i++) { if (i%2 != 0) continue; values.add(data[i]); } return values.toArray(); // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2} double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); } Test Passes Replace 0 with 1

Mutation testing approximate code? Approximate Transformation Instance % Mutants Killed Exact Version N/A 95 Loop Perforation (Line 04) Skip every 2nd iteration 100 Skip every 4th iteration 95 Only execute every 4th iteration If approximate version is proxy of exact version, is mutation score of approximate version also proxy of mutation score of exact version? If so, what are the certain conditions where they are good proxies? What precisely do these changes in percentages mean?

Directions for Research If approximate version is proxy of exact version, is mutation score of approximate version also proxy of mutation score of exact version? Do I get same confidence in quality of tests at cheaper cost? If so, what are the exact conditions where they are good proxies?

Approximate Code to Speed up Testing? // MathArrays.java 01. static double[] unique(double[] data) { 02. TreeSet values = 03. new TreeSet<>(); 04. for (int i = 0; 05. i < data.length; i++) { 06. if (i%2 != 0) continue; 07. values.add(data[i]); 08. } 09. int count = values.size() 10. double[] out = new double[count]; 11. Iterator iterator = 12. values.descendingIterator(); 13. int i = 0; 14. while (iterator.hasNext()) { 15. if (i == data.length / 2 + 1) break; 16. out[i++] = iterator.next(); 17. } 19. return out; 20. } // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2} double[] values = {11, 3, 0, -1}; assertArrayEquals(values, MathArrays.unique(x), 0); } Loop (line 04) takes 68% of runtime Perforation can cut time in half

Approximate Code to Speed up Testing? // MathArrays.java 01. static double[] unique(double[] data) { 02. TreeSet values = 03. new TreeSet<>(); 04. for (int i = 0; 05. i < data.length; i++) { 06. if (i%2 != 0) continue; 07. values.add(data[i]); 08. } 09. int count = values.size() 10. double[] out = new double[count]; 11. Iterator iterator = 12. values.descendingIterator(); 13. int i = 0; 14. while (iterator.hasNext()) { 15. if (i == data.length / 2 + 1) break; 16. out[i++] = iterator.next(); 17. } 19. return out; 20. } // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2} double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertApproxEquals(values, MathArrays.unique(x), 0 0.8); } Introduce new assertions?

Example: Mutation Testing (SURVIVED) // MathArrays.java static double[] unique(double[] data) { TreeSet values = new TreeSet<>(); for (int i = 0; i < data.length; i++) { if (i%2 != 0) continue; values.add(data[i]); } int count = values.size() double[] out = new double[count]; … return out; // MathArrays.java static double[] unique(double[] data) { TreeSet values = new TreeSet<>(); for (int i = 1; i < data.length; i++) { if (i%2 != 0) continue; values.add(data[i]); } int count = values.size() double[] out = new double[count]; … return out; Constant Replacement Original Mutant // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2} double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); } Test passes on original Test passes on mutant => Mutant SURVIVED

Example: Mutation Testing (KILLED) // MathArrays.java static double[] unique(double[] data) { TreeSet values = new TreeSet<>(); for (int i = 0; i < data.length; i++) { if (i%2 != 0) continue; values.add(data[i]); } int count = values.size() double[] out = new double[count]; … return out; // MathArrays.java static double[] unique(double[] data) { TreeSet values = new TreeSet<>(); for (int i = 0; i <= data.length; i++) { if (i%2 != 0) continue; values.add(data[i]); } int count = values.size() double[] out = new double[count]; … return out; Boundary Mutator Original Mutant // MathArraysTest.java void testUnique() { double[] x = {0, 9, 3, 0, 11, 7, 3, 5, −1, −2} double[] values = {11, 9, 7, 5, 3, 0, -1, -2}; assertArrayEquals(values, MathArrays.unique(x), 0); } Test passes on original Test passes on mutant => Mutant KILLED

Example: Loop Perforation // MathArrays.java static double[] unique(double[] data) { TreeSet values = new TreeSet<>(); for (int i = 0; i < data.length; i++) { if (i%2 != 0) continue; values.add(data[i]); } int count = values.size() double[] out = new double[count]; … return out; // MathArrays.java static double[] unique(double[] data) { TreeSet values = new TreeSet<>(); for (int i = 0; i < data.length; i+=2) { if (i%2 != 0) continue; values.add(data[i]); } int count = values.size() double[] out = new double[count]; … return out; Skip Every Other Iteration Original Mutant