1. Mutation Testing Meets Approximate Computing
Milos Gligoric1, Sarfraz Khurshid1, Sasa Misailovic2, August Shi2
ICSE NIER 2017
Buenos Aires, Argentina
May 24, 2017
“My name is August Shi. I am from the University of Illinois at Urbana-Champaign. I will be presenting our work Mutation Testing Meets Approximate Computing. This is joint work with Milos and Sarfraz, who are from the University of Texas at Austin, and with Sasa, who is also from the University of Illinois at Urbana-Champaign. Our work involves both mutation testing and approximate computing, so I will start with a bit of background on both.” 1 2
CCF , CCF ,
CCF , CCF ,
CCF , CNS

2. Mutation Testing Goal: Evaluate quality of test suites
How: Apply transformations (mutation operators) on the code and run tests to see if they can detect the changes
Example:
x = x + 1
“Mutation testing is a technique used for many years, particularly in software testing research, to evaluate quality of test suites”
Say transformations = mutation operators

3. Mutation Testing Goal: Evaluate quality of test suites
How: Apply transformations (mutation operators) on the code and run tests to see if they can detect the changes
Example:
x = x + 2
Problems:
Evaluation of quality limited by mutation operators
Too slow
“Mutation testing is a traditional technique used in research to evaluate quality of test suites”
Say transformations = mutation operators

4. Approximate Computing
Goal: Improve performance of code
How: Apply transformations that may lead to (slightly) inaccurate results
Example:
for (i = 0; i < n; i = i + 1)
“Approximate computing is a new, emerging area of research, especially in the programming systems community, that does something similar, but has a different goal”
Point out how transforming code is a common factor in the “how” part between the two

5. Approximate Computing
Goal: Improve performance of code
How: Apply transformations that may lead to (slightly) inaccurate results
Example:
for (i = 0; i < n; i = i + 2)
Problems:
Not sure where in exact code to apply approximations
Unclear how to check quality of tests on already approximate code
“Approximate computing is a new, up-and-coming area of research that does something similar, but has a different goal”
Goal of performance, e.g., speed (tie in with later part)
Point out how transforming code is a common factor in the “how” part between the two

6. How can Mutation Testing and Approximate Computing improve one another?
“More specifically, can we use one to mitigate the problems of the other?”

7. Improving One Another
Approximate computing to provide new mutation operators for evaluating quality of tests
Approximate computing to improve speed of mutation testing
Mutation testing to point out opportunities for applying approximations on exact code
Mutation testing to evaluate quality of tests on (already) approximate code

8. Example Code: Commons-Math
// MathArrays.java
static double[] unique(double[] data) {
TreeSet<Double> values =
new TreeSet<>();
for (int i = 0; i < data.length; i++) {
0 values.add(data[i]); }
int count = values.size();
double[] out = new double[count];
Iterator<Double> iterator =
values.descendingIterator();
int i = 0;
while (iterator.hasNext()) {
out[i++] = iterator.next();
return out;
// MathArraysTest.java
void testUnique() {
double[] x = {0, 9, 3, 0, 11,
7, 3, 5, −1, −2};
double[] values = {11, 9, 7,
5, 3, 0, -1, -2};
assertArrayEquals(values,
MathArrays.unique(x), 0); }
Test Passes
“I would like to demonstrate some of these ideas using an example…”

9. Mutant: Constant Replacement
// MathArrays.java
static double[] unique(double[] data) {
TreeSet<Double> values =
new TreeSet<>();
for (int i = 0; i < data.length; i++) {
values.add(data[i]); }
int count = values.size();
double[] out = new double[count];
Iterator<Double> iterator =
values.descendingIterator();
int i = 1;
while (iterator.hasNext()) {
out[i++] = iterator.next();
return out;
// MathArraysTest.java
void testUnique() {
double[] x = {0, 9, 3, 0, 11,
7, 3, 5, −1, −2};
double[] values = {11, 9, 7,
5, 3, 0, -1, -2};
assertArrayEquals(values,
MathArrays.unique(x), 0); }
Test Fails
Mutant Killed
Replace 0 with 1

10. Mutant: Constant Replacement
// MathArrays.java
static double[] unique(double[] data) {
TreeSet<Double> values =
new TreeSet<>();
for (int i = 1; i < data.length; i++) {
0 values.add(data[i]); }
int count = values.size();
double[] out = new double[count];
Iterator<Double> iterator =
values.descendingIterator();
int i = 0;
while (iterator.hasNext()) {
out[i++] = iterator.next();
return out;
// MathArraysTest.java
void testUnique() {
double[] x = {0, 9, 3, 0, 11,
7, 3, 5, −1, −2};
double[] values = {11, 9, 7,
5, 3, 0, -1, -2};
assertArrayEquals(values,
MathArrays.unique(x), 0); }
Test Passes
Mutant Survived
Replace 0 with 1

11. Approximate: Loop Perforation
// MathArrays.java
static double[] unique(double[] data) {
TreeSet<Double> values =
new TreeSet<>();
for (int i = 0; i < data.length; i+=2)
values.add(data[i]); }
int count = values.size();
double[] out = new double[count];
Iterator<Double> iterator =
values.descendingIterator();
int i = 0;
while (iterator.hasNext()) {
out[i++] = iterator.next();
return out;
// MathArraysTest.java
void testUnique() {
double[] x = {0, 9, 3, 0, 11,
7, 3, 5, −1, −2};
double[] values = {11, 9, 7,
5, 3, 0, -1, -2};
assertArrayEquals(values,
MathArrays.unique(x), 0); }
Test Fails
Normally, this transformation would cause the test to fail
In such a case, the approximation is invalid.
So a developer has two options: the developer can do a different approximation, or if the developer has determined the approximation is good and it’s the test that’s too rigid, the developer can modify the test to accept the approximation
Skip every other iteration
68% of runtime is in this loop

12. Approximate: Loop Perforation
Modify assertion
Approximate: Loop Perforation
// MathArrays.java
static double[] unique(double[] data) {
TreeSet<Double> values =
new TreeSet<>();
for (int i = 0; i < data.length; i+=2)
values.add(data[i]); }
int count = values.size();
double[] out = new double[count];
Iterator<Double> iterator =
values.descendingIterator();
int i = 0;
while (iterator.hasNext()) {
out[i++] = iterator.next();
return out;
// MathArraysTest.java
void testUnique() {
double[] x = {0, 9, 3, 0, 11,
7, 3, 5, −1, −2};
double[] values = {11, 9, 7,
5, 3, 0, -1, -2};
assertArraySubset(values,
MathArrays.unique(x), 0); }
Test Passes
For the sake of our example, let’s say the developer chooses to modify the test assertion to allow for this approximation
Let’s say there is this new assertion that checks if elements are a subset of another (though assertion does not actually exist)
Because in approximate computing, we need to have the test pass
Approximation is Acceptable
Skip every other iteration
68% of runtime is in this loop

13. Comparison of Transformation Results
Mutation
Testing
Approximate
Computing
Failing Test
While we see that both mutation testing and approximate computing do transformations, they expect different things from the results of tests due to transformations
There seems to be a conflict between expectations of if the test passes or fails after transformations based on which mindset a developer is in
Passing Test

14. Comparison of Transformation Results
Mutation
Testing
Approximate
Computing
Failing Test
When we start considering how mutation testing and approximate computing can improve one another, this contrast in expectations is not so clear cut
Passing Test

15. Approx. Transformation as Operator
// MathArrays.java
static double[] unique(double[] data) {
TreeSet<Double> values =
new TreeSet<>();
for (int i = 0; i < data.length; i++)
values.add(data[i]); }
int count = values.size();
double[] out = new double[count];
Iterator<Double> iterator =
values.descendingIterator();
int i = 0;
while (iterator.hasNext()) {
out[i++] = iterator.next();
return out;
// MathArraysTest.java
void testUnique() {
double[] x = {0, 9, 3, 0, 11,
7, 3, 5, −1, −2};
double[] values = {11, 9, 7,
5, 3, 0, -1, -2};
assertArrayEquals(values,
MathArrays.unique(x), 0); }
One of our ideas for approximate computing to help mutation testing…

16. Approx. Transformation as Operator
// MathArrays.java
static double[] unique(double[] data) {
TreeSet<Double> values =
new TreeSet<>();
for (int i = 0; i < data.length; i+=2)
values.add(data[i]); }
int count = values.size();
double[] out = new double[count];
Iterator<Double> iterator =
values.descendingIterator();
int i = 0;
while (iterator.hasNext()) {
out[i++] = iterator.next();
return out;
// MathArraysTest.java
void testUnique() {
double[] x = {0, 9, 3, 0, 11,
7, 3, 5, −1, −2};
double[] values = {11, 9, 7,
5, 3, 0, -1, -2};
assertArrayEquals(values,
MathArrays.unique(x), 0); }
Test Fails
Mutant Killed
Replace i++ with i+=2
Perforates loop

17. Questions for Approx. Operators
Do killed approximate mutants indicate different strengths? Do surviving approximate mutants indicate new weaknesses in the test suite?
How do mutants generated by approximate computing differ from traditional mutants?
Are approximate mutants faster than traditional mutants?
What are characteristics of mutants generated by approximate transformations for mutation testing?

18. Mutants Find Approx. Opportunities
// MathArrays.java
static double[] unique(double[] data) {
TreeSet<Double> values =
new TreeSet<>();
for (int i = 0; i < data.length; i++)
values.add(data[i]); }
int count = values.size();
double[] out = new double[count];
Iterator<Double> iterator =
values.descendingIterator();
int i = 0;
while (iterator.hasNext()) {
out[i++] = iterator.next();
return out;
// MathArraysTest.java
void testUnique() {
double[] x = {0, 9, 3, 0, 11,
7, 3, 5, −1, −2};
double[] values = {11, 9, 7,
5, 3, 0, -1, -2};
assertArrayEquals(values,
MathArrays.unique(x), 0); }

19. Mutants Find Approx. Opportunities
// MathArrays.java
static double[] unique(double[] data) {
TreeSet<Double> values =
new TreeSet<>();
for (int i = 1; i < data.length; i++)
values.add(data[i]); }
int count = values.size();
double[] out = new double[count];
Iterator<Double> iterator =
values.descendingIterator();
int i = 0;
while (iterator.hasNext()) {
out[i++] = iterator.next();
return out;
// MathArraysTest.java
void testUnique() {
double[] x = {0, 9, 3, 0, 11,
7, 3, 5, −1, −2};
double[] values = {11, 9, 7,
5, 3, 0, -1, -2};
assertArrayEquals(values,
MathArrays.unique(x), 0); }
Test Passes
For example, the 0 is redundant in the first location, so perhaps there is some leeway for approximation here
Approximable?
Replace 0 with 1

20. Questions for Approx. Opportunities
How can we classify surviving mutants? Are they good for approximate computing?
What approximations are applicable for which surviving mutants?
How can we tailor mutants for the purpose of finding approximate computing opportunities?
Tailored mutants have benefit of being automatic, to test the waters before more effort for actual, better approximation

21. Improving One Another More in paper!
Approximate computing to provide new mutation operators for evaluating quality of tests
Approximate computing to improve speed of mutation testing
Mutation testing to point out opportunities for applying approximations on exact code
Mutation testing to evaluate quality of tests on (already) approximate code
More in paper!

22. Conclusions Approximate computing can provide new mutation operators
Mutation testing can show opportunities for approximate computing on exact code
There is so much more we can do (More directions in the paper)
Many more directions for exploring the combination of mutation testing and approximate computing to improve one another
August Shi:

23. BACKUP

24. Mutating Approximate Code?
// MathArrays.java
static double[] unique(double[] data) {
TreeSet<Double> values =
new TreeSet<>();
for (int i = 1; i < data.length; i++) {
if (i%2 != 0) continue;
values.add(data[i]); }
return values.toArray();
// MathArraysTest.java
void testUnique() {
double[] x = {0, 9, 3, 0, 11,
7, 3, 5, −1, −2}
double[] values = {11, 9, 7,
5, 3, 0, -1, -2};
assertArrayEquals(values,
MathArrays.unique(x), 0); }
Test Passes
Replace 0 with 1

25. Mutation testing approximate code?
Approximate Transformation
Instance
% Mutants Killed
Exact Version
N/A 95
Loop Perforation (Line 04)
Skip every 2nd iteration
100
Skip every 4th iteration 95
Only execute every 4th iteration
If approximate version is proxy of exact version, is mutation score of approximate version also proxy of mutation score of exact version?
If so, what are the certain conditions where they are good proxies?
What precisely do these changes in percentages mean?

26. Directions for Research
If approximate version is proxy of exact version, is mutation score of approximate version also proxy of mutation score of exact version?
Do I get same confidence in quality of tests at cheaper cost?
If so, what are the exact conditions where they are good proxies?

27. Approximate Code to Speed up Testing?
// MathArrays.java
01. static double[] unique(double[] data) {
TreeSet values =
new TreeSet<>();
for (int i = 0;
i < data.length; i++) {
if (i%2 != 0) continue;
values.add(data[i]); }
int count = values.size()
double[] out = new double[count];
Iterator iterator =
values.descendingIterator();
int i = 0;
while (iterator.hasNext()) {
if (i == data.length / 2 + 1) break;
out[i++] = iterator.next(); }
return out;
20. }
// MathArraysTest.java
void testUnique() {
double[] x = {0, 9, 3, 0, 11,
7, 3, 5, −1, −2}
double[] values = {11, 3,
0, -1};
assertArrayEquals(values,
MathArrays.unique(x), 0); }
Loop (line 04) takes
68% of runtime
Perforation can cut
time in half

28. Approximate Code to Speed up Testing?
// MathArrays.java
01. static double[] unique(double[] data) {
TreeSet values =
new TreeSet<>();
for (int i = 0;
i < data.length; i++) {
if (i%2 != 0) continue;
values.add(data[i]); }
int count = values.size()
double[] out = new double[count];
Iterator iterator =
values.descendingIterator();
int i = 0;
while (iterator.hasNext()) {
if (i == data.length / 2 + 1) break;
out[i++] = iterator.next(); }
return out;
20. }
// MathArraysTest.java
void testUnique() {
double[] x = {0, 9, 3, 0, 11,
7, 3, 5, −1, −2}
double[] values = {11, 9, 7,
5, 3, 0, -1, -2};
assertApproxEquals(values,
MathArrays.unique(x), 0
0.8); }
Introduce new assertions?

29. Example: Mutation Testing (SURVIVED)
// MathArrays.java
static double[] unique(double[] data) {
TreeSet values =
new TreeSet<>();
for (int i = 0;
i < data.length; i++) {
if (i%2 != 0) continue;
values.add(data[i]); }
int count = values.size()
double[] out = new double[count]; …
return out;
// MathArrays.java
static double[] unique(double[] data) {
TreeSet values =
new TreeSet<>();
for (int i = 1;
i < data.length; i++) {
if (i%2 != 0) continue;
values.add(data[i]); }
int count = values.size()
double[] out = new double[count]; …
return out;
Constant
Replacement
Original
Mutant
// MathArraysTest.java
void testUnique() {
double[] x = {0, 9, 3, 0, 11,
7, 3, 5, −1, −2}
double[] values = {11, 9, 7, 5, 3,
0, -1, -2};
assertArrayEquals(values,
MathArrays.unique(x), 0); }
Test passes on original
Test passes on mutant => Mutant SURVIVED

30. Example: Mutation Testing (KILLED)
// MathArrays.java
static double[] unique(double[] data) {
TreeSet values =
new TreeSet<>();
for (int i = 0;
i < data.length; i++) {
if (i%2 != 0) continue;
values.add(data[i]); }
int count = values.size()
double[] out = new double[count]; …
return out;
// MathArrays.java
static double[] unique(double[] data) {
TreeSet values =
new TreeSet<>();
for (int i = 0;
i <= data.length; i++) {
if (i%2 != 0) continue;
values.add(data[i]); }
int count = values.size()
double[] out = new double[count]; …
return out;
Boundary
Mutator
Original
Mutant
// MathArraysTest.java
void testUnique() {
double[] x = {0, 9, 3, 0, 11,
7, 3, 5, −1, −2}
double[] values = {11, 9, 7, 5, 3,
0, -1, -2};
assertArrayEquals(values,
MathArrays.unique(x), 0); }
Test passes on original
Test passes on mutant => Mutant KILLED

31. Example: Loop Perforation
// MathArrays.java
static double[] unique(double[] data) {
TreeSet values =
new TreeSet<>();
for (int i = 0;
i < data.length; i++) {
if (i%2 != 0) continue;
values.add(data[i]); }
int count = values.size()
double[] out = new double[count]; …
return out;
// MathArrays.java
static double[] unique(double[] data) {
TreeSet values =
new TreeSet<>();
for (int i = 0;
i < data.length; i+=2) {
if (i%2 != 0) continue;
values.add(data[i]); }
int count = values.size()
double[] out = new double[count]; …
return out;
Skip Every
Other Iteration
Original
Mutant