1. A Systematic Review of Cost Reduction Techniques for Mutation Testing: Preliminary Results
Fabiano Ferrari
Software Engineering
Federal University of São Carlos
São Carlos, SP Brazil
www2.dc.ufscar.br/~fabiano/
Alessandro Pizzoleto
São Carlos, SP Brazil
lapes.dc.ufscar.br/members/phd-students
Jeff Offutt
George Mason University
Fairfax, VA USA

2. Context and Motivation
Mutation testing has been around for 40 years
Strong test criterion:
Highly effective compared with other test criteria
Adequate for test quality assessment
But…
Mutation is expensive!
© Ferrari, Pizzoleto, Offutt

3. Context and Motivation
Much research has focused on cost reduction
Partial mutant execution (weak mutation, firm mutation …)
Grouping (mutant schemata, mutant clustering …)
Mutant reduction (selective mutation, random mutation …)
Fast execution (parallel execution, compiler optimization …)
Questions
How many techniques have been explored?
What are the results?
What are the tradeoffs?
© Ferrari, Pizzoleto, Offutt

4. Goal and Method of our Study
Characterizing the state-of-the-art of cost reduction of mutation testing
Method:
Systematic Literature Review (SLR)
Rigorous approach to identify, evaluate, and interpret all available evidence about a particular topic
© Ferrari, Pizzoleto, Offutt

5. © Ferrari, Pizzoleto, Offutt
Review Setup
© Ferrari, Pizzoleto, Offutt

6. Review Setup Research Questions: Starting point
RQ1: Which techniques support mutation cost reduction?
RQ2: Which metrics can measure mutation cost reduction?
RQ3: What are the savings (benefit) and loss of effectiveness (mutation score) for the techniques?
Starting point
Studies cited in Jia & Harman’s 2011 survey
© Ferrari, Pizzoleto, Offutt

7. © Ferrari, Pizzoleto, Offutt
Review Setup
Inclusion (I) and Exclusion (E) criteria:
(I-1) The study proposes a way to reduce the cost of mutation or
(I-2) The study applies a way to reduce the cost of mutation
and
(I-3) The study is peer-reviewed and published either in a conference, workshop, or journal
(E-1) The study uses a technique that can reduce the cost of mutation, but cost reduction is not the main focus
(E-2) The study uses mutation testing information for test quality assessment
© Ferrari, Pizzoleto, Offutt

8. Review Setup Study Selection Process 1. Perform Automatic Search
2. Perform Snowballing
3. Perform Author Survey
© Ferrari, Pizzoleto, Offutt

9. Review Setup (2) Study Selection Process 1. Perform Automatic Search
2. Perform Snowballing
3. Perform Author Survey
1.1 Customize Search String
1.2 Query Databases
1.3 Perform Pre-Selection
1.4 Perform Final Selection
© Ferrari, Pizzoleto, Offutt

10. Review Setup (3) Study Selection Process 1. Perform Automatic Search
2. Perform Snowballing
3. Perform Author Survey
2.1 Retrieve References from Selected Studies
2.2 Filter Duplicates
2.3 Perform Pre-Selection
2.4 Perform Final Selection
© Ferrari, Pizzoleto, Offutt

11. Review Setup (4) Study Selection Process 1. Perform Automatic Search
2. Perform Snowballing
3. Perform Author Survey
3.1 Contact Authors of Selected Studies
3.2 Retrieve References of Suggested Studies
3.3 Filter Duplicates
3.4 Perform Pre-Selection
3.5 Perform Final Selection
© Ferrari, Pizzoleto, Offutt

12. © Ferrari, Pizzoleto, Offutt
Results
© Ferrari, Pizzoleto, Offutt

13. Results from Search Study Selection Process 1. Perform Automatic
2. Perform Snowballing
3. Perform Author Survey
Round 1 – April 2016
Round 2 – September 2017
1.1 Customize Search String
1.2 Query Databases
1.3 Perform Pre-Selection
1.4 Perform Final Selection
© Ferrari, Pizzoleto, Offutt

14. Results from Search Study Selection Process 1. Perform Automatic
2. Perform Snowballing
3. Perform Author Survey
(“mutation testing” or “mutation analysis” or “mutant analysis”) and
(“cost reduction” or “sufficient operator” or “sufficient mutation” or “constrained mutation” or “selective mutation” or “weak mutation” or “random selection” or “random mutation” or “random mutants” or “equivalent mutant” or “equivalent mutation”)
1.1 Customize Search String
1.2 Query Databases
1.3 Perform Pre-Selection
1.4 Perform Final Selection
© Ferrari, Pizzoleto, Offutt

15. Results from Search Study Selection Process 1. Perform Automatic
2. Perform Snowballing
3. Perform Author Survey
IEEE Xplore, ACM Digital Library,
Elsevier ScienceDirect, Springer SpringerLink,
Wiley Online Library, Expert Suggestions
1.1 Customize Search String
1.2 Query Databases
1.3 Perform Pre-Selection
1.4 Perform Final Selection
© Ferrari, Pizzoleto, Offutt

16. Results from Search Study Selection Process 1. Perform Automatic
2. Perform Snowballing
3. Perform Author Survey
Round 1: 85 studies
Round 2 : 20 studies
1.1 Customize Search String
1.2 Query Databases
1.3 Perform Pre-Selection
1.4 Perform Final Selection
© Ferrari, Pizzoleto, Offutt

17. Results from Snowballin
Study Selection Process
1. Perform Automatic
Search
2. Perform Snowballing
3. Perform Author Survey
Over studies selected in round 1
19 studies
2.1 Retrieve References from Selected Studies
2.2 Filter Duplicates
2.3 Perform Pre-Selection
2.4 Perform Final Selection
© Ferrari, Pizzoleto, Offutt

18. Results from Survey Study Selection Process 1. Perform Automatic
Search
2. Perform Snowballing
3. Perform Author Survey
Over all 124 selected studies
128 authors contacted by
41 replies
131 suggestions
22 more studies
3.1 Contact Authors of Selected Studies
3.2 Retrieve References of Suggested Studies
3.3 Filter Duplicates
3.4 Perform Pre-Selection
3.5 Perform Final Selection
© Ferrari, Pizzoleto, Offutt

19. © Ferrari, Pizzoleto, Offutt
Summary of Papers
146 selected studies
51 journal papers
86 conference and workshop full papers
9 conference and workshop short papers
© Ferrari, Pizzoleto, Offutt

20. Categorizing by Technique
© Ferrari, Pizzoleto, Offutt

21. © Ferrari, Pizzoleto, Offutt
Counting by Year
© Ferrari, Pizzoleto, Offutt

22. Analysis and Discussion
© Ferrari, Pizzoleto, Offutt

23. Primary Goals for Cost Reduction
6 primary cost reduction goals:
Reducing the number of mutants
Detecting equivalent mutants
Executing faster
Reducing the number of test cases or the number of test case executions
Avoiding the creation of certain mutants
Automatically generating test cases
© Ferrari, Pizzoleto, Offutt

24. 22 Techniques for Cost Reduction
Random mutation
Higher order mutation
Weak mutation
Firm mutation
Parallel execution
Data-flow analysis
Minimization and prioritization of test sets
Compiler optimization
Constrained mutation
Metamutants (schemata)
Control-flow analysis
© Ferrari, Pizzoleto, Offutt

25. 22 Techniques for Cost Reduction
Random mutation
Higher order mutation
Weak mutation
Firm mutation
Parallel execution
Data-flow analysis
Minimization and prioritization of test sets
Compiler optimization
Constrained mutation
Metamutants (schemata)
Control-flow analysis
Selective mutation
Serial execution
Sufficient operators
Optimization of generation, execution, and analysis of mutants
Evolutionary algorithms
Sequential and orderly execution
One-op mutation
Execution trace analysis
Model-based testing
State-based analysis
Minimal mutation
© Ferrari, Pizzoleto, Offutt

26. Techniques for Cost Reduction
© Ferrari, Pizzoleto, Offutt

27. Measurement of Cost Reduction
18 metrics:
Number of test cases required to achieve mutant coverage
Number of mutants to be executed
Mutant execution speedup
Mutant execution efficiency
Test generation speedup
Number of equivalent mutants automatically detected
Number of cycles or generations) in evolutionary algorithms
Mutant compilation speedup
Mutant generation speedup
© Ferrari, Pizzoleto, Offutt

28. Measurement of Cost Reduction
18 metrics:
Number of test cases required to achieve mutant coverage
Number of mutants to be executed
Mutant execution speedup
Mutant execution efficiency
Test generation speedup
Number of equivalent mutants automatically detected
Number of cycles or generations) in evolutionary algorithms
Mutant compilation speedup
Mutant generation speedup
Completeness of generated test suite
Number of killable mutants automatically detected
Number of equivalent mutants generated
Number of cycles in test generation algorithms
Mutant analysis speedup
Probability to kill mutants
Number of test case executions
Number of duplicated mutants automatically detected
Variation in mutation score
© Ferrari, Pizzoleto, Offutt

29. Measurement of Cost Reduction
© Ferrari, Pizzoleto, Offutt

30. © Ferrari, Pizzoleto, Offutt
Conclusions
Many different techniques
Some are combined to achieve single or multiple goals
Selecting subsets of mutants
Automatic equivalent mutant detection
Fast and fewer executions
Metrics to evaluate them include:
Number of mutants to be executed
Mutant execution speedup
Recent and active cost reduction approaches include:
Evolutionary algorithms
Control-flow analysis
Selective mutation (including minimal mutation)
© Ferrari, Pizzoleto, Offutt

31. © Ferrari, Pizzoleto, Offutt
Future Work
Extensive analysis of selected studies
Overall characterization of studies, techniques and metrics
Research Questions RQ1 and RQ2
Analysis of savings and loss of effectiveness for the techniques
Research Question RQ3
Much deeper and longer journal paper
Regular updates of the dataset
Fabiano Ferrari
Questions :
© Ferrari, Pizzoleto, Offutt