A Systematic Review of Cost Reduction Techniques for Mutation Testing: Preliminary Results Fabiano Ferrari Software Engineering Federal University of São Carlos São Carlos, SP Brazil fcferrari@ufscar.br www2.dc.ufscar.br/~fabiano/ Alessandro Pizzoleto São Carlos, SP Brazil alessandro.pizzoleto@ufscar.br lapes.dc.ufscar.br/members/phd-students Jeff Offutt George Mason University Fairfax, VA USA offutt@gmu.edu www.cs.gmu.edu/~offutt/

Context and Motivation Mutation testing has been around for 40 years Strong test criterion: Highly effective compared with other test criteria Adequate for test quality assessment But… Mutation is expensive! © Ferrari, Pizzoleto, Offutt

Context and Motivation Much research has focused on cost reduction Partial mutant execution (weak mutation, firm mutation …) Grouping (mutant schemata, mutant clustering …) Mutant reduction (selective mutation, random mutation …) Fast execution (parallel execution, compiler optimization …) Questions How many techniques have been explored? What are the results? What are the tradeoffs? © Ferrari, Pizzoleto, Offutt

Goal and Method of our Study Characterizing the state-of-the-art of cost reduction of mutation testing Method: Systematic Literature Review (SLR) Rigorous approach to identify, evaluate, and interpret all available evidence about a particular topic © Ferrari, Pizzoleto, Offutt

© Ferrari, Pizzoleto, Offutt Review Setup © Ferrari, Pizzoleto, Offutt

Review Setup Research Questions: Starting point RQ1: Which techniques support mutation cost reduction? RQ2: Which metrics can measure mutation cost reduction? RQ3: What are the savings (benefit) and loss of effectiveness (mutation score) for the techniques? Starting point Studies cited in Jia & Harman’s 2011 survey © Ferrari, Pizzoleto, Offutt

© Ferrari, Pizzoleto, Offutt Review Setup Inclusion (I) and Exclusion (E) criteria: (I-1) The study proposes a way to reduce the cost of mutation or (I-2) The study applies a way to reduce the cost of mutation and (I-3) The study is peer-reviewed and published either in a conference, workshop, or journal (E-1) The study uses a technique that can reduce the cost of mutation, but cost reduction is not the main focus (E-2) The study uses mutation testing information for test quality assessment © Ferrari, Pizzoleto, Offutt

Review Setup Study Selection Process 1. Perform Automatic Search 2. Perform Snowballing 3. Perform Author Survey © Ferrari, Pizzoleto, Offutt

Review Setup (2) Study Selection Process 1. Perform Automatic Search 2. Perform Snowballing 3. Perform Author Survey 1.1 Customize Search String 1.2 Query Databases 1.3 Perform Pre-Selection 1.4 Perform Final Selection © Ferrari, Pizzoleto, Offutt

Review Setup (3) Study Selection Process 1. Perform Automatic Search 2. Perform Snowballing 3. Perform Author Survey 2.1 Retrieve References from Selected Studies 2.2 Filter Duplicates 2.3 Perform Pre-Selection 2.4 Perform Final Selection © Ferrari, Pizzoleto, Offutt

Review Setup (4) Study Selection Process 1. Perform Automatic Search 2. Perform Snowballing 3. Perform Author Survey 3.1 Contact Authors of Selected Studies 3.2 Retrieve References of Suggested Studies 3.3 Filter Duplicates 3.4 Perform Pre-Selection 3.5 Perform Final Selection © Ferrari, Pizzoleto, Offutt

© Ferrari, Pizzoleto, Offutt Results © Ferrari, Pizzoleto, Offutt

Results from Search Study Selection Process 1. Perform Automatic 2. Perform Snowballing 3. Perform Author Survey Round 1 – April 2016 Round 2 – September 2017 1.1 Customize Search String 1.2 Query Databases 1.3 Perform Pre-Selection 1.4 Perform Final Selection © Ferrari, Pizzoleto, Offutt

Results from Search Study Selection Process 1. Perform Automatic 2. Perform Snowballing 3. Perform Author Survey (“mutation testing” or “mutation analysis” or “mutant analysis”) and (“cost reduction” or “sufficient operator” or “sufficient mutation” or “constrained mutation” or “selective mutation” or “weak mutation” or “random selection” or “random mutation” or “random mutants” or “equivalent mutant” or “equivalent mutation”) 1.1 Customize Search String 1.2 Query Databases 1.3 Perform Pre-Selection 1.4 Perform Final Selection © Ferrari, Pizzoleto, Offutt

Results from Search Study Selection Process 1. Perform Automatic 2. Perform Snowballing 3. Perform Author Survey IEEE Xplore, ACM Digital Library, Elsevier ScienceDirect, Springer SpringerLink, Wiley Online Library, Expert Suggestions 1.1 Customize Search String 1.2 Query Databases 1.3 Perform Pre-Selection 1.4 Perform Final Selection © Ferrari, Pizzoleto, Offutt

Results from Search Study Selection Process 1. Perform Automatic 2. Perform Snowballing 3. Perform Author Survey Round 1: 85 studies Round 2 : 20 studies 1.1 Customize Search String 1.2 Query Databases 1.3 Perform Pre-Selection 1.4 Perform Final Selection © Ferrari, Pizzoleto, Offutt

Results from Snowballin Study Selection Process 1. Perform Automatic Search 2. Perform Snowballing 3. Perform Author Survey Over studies selected in round 1 19 studies 2.1 Retrieve References from Selected Studies 2.2 Filter Duplicates 2.3 Perform Pre-Selection 2.4 Perform Final Selection © Ferrari, Pizzoleto, Offutt

Results from Survey Study Selection Process 1. Perform Automatic Search 2. Perform Snowballing 3. Perform Author Survey Over all 124 selected studies 128 authors contacted by e-mail 41 replies 131 suggestions 22 more studies 3.1 Contact Authors of Selected Studies 3.2 Retrieve References of Suggested Studies 3.3 Filter Duplicates 3.4 Perform Pre-Selection 3.5 Perform Final Selection © Ferrari, Pizzoleto, Offutt

© Ferrari, Pizzoleto, Offutt Summary of Papers 146 selected studies 51 journal papers 86 conference and workshop full papers 9 conference and workshop short papers © Ferrari, Pizzoleto, Offutt

Categorizing by Technique © Ferrari, Pizzoleto, Offutt

© Ferrari, Pizzoleto, Offutt Counting by Year © Ferrari, Pizzoleto, Offutt

Analysis and Discussion © Ferrari, Pizzoleto, Offutt

Primary Goals for Cost Reduction 6 primary cost reduction goals: Reducing the number of mutants Detecting equivalent mutants Executing faster Reducing the number of test cases or the number of test case executions Avoiding the creation of certain mutants Automatically generating test cases © Ferrari, Pizzoleto, Offutt

22 Techniques for Cost Reduction Random mutation Higher order mutation Weak mutation Firm mutation Parallel execution Data-flow analysis Minimization and prioritization of test sets Compiler optimization Constrained mutation Metamutants (schemata) Control-flow analysis © Ferrari, Pizzoleto, Offutt

22 Techniques for Cost Reduction Random mutation Higher order mutation Weak mutation Firm mutation Parallel execution Data-flow analysis Minimization and prioritization of test sets Compiler optimization Constrained mutation Metamutants (schemata) Control-flow analysis Selective mutation Serial execution Sufficient operators Optimization of generation, execution, and analysis of mutants Evolutionary algorithms Sequential and orderly execution One-op mutation Execution trace analysis Model-based testing State-based analysis Minimal mutation © Ferrari, Pizzoleto, Offutt

Techniques for Cost Reduction © Ferrari, Pizzoleto, Offutt

Measurement of Cost Reduction 18 metrics: Number of test cases required to achieve mutant coverage Number of mutants to be executed Mutant execution speedup Mutant execution efficiency Test generation speedup Number of equivalent mutants automatically detected Number of cycles or generations) in evolutionary algorithms Mutant compilation speedup Mutant generation speedup © Ferrari, Pizzoleto, Offutt

Measurement of Cost Reduction 18 metrics: Number of test cases required to achieve mutant coverage Number of mutants to be executed Mutant execution speedup Mutant execution efficiency Test generation speedup Number of equivalent mutants automatically detected Number of cycles or generations) in evolutionary algorithms Mutant compilation speedup Mutant generation speedup Completeness of generated test suite Number of killable mutants automatically detected Number of equivalent mutants generated Number of cycles in test generation algorithms Mutant analysis speedup Probability to kill mutants Number of test case executions Number of duplicated mutants automatically detected Variation in mutation score © Ferrari, Pizzoleto, Offutt

Measurement of Cost Reduction © Ferrari, Pizzoleto, Offutt

© Ferrari, Pizzoleto, Offutt Conclusions Many different techniques Some are combined to achieve single or multiple goals Selecting subsets of mutants Automatic equivalent mutant detection Fast and fewer executions Metrics to evaluate them include: Number of mutants to be executed Mutant execution speedup Recent and active cost reduction approaches include: Evolutionary algorithms Control-flow analysis Selective mutation (including minimal mutation) © Ferrari, Pizzoleto, Offutt

© Ferrari, Pizzoleto, Offutt Future Work Extensive analysis of selected studies Overall characterization of studies, techniques and metrics Research Questions RQ1 and RQ2 Analysis of savings and loss of effectiveness for the techniques Research Question RQ3 Much deeper and longer journal paper Regular updates of the dataset Fabiano Ferrari fcferrari@ufscar.br Questions : © Ferrari, Pizzoleto, Offutt