Establishing Theoretical Minimal Sets of Mutants ICST 2014 Paul Ammann Joint work with Marcio Eduardo Delamaro Jeff Offutt April 1, 2014

2 Outline The situation – Researchers use mutation analysis to evaluate test selection strategies The problem – What do mutation scores mean? The model – Motivating idea: Minimal mutant sets don’t have redundant mutants Need to define notion of redundancy – Main result: Dynamic subsumption = Minimal mutant sets Reduced mutation: Is it close to minimal? – Apply model to Siemens suite Result: Huge gap – Good news: That’s an opportunity!

Researchers Use Mutation Analysis to Evaluate Test Selection Strategies Test Selection Strategy C Test Selection Strategy B Test Selection Strategy A Carefully Chosen Artifacts “Good” Tests Deep Analysis Test Set C Select Test Sets with Test Selection Strategies Test Set B Test Set A Measure Exactly What Does A Score of 91% Mean? 3

The Problem With Mutation Scores m1m1 m2m2 m3m3 m4m4 t1t1 t2t2 t3t3 t4t4 t5t5 Evaluate 3 Test Sets with 4 Mutants: A: {t 1, t 2 } B : {t 2, t 5 } C: {t 3 } Mutation Scores for 3 Test Sets B scores 75% Is that good? 4

Let’s Add Some More Mutants m1m1 m2m2 m3m3 m4m4 m5m5 m6m6 m7m7 m8m8 m9m9 m 10 t1t1 t2t2 t3t3 t4t4 t5t5 Evaluate 3 Test Sets with 10 Mutants: A: {t 1, t 2 } B: {t 2, t 5 } C : {t 3 } Mutation Scores for 3 Test Sets Now B scores 90%! Did B just get better? Every test kills m 8 What’s the point? The same tests kill m 3 and m 6. We say that T does not distinguish m 3 from m 6 Ditto for m 5 and m 9 5

Let’s Throw Away Some Mutants m1m1 m2m2 t1t1 t2t2 t3t3 t4t4 t5t5 Evaluate 3 Test Sets with 2 Mutants: A: {t 1, t 2 } B : {t 2, t 5 } C : {t 3 } Mutation Scores for 3 Test Sets Now B scores 100% Did B get even better? 6

All Together Now m1m1 m2m2 m3m3 m4m4 m5m5 m6m6 m7m7 m8m8 m9m9 m 10 t1t1 t2t2 t3t3 t4t4 t5t5 Evaluate 3 Test Sets with Various Mutants: A: {t 1, t 2 } B : {t 2, t 5 } C: {t 3 } Cumulative Scores Is B lousy or good? What about C? 7

What Makes a Mutant Redundant? m1m1 m2m2 m3m3 m4m4 t1t1 t2t2 t3t3 t4t4 t5t5 Choose M = {m 1, m 2, m 3, m 4 } Choose T = {t 1, t 2, t 3, t 4, t 5 } Minimal test sets wrt M and T: {t 1, t 2 }, {t 1, t 3 }, {t 4 } Try removing m 1 : M 1 = M - {m 1 } Minimal test sets wrt M 1 and T: {t 1, t 2 }, {t 1, t 3 }, {t 4 } No change, so m 1 is redundant Basic Idea: Throwing away a redundant mutant has no effect on the minimal test sets. Try removing m 3 : M 3 = M - {m 3 } Minimal test sets wrt M 3 and T: {t 1 }, {t 4 } A change, so m 3 is not redundant Try removing m 4 : M 4 = M - {m 4 } Minimal test sets wrt M 4 and T: {t 1, t 2 }, {t 1, t 3 }, {t 2, t 5 }, {t 3, t 5 }, {t 4 } A change, so m 4 is not redundant Ditto for M 2 = M - {m 2 } 8

9 Minimal Sets of Mutants Definition – M is minimal if it does not contain redundant mutants Minimal mutant sets from the definition – Requires computing all minimal test sets, which is NP complete  We need an efficient algorithm for finding minimal mutant sets – Turn to dynamic subsumption Subsumption with respect to a test set

Dynamic Subsumption 10 Test set T Tests that kill m j Tests that kill m i Tests that kill m k m i → m j m i → m k ✔ ✖ ? ?

11 Efficiently Computing Minimal Sets of Mutants Formally: m x dynamically subsumes m y wrt T iff – Some test in T kills m x – Every test in T that kills m x also kills m y Main result: Mutant set M minimal wrt T = no dynamic subsumption in M Properties – Only need to consider mutants in pairs Groups of mutants do not make another mutant redundant – Fast – Every minimal mutant set has the same cardinality Contrast with minimal test sets

12 What Does This Mean in Practice? Apply the definitions to the Siemens test bed – See what happens! 7 programs – print_tokens – print_tokens2 – replace – schedule – schedule2 – tcas – totinfo Extensive hand-crafted test set

13 Test Characteristics Notes: – 512 is an artifact of the Proteum tool Our approach applies with any test set – Most tests used were also distinguished – Minimal test set size modest compared to number of tests used ProgramTests Available Tests Used Distinguished Tests Minimal Tests print_tokens print_tokens replace schedule schedule tcas totinfo

14 Mutant Characteristics “Killed Mutants” means those killed by the test set of size 512 Vast majority of remainder are equivalent ProgramTotal Mutants Killed Mutants Distinguished Mutants Minimal Mutants print_tokens print_tokens replace schedule schedule tcas totinfo Most mutants are redundant! –Tiny fraction of mutants are actually minimal wrt 512 tests! –print_tokens: Killing the right 28 mutants guarantees killing all 3711

15 How Good Are Reduced Mutation Strategies? We considered five approaches to reduced mutation – STMT: Statement deletion (Proteum SSDL) – ROR: Relation operators (Proteum ORRN) – CON: Replace scalars with constants (Proteum CCSR) – 5RND: 5% Random selection of all mutants – SELECT: “Selective” mutation (Proteum OOAN, OLLN, ORRN, OLNG) Method: – Choose test sets adequate for each reduced mutation approach wrt test sets analyzed earlier – Compute mutation score Against all mutants Against minimal mutant set Equivalent mutants hand-identified and removed

16 Reduced Mutation Scores: Raw vs. Minimal Notes: – Table entries: Raw Mutation Score: Minimal Mutation Score – Raw Reduced mutation scores make test strategies look good – Minimal Reduced mutation scores do not ProgramSTMTRORCON5RNDSELECT print_tokens99 : 7898 : 7799 : 7899 : 8299 : 81 print_tokens299 : 4799 : 5699 : 4999 : 4899 : 57 replace97 : 3197 : 3899 : 5799 : 5698 : 48 schedule97 : 6894 : 5398 : 6598 : 6797 : 65 schedule297 : 7292 : 5698 : 7798 : 7297 : 72 tcas88 : 2790 : 3888 : 3394 : 4593 : 43 totinfo97 : 3899 : 5999 : 3999 : 5499 : 60

17 Closer Look: Raw vs. Minimal for STMT Raw mutation scores show little variation Minimal mutation scores show a lot

18 Reduced Mutation: Mutants vs. Tests Notes: – Table entries: Number of Mutants : Size of Minimal Test Set – Reduced approaches Generate many more mutants than minimal But not nearly enough tests ProgramSTMTRORCON5RNDSELECTMinimal print_tokens196 : 1198 : 9358 : : : 1028 : 12.4 print_tokens2203 : 5192 : 8445 : 8198 : 9244 : 930 : 12.1 replace219 : : : : : 3558 : 44.4 schedule127 : 1049 : 778 : 1395 : 1284 : 1042 : 14.5 schedule2117 : 975 : 6119 : : : 1246 : 17.1 tcas42 : 1245 : 1466 : 1499 : : 1861 : 41.4 totinfo110 : 6167 : : : : 1519 : 13.3

19 Closer Look: Mutants and Tests for STMT STMT usually generates too many mutants Unfortunately, they aren’t the right ones – Hence, not nearly enough tests

20 Discussion Huge gap: Reduced mutation vs. minimal mutant sets – Research opportunity! The problem with reduced mutation – Reduced approaches don’t consider specific program under test – Maybe it’s time to change that Can we analyze specific mutants in a specific program? Problem with minimal mutant sets for practical testing – Need mutation adequate tests to compute minimal mutant sets! Aren’t we done at that point? There is a lot we don’t know about minimal mutant sets – Let’s look at an example from yesterday’s Mutation workshop

Subsumption Graph Example: cal() 31 nodes of indistinguished mutants 7 nodes of minimal mutants – muJava generated 145 non-equivalent mutants – we only need 7 for given test set Static analysis can refine this graph 21

22 Questions? Contact: – {pammann, –