1. Establishing Theoretical Minimal Sets of Mutants ICST 2014 Paul Ammann Joint work with Marcio Eduardo Delamaro Jeff Offutt April 1, 2014

2. 2 Outline The situation – Researchers use mutation analysis to evaluate test selection strategies The problem – What do mutation scores mean? The model – Motivating idea: Minimal mutant sets don’t have redundant mutants Need to define notion of redundancy – Main result: Dynamic subsumption = Minimal mutant sets Reduced mutation: Is it close to minimal? – Apply model to Siemens suite Result: Huge gap – Good news: That’s an opportunity!

3. Researchers Use Mutation Analysis to Evaluate Test Selection Strategies Test Selection Strategy C Test Selection Strategy B Test Selection Strategy A Carefully Chosen Artifacts “Good” Tests Deep Analysis Test Set C Select Test Sets with Test Selection Strategies Test Set B Test Set A Measure Exactly What Does A Score of 91% Mean? 3

4. The Problem With Mutation Scores m1m1 m2m2 m3m3 m4m4 t1t1 t2t2 t3t3 t4t4 t5t5 Evaluate 3 Test Sets with 4 Mutants: A: {t 1, t 2 } B : {t 2, t 5 } C: {t 3 } Mutation Scores for 3 Test Sets B scores 75% Is that good? 4

5. Let’s Add Some More Mutants m1m1 m2m2 m3m3 m4m4 m5m5 m6m6 m7m7 m8m8 m9m9 m 10 t1t1 t2t2 t3t3 t4t4 t5t5 Evaluate 3 Test Sets with 10 Mutants: A: {t 1, t 2 } B: {t 2, t 5 } C : {t 3 } Mutation Scores for 3 Test Sets Now B scores 90%! Did B just get better? Every test kills m 8 What’s the point? The same tests kill m 3 and m 6. We say that T does not distinguish m 3 from m 6 Ditto for m 5 and m 9 5

6. Let’s Throw Away Some Mutants m1m1 m2m2 t1t1 t2t2 t3t3 t4t4 t5t5 Evaluate 3 Test Sets with 2 Mutants: A: {t 1, t 2 } B : {t 2, t 5 } C : {t 3 } Mutation Scores for 3 Test Sets Now B scores 100% Did B get even better? 6

7. All Together Now m1m1 m2m2 m3m3 m4m4 m5m5 m6m6 m7m7 m8m8 m9m9 m 10 t1t1 t2t2 t3t3 t4t4 t5t5 Evaluate 3 Test Sets with Various Mutants: A: {t 1, t 2 } B : {t 2, t 5 } C: {t 3 } Cumulative Scores Is B lousy or good? What about C? 7

8. What Makes a Mutant Redundant? m1m1 m2m2 m3m3 m4m4 t1t1 t2t2 t3t3 t4t4 t5t5 Choose M = {m 1, m 2, m 3, m 4 } Choose T = {t 1, t 2, t 3, t 4, t 5 } Minimal test sets wrt M and T: {t 1, t 2 }, {t 1, t 3 }, {t 4 } Try removing m 1 : M 1 = M - {m 1 } Minimal test sets wrt M 1 and T: {t 1, t 2 }, {t 1, t 3 }, {t 4 } No change, so m 1 is redundant Basic Idea: Throwing away a redundant mutant has no effect on the minimal test sets. Try removing m 3 : M 3 = M - {m 3 } Minimal test sets wrt M 3 and T: {t 1 }, {t 4 } A change, so m 3 is not redundant Try removing m 4 : M 4 = M - {m 4 } Minimal test sets wrt M 4 and T: {t 1, t 2 }, {t 1, t 3 }, {t 2, t 5 }, {t 3, t 5 }, {t 4 } A change, so m 4 is not redundant Ditto for M 2 = M - {m 2 } 8

9. 9 Minimal Sets of Mutants Definition – M is minimal if it does not contain redundant mutants Minimal mutant sets from the definition – Requires computing all minimal test sets, which is NP complete  We need an efficient algorithm for finding minimal mutant sets – Turn to dynamic subsumption Subsumption with respect to a test set

10. Dynamic Subsumption 10 Test set T Tests that kill m j Tests that kill m i Tests that kill m k m i → m j m i → m k ✔ ✖ ? ?

11. 11 Efficiently Computing Minimal Sets of Mutants Formally: m x dynamically subsumes m y wrt T iff – Some test in T kills m x – Every test in T that kills m x also kills m y Main result: Mutant set M minimal wrt T = no dynamic subsumption in M Properties – Only need to consider mutants in pairs Groups of mutants do not make another mutant redundant – Fast – Every minimal mutant set has the same cardinality Contrast with minimal test sets

12. 12 What Does This Mean in Practice? Apply the definitions to the Siemens test bed – See what happens! 7 programs – print_tokens – print_tokens2 – replace – schedule – schedule2 – tcas – totinfo Extensive hand-crafted test set

13. 13 Test Characteristics Notes: – 512 is an artifact of the Proteum tool Our approach applies with any test set – Most tests used were also distinguished – Minimal test set size modest compared to number of tests used ProgramTests Available Tests Used Distinguished Tests Minimal Tests print_tokens407351249912.4 print_tokens2405851247912.1 replace554251251044.4 schedule265051248214.5 schedule2105251247917.1 tcas160851242841.4 totinfo407351245213.3

14. 14 Mutant Characteristics “Killed Mutants” means those killed by the test set of size 512 Vast majority of remainder are equivalent ProgramTotal Mutants Killed Mutants Distinguished Mutants Minimal Mutants print_tokens4336371143728 print_tokens24746404243930 replace111018783230958 schedule2109183852042 schedule22627213246146 tcas2384195759661 totinfo6698582183519 Most mutants are redundant! –Tiny fraction of mutants are actually minimal wrt 512 tests! –print_tokens: Killing the right 28 mutants guarantees killing all 3711

15. 15 How Good Are Reduced Mutation Strategies? We considered five approaches to reduced mutation – STMT: Statement deletion (Proteum SSDL) – ROR: Relation operators (Proteum ORRN) – CON: Replace scalars with constants (Proteum CCSR) – 5RND: 5% Random selection of all mutants – SELECT: “Selective” mutation (Proteum OOAN, OLLN, ORRN, OLNG) Method: – Choose test sets adequate for each reduced mutation approach wrt test sets analyzed earlier – Compute mutation score Against all mutants Against minimal mutant set Equivalent mutants hand-identified and removed

16. 16 Reduced Mutation Scores: Raw vs. Minimal Notes: – Table entries: Raw Mutation Score: Minimal Mutation Score – Raw Reduced mutation scores make test strategies look good – Minimal Reduced mutation scores do not ProgramSTMTRORCON5RNDSELECT print_tokens99 : 7898 : 7799 : 7899 : 8299 : 81 print_tokens299 : 4799 : 5699 : 4999 : 4899 : 57 replace97 : 3197 : 3899 : 5799 : 5698 : 48 schedule97 : 6894 : 5398 : 6598 : 6797 : 65 schedule297 : 7292 : 5698 : 7798 : 7297 : 72 tcas88 : 2790 : 3888 : 3394 : 4593 : 43 totinfo97 : 3899 : 5999 : 3999 : 5499 : 60

17. 17 Closer Look: Raw vs. Minimal for STMT Raw mutation scores show little variation Minimal mutation scores show a lot

18. 18 Reduced Mutation: Mutants vs. Tests Notes: – Table entries: Number of Mutants : Size of Minimal Test Set – Reduced approaches Generate many more mutants than minimal But not nearly enough tests ProgramSTMTRORCON5RNDSELECTMinimal print_tokens196 : 1198 : 9358 : 10190 : 11138 : 1028 : 12.4 print_tokens2203 : 5192 : 8445 : 8198 : 9244 : 930 : 12.1 replace219 : 23264 : 271053 : 44443 : 39499 : 3558 : 44.4 schedule127 : 1049 : 778 : 1395 : 1284 : 1042 : 14.5 schedule2117 : 975 : 6119 : 13110 : 13121 : 1246 : 17.1 tcas42 : 1245 : 1466 : 1499 : 24113 : 1861 : 41.4 totinfo110 : 6167 : 13469 : 12294 : 15332 : 1519 : 13.3

19. 19 Closer Look: Mutants and Tests for STMT STMT usually generates too many mutants Unfortunately, they aren’t the right ones – Hence, not nearly enough tests

20. 20 Discussion Huge gap: Reduced mutation vs. minimal mutant sets – Research opportunity! The problem with reduced mutation – Reduced approaches don’t consider specific program under test – Maybe it’s time to change that Can we analyze specific mutants in a specific program? Problem with minimal mutant sets for practical testing – Need mutation adequate tests to compute minimal mutant sets! Aren’t we done at that point? There is a lot we don’t know about minimal mutant sets – Let’s look at an example from yesterday’s Mutation workshop

21. Subsumption Graph Example: cal() 31 nodes of indistinguished mutants 7 nodes of minimal mutants – muJava generated 145 non-equivalent mutants – we only need 7 for given test set Static analysis can refine this graph 21

22. 22 Questions? Contact: – {pammann, offutt}@gmu.edu – delamaro@icmc.usp.br