Download presentation
Presentation is loading. Please wait.
Published byErnest Clark Modified over 9 years ago
1
Yue Jia, Mark Harman King’s College London CREST Constructing Subtle Faults Using Higher Order Mutation Testing Higher Order Mutation Testing
2
Agenda Background Problems Solution Empirical Study Summary Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionEmpirical StudySummary
3
Mutation Testing Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background
4
Mutation Testing Which test case is better? Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background
5
Mutation Testing Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background
6
Mutation Testing Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background
7
Mutation Testing Generate Mutants ----------------- ----------------- ----------------- + > First Order MutantOriginal Program Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background
8
Mutation Testing Generate Mutants ----------------- ----------------- ----------------- + > + > Higher Order MutantFirst Order MutantOriginal Program Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background
9
Mutation Testing Generate Mutants ----------------- ----------------- ----------------- + > + > Higher Order MutantFirst Order MutantOriginal Program ----------------- ----------------- ----------------- + > ----------------- ----------------- ----------------- ----------------- ----------------- ---------------- + ----------------- ----------------- ----------------- > First Order MutantHigher Order Mutant Mark Harman and Yue Jia King’s College London, CREST Centre ProblemsSolutionEmpirical StudySummary Background
10
Problem 1970s Computational cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
11
Problem 1970s Computational cost e.g. Triangle LoC 50 FOM 500 HOM 2 500 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
12
Problem 1970s Computational cost e.g. Triangle LoC 50 FOM 500 HOM 2 500 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
13
Problem 1970s Computational cost e.g. Triangle LoC 50 FOM 500 HOM 2 500 cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionEmpirical StudySummary Problems
14
Solution 1980s Coupling effect hypothesis cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
15
Solution 1980s Coupling effect hypothesis cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
16
Solution 1980s Simple Complex cost If a test set kills simple faults, it also kills the complex faults Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
17
Solution 1980s Coupling effect hypothesis cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
18
Solution 1990s Selective mutation / Mutation sampling cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
19
Selected First Order Mutants + + - - > > < < a a b b Trivial Fault Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
20
Subtle Fault 90% of the real faults are complex faults (Purushothaman and Perry) ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ---------------------------- HOMs -> Subtle faults Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
21
Subtle Fault Solution Programmer Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
22
Subtle Fault Solution Programmer 1 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
23
Subtle Fault Solution Programmer 2 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
24
Subtle Fault Solution Programmer 3 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
25
Subtle Fault 4 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems Solution Programmer
26
Solution 1990s Selective mutation / Mutation sampling cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
27
Solution 1980s Coupling effect hypothesis cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
28
cost Problem 1970s Computational cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundSolutionEmpirical StudySummary Problems
29
Solution Higher order mutation testing cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution
30
Higher order mutation testing …Search Based Optimization cost Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution
31
Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Reduced number of equivalent mutants Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution
32
Higher Order Mutant Most common case Test set T Ta Tb Tc Tabc Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution
33
Higher Order Mutant Uncommon case Ta Tb Tc Tabc Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution
34
Higher Order Mutant Subsuming higher order mutant Ta Tb Tc Tabc Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution
35
Higher Order Mutant Subsuming higher order mutant Ta Tb Tc Tabc Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution
36
Higher Order Mutant Strongly Subsuming higher order mutant Ta Tb Tc Tabc If a test case kills HOM M abc, it also kills the FOM F a, F b and F c Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution
37
Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Reduced number of equivalent mutants Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution
38
Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Mutants Reduced number of equivalent mutants Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution
39
Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Mutants Test cases Reduced number of equivalent mutants Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution
40
Higher Order Mutation Testing Increase the subtlety Reduce the Oracle effort Mutants Test cases Reduced number of equivalent mutants Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsEmpirical StudySummary Solution
41
Results Mutants LoC FOM sHOM Triangle 50 584 47 Tcas 150 679 98 Schedule2 350 1014 78 Totinfo 500 2570 320 Printtokens 750 866 67 Space 6000 7570 522 Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolution Empirical Study Summary
42
Algorithms Greedy Hill Climbing Genetic Algorithm Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study
43
Results Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study
44
Triangle Program Classify triangle by the lengths of the sides Equilateral Isosceles Scalene Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study
45
Strongly Subsuming HOM if(trian == 1 && a + b > c) if(trian > 1 && a + b > c) if(trian == 1 && a + b <= c) a == c && a + b > c && a + c <= b b == c && a + b > a && b + c <= a a == b && a + b > c a == b && a + b <= c Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study
46
Tool MiLu Deer Horse Cow Donkey rarevaluable Strongly Subsuming HOM (Père David's Deer) Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study
47
www.dcs.kcl.ac.uk/pg/jiayue/milu Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionSummary Empirical Study
48
Summary Higher order mutation testing Quality of mutant Computational Cost Question Are real faults higher order mutants? Thanks Mark Harman and Yue Jia King’s College London, CREST Centre BackgroundProblemsSolutionEmpirical Study Summary
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.