Download presentation
Presentation is loading. Please wait.
Published bySophie Park Modified over 9 years ago
1
Theory and Practice, Do They Match ? A Case with Spectrum-Based Fault Localization Tien-Duy B. Le, Ferdian Thung, and David Lo School of Information Systems Singapore Management University 1
2
Spectrum-Based Fault Localization Locating buggy program elements by –Analyzing two sets of execution traces Normal traces and faulty traces –Assigning suspiciousness scores to program elements Two well-known SBFL formulas –Tarantula –Ochiai 2
3
Spectrum-Based Fault Localization Xie et al. “A Theoretical Analysis of the Risk Evaluation Formulas for Spectrum-based Fault Localization” (TOSEM, 2013) –Two families of SBFL formulas ER1 and ER5 (5 formulas in total) Theoretically proven to outperform Ochiai and Tarantula –Under the assumption: test coverage is 100% 3
4
Our Goal 4 vs. Benchmark Programs Theoretically Best SBFL Formula by Xie et. al. Popular SBFL Formula ?
5
Notations NotionDescription Number of successful test cases that execute e Number of failing test cases that execute e Number of successful test cases that do not execute e Number of failing test cases that do not execute e 5
6
Popular SBFL Formulas Tarantula Ochiai 6
7
Theoretically Best SBFL Formulas 7
8
Dataset 10 programs, 199 faulty versions –Siemens test suite –Space, NanoXML, XML-Security Evaluation Metric –The lower the EXAM score, the better the performance 8
9
Results TechniqueAverage % InspectedStandard Deviation Tarantula23.37%23.44% Ochiai21.02%21.96% ER1 a 33.34%35.22% ER1 b 21.09%19.48% ER5 a 43.04%19.63% ER5 b 43.04%19.63% ER5 c 54.95%26.83% 9 Ochiai has the lowest EXAM score (21.02%)
10
Results TechniqueAverage % InspectedStandard Deviation Tarantula23.37%23.44% Ochiai21.02%21.96% ER1 a 33.34%35.22% ER1 b 21.09%19.48% ER5 a 43.04%19.63% ER5 b 43.04%19.63% ER5 c 54.95%26.83% 10 Tarantula‘s EXAM score is lower than 4 out of the 5 theoretically best SBFL formulas
11
Results TechniqueAverage % InspectedStandard Deviation Tarantula23.37%23.44% Ochiai21.02%21.96% ER1 a 33.34%35.22% ER1 b 21.09%19.48% ER5 a 43.04%19.63% ER5 b 43.04%19.63% ER5 c 54.95%26.83% 11 Wilcoxon signed rank test (significance level of 0.05) Ochiai is statistically better than ER5 a, ER5 b, ER5 c
12
100% Test Coverage Assumption For 135 out of the 199 faulty versions –Test coverage < 100% Average test coverage of the 199 versions –84.97% Theoretically best SBFL formulas cannot outperform popular SBFL formulas 12
13
Conclusion We conduct an empirical study on 10 programs with 199 versions –Compare performance of 5 theoretically best SBFL formulas with Tarantula and Ochiai We find that: –Ochiai outperforms all theoretically best formulas –Tarantula outperforms 4 out of the 5 formulas –Assumption of 100% test coverage is not valid in many cases 13
14
Future work In-depth study how test coverage and other factors affect effectiveness of SBFL formulas Theoretically analyze performance of SBFL formulas –Assumption: test coverage < 100% 14
15
15 Thank you! Questions? Comments? Advice? {btdle.2012, ferdiant.2013, davidlo}@smu.edu.sg
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.