Theory and Practice, Do They Match ? A Case with Spectrum-Based Fault Localization Tien-Duy B. Le, Ferdian Thung, and David Lo School of Information Systems Singapore Management University 1

Spectrum-Based Fault Localization Locating buggy program elements by –Analyzing two sets of execution traces Normal traces and faulty traces –Assigning suspiciousness scores to program elements Two well-known SBFL formulas –Tarantula –Ochiai 2

Spectrum-Based Fault Localization Xie et al. “A Theoretical Analysis of the Risk Evaluation Formulas for Spectrum-based Fault Localization” (TOSEM, 2013) –Two families of SBFL formulas ER1 and ER5 (5 formulas in total) Theoretically proven to outperform Ochiai and Tarantula –Under the assumption: test coverage is 100% 3

Our Goal 4 vs. Benchmark Programs Theoretically Best SBFL Formula by Xie et. al. Popular SBFL Formula ?

Notations NotionDescription Number of successful test cases that execute e Number of failing test cases that execute e Number of successful test cases that do not execute e Number of failing test cases that do not execute e 5

Popular SBFL Formulas Tarantula Ochiai 6

Theoretically Best SBFL Formulas 7

Dataset 10 programs, 199 faulty versions –Siemens test suite –Space, NanoXML, XML-Security Evaluation Metric –The lower the EXAM score, the better the performance 8

Results TechniqueAverage % InspectedStandard Deviation Tarantula23.37%23.44% Ochiai21.02%21.96% ER1 a 33.34%35.22% ER1 b 21.09%19.48% ER5 a 43.04%19.63% ER5 b 43.04%19.63% ER5 c 54.95%26.83% 9 Ochiai has the lowest EXAM score (21.02%)

Results TechniqueAverage % InspectedStandard Deviation Tarantula23.37%23.44% Ochiai21.02%21.96% ER1 a 33.34%35.22% ER1 b 21.09%19.48% ER5 a 43.04%19.63% ER5 b 43.04%19.63% ER5 c 54.95%26.83% 10 Tarantula‘s EXAM score is lower than 4 out of the 5 theoretically best SBFL formulas

Results TechniqueAverage % InspectedStandard Deviation Tarantula23.37%23.44% Ochiai21.02%21.96% ER1 a 33.34%35.22% ER1 b 21.09%19.48% ER5 a 43.04%19.63% ER5 b 43.04%19.63% ER5 c 54.95%26.83% 11 Wilcoxon signed rank test (significance level of 0.05)  Ochiai is statistically better than ER5 a, ER5 b, ER5 c

100% Test Coverage Assumption For 135 out of the 199 faulty versions –Test coverage < 100% Average test coverage of the 199 versions –84.97%  Theoretically best SBFL formulas cannot outperform popular SBFL formulas 12

Conclusion We conduct an empirical study on 10 programs with 199 versions –Compare performance of 5 theoretically best SBFL formulas with Tarantula and Ochiai We find that: –Ochiai outperforms all theoretically best formulas –Tarantula outperforms 4 out of the 5 formulas –Assumption of 100% test coverage is not valid in many cases 13

Future work In-depth study how test coverage and other factors affect effectiveness of SBFL formulas Theoretically analyze performance of SBFL formulas –Assumption: test coverage < 100% 14

15 Thank you! Questions? Comments? Advice? {btdle.2012, ferdiant.2013,