Locating Causes of Program Failures Texas State University CS 5393 Software Quality Project Yin Deng

2 Topics  Introduction  What is the problem?  Overview of major solutions  A Sample Failure  Case Study  Complexity and other issues  Conclusion  Related Material

3 Introduction  Locating Causes of Program Failures Holger Cleve and Andreas Zeller ICSE 2005, research papers on Fault Localization  Holger Cleve is one of the members in software engineering research group at Saarland University in Germany.  Andreas Zeller is a full professor and the chair of software engineering research group at Saarland University. His research in SE concerns especially the analysis of why large, complex software systems fail to work as they should.

4 What ’ s the Problem?  Definitions Failure: A program ’ s behavior doesn ’ t satisfy its requirement specification. Fault / Infection: An incorrect intermediate state that may be entered during program execution. Failure  Infection  Defect in code, but not vice versa.  Problem Why does program fail? How to find the defects that cause a software failure?

5 Overview of major solutions  Searching in Space Across a program state to find the infected variable(s), often among thousands. Focus on the difference between the program states where the failure occurs, and the states where the failure does not occur. Using Delta Debugging, those initial differences can be systematically narrowed down to a small set of variables.  Searching in Time Search over millions of program states to find the moment when the defect was executed. Focus on cause transitions (CTS)!

6 Searching in Space  Compare the program states of a passing run r and a falling run r  at a certain moment.  Of all different states, only some may be relevant for the failure.  How to find a subset of relevant variables that is as small as possible?  Delta Debugging, which behaves very much like a binary search.

7 Searching in Time  A cause transition is where a cause originates. It points to program code that causes the transition and hence the failure.  During transitions, some variables cease to be a failure cause and other variables begin.  Cause transitions are not only good locations for fixes, they actually locate the defects that cause the failure.

8 Example – Source Code 26 int main(int argc, char *argv[]) 27 { 28 int i = 0; 29 int *a = NULL; a = (int *)malloc((argc - 1) * sizeof(int)); 32 for (i = 0; i < argc - 1; i++) 33 a[i] = atoi(argv[i + 1]); shell_sort(a, argc); for (i = 0; i < argc - 1; i++) 38 printf("%d ", a[i]); 39 printf("\n"); free(a); 42 return 0; 43 } 1 /* sample.c -- Sample C program */ 2 3 #include 4 #include 5 6 static void shell_sort(int a[], int size) 7 { 8 int i, j; 9 int h = 1; 10 do { 11 h = h * 3 + 1; 12 } while (h <= size); 13 do { 14 h /= 3; 15 for (i = h; i < size; i++) 16 { 17 int v = a[i]; 18 for (j = i; j >= h && a[j - h] > v; j -= h) 19 a[j] = a[j - h]; 20 if (i != j) 21 a[j] = v; 22 } 23 } while (h != 1); 24 } 25

9 Example – Running Result  A passing run r $ sample  A falling run r  $ sample What’s wrong?

10 Example – Searching in Space State differences between r and r . One of these differences causes sample to fail.

11 Example – Searching in Space (cont.)  Procedures Runs r up to Line 9 Applies half of the differences on r Resumes execution and determines the outcome.  Result Line 9, a[2] being zero causes the sample failure. What causes a[2] be zero?

12 Example – Searching in Time

13 Example – Searching in Time (cont.)  Procedures Find an interval of matches to start with; there was a cause transition between argc in step 1 and a[0] in Step 44; Use Delta Debugging to find relevant variables between argc and a[0] (function calls are preferred), a[2] is isolated; CTS : Step 26 (a[2] again); CTS : Step 35 (v).  Result argc  a[2] in Lines 32 – 35 (Steps 8 – 11); a[2]  v in Line 17 (Step 29); v  a[0] in Line 21 (Step 36).

14 Example – Debugging Result

15 Case Study: The GCC Failure The program that crashes GCC

16 Complexity  Searching in space Best case: Delta Debugging needs 2s log k test runs to isolate s failure-inducing variables from k state differences. worst case is k 2 + 3k In practice, Delta Debugging is much more logarithmic than linear.  Searching in time A simple binary search over n program steps, repeated for each cause transition. For m cause transitions, we need m log n runs of Delta Debugging.

17 Practical Issues  Accessing state Currently using GDB, which is painfully slow; More efficient ways need to be explored.  Capturing accurate states Several heuristics are used to determine state transferring; When such heuristics fail, the state cannot be transferred.  Incomparable states When control flow reaches different points in r and r , the resulting states are not comparable, simply because the set of local variables is different. Some efforts are required to determine when the control flows of r and r  diverge and converge.

18 Conclusion  Cause transitions locate the software defect that causes a given failure, performing twice as well as any other technique previously known.  The technique requires an automated test, a mean to observe and manipulate the program state, as well as at least one alternate passing test run.  The technique could be used as an add-on to running an automated test suite; we not only know that a test has failed, but also why and where it failed.

19 Related Material  Isolating cause-effect chains from computer programs. A. Zeller. In W. G. Griswold, editor, Proc. Tenth ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-10), pages 1 – 10, Charleston, South Carolina, Nov ACM Press.  Simplifying and isolating failure-inducing input. A. Zeller and R. Hildebrandt. IEEE Transactions on Software Engineering, 28(2):183 – 200, Feb  Visualizing memory graphs. T. Zimmermann and A. Zeller. In S. Diehl, editor, Proc. of the International Dagstuhl Seminar on Software Visualization, volume 2269 of Lecture Notes in Computer Science, pages 191 – 204, Dagstuhl, Germany, May Springer-Verlag.  Why Programs Fail: A Guide to Systematic Debugging. A. Zeller. Morgan Kaufmann Publisher, October, ISBN

20 Any Question?