1. Fault Location via State Alteration CS 206 Fall 2009

2. 2 Value Replacement: Overview INPUT: Faulty program and test suite (1+ failing runs) TASK: (1) Perform value replacements in failing runs (2) Rank program statements according to collected information OUTPUT: Ranked list of program statements Aggressive state alteration to locate faulty program statements [Jeffrey et. al., ISSTA 2008]

3. 3 Alter State by Replacing Values Passing ExecutionFailing Execution Correct Output Failing Execution: Altered State ERROR REPLACE VALUES ERROR Incorrect Output Correct? / Incorrect?

4. 4 1: read (x, y); 2: a := x - y; 3: if (x < y) 4: write (a); else 5: write (a + 1); Example of a Value Replacement (output: ?)PASSING EXECUTION: 1 10 0 1 1 (F) 1 2 3 5 1 1 1

5. 5 Example of a Value Replacement FAILING EXECUTION:(expected output: 1) (actual output: ?) 1: read (x, y); 2: a := x + y; 3: if (x < y) 4: write (a); else 5: write (a + 1); ERROR: plus should be minus 1 1 1 12 2 1 1 (F) 1 2 3 5 ERROR 3

6. 6 Example of a Value Replacement STATE ALTERATION: 1: read (x, y); 2: a := x + y; 3: if (x < y) 4: write (a); else 5: write (a + 1); 1 1 1 2 1 12 ERROR (expected output: 1) (actual output: ?) Original Values 0 11 Alternate Values REPLACE VALUES 1 1 0 1 (T) 3 4 Interesting Value Mapping Pair (IVMP): Location: statement 2, instance 1 Original: {a = 2, x = 1, y = 1} Alternate: {a = 1, x = 0, y = 1}

7. 7 Searching for IVMPs in a Failing Run Step 1: Compute the Value Profile Set of values used at each statement with respect to all available test case executions Step 2: Replace values to search for IVMPs For each statement instance in failing run For each alternate set of values in value profile Replace values to see if an IVMP is found Endfor

8. 8 Searching for IVMPs: Example 1: read (x, y); 2: a := x + y; // + should be – 3: if (x < y) 4: write (a); else 5: write (a + 1); Test Case (x, y) Actual Output Expected Output (1, 1)31 (-1, 0) (0, 0)11 x = 1 y = 1 \ -1 \ 0 VALUE REPLACEMENT RESULTING OUTPUT IVMP? 1: read (x, y); VALUE PROFILE a = 2 output = 3 x = 0 y = 0 branch = F x = -1 y = 0 branch = T x = 1 y = 1 branch = F x = 0 y = 0 a = 0 x = -1 y = 0 a = -1 x = 1 y = 1 a = 2 x = 0 y = 0 x = -1 y = 0 x = 1 y = 1 1: 2: 3: 4: 5: a = -1 output = -1 a = 0 output = 1 (1,1)(-1,0) (0,0)

9. 9 Searching for IVMPs: Example 1: read (x, y); 2: a := x + y; // + should be – 3: if (x < y) 4: write (a); else 5: write (a + 1); Test Case (x, y) Actual Output Expected Output (1, 1)31 (-1, 0) (0, 0)11 1: read (x, y); IVMPs Identified: stmt 1, inst 1: ( {x=1, y=1}  {x=0, y=0} ) x = 1 y = 1 \ 0 1 VALUE REPLACEMENT RESULTING OUTPUT IVMP? a = 2 output = 3 x = 0 y = 0 branch = F x = -1 y = 0 branch = T x = 1 y = 1 branch = F x = 0 y = 0 a = 0 x = -1 y = 0 a = -1 x = 1 y = 1 a = 2 x = 0 y = 0 x = -1 y = 0 x = 1 y = 1 1: 2: 3: 4: 5: VALUE PROFILE a = -1 output = -1 a = 0 output = 1

10. 10 Searching for IVMPs: Example 1: read (x, y); 2: a := x + y; // + should be – 3: if (x < y) 4: write (a); else 5: write (a + 1); Test Case (x, y) Actual Output Expected Output (1, 1)31 (-1, 0) (0, 0)11 IVMPs Identified: stmt 1, inst 1: ( {x=1, y=1}  {x=0, y=0} ) 2: a := x + y; stmt 2, inst 1: ( {x=1, y=1, a=2}  {x=0, y=0, a=0} ) x = 1 y = 1 a = 2 \ -1 \ 0 \ -1 x = 1 y = 1 a = 2 \ 0 a = 2 output = 3 x = 0 y = 0 branch = F x = -1 y = 0 branch = T x = 1 y = 1 branch = F x = 0 y = 0 a = 0 x = -1 y = 0 a = -1 x = 1 y = 1 a = 2 x = 0 y = 0 x = -1 y = 0 x = 1 y = 1 1: 2: 3: 4: 5: VALUE PROFILE a = -1 output = -1 a = 0 output = 1

11. 11 Searching for IVMPs: Example 1: read (x, y); 2: a := x + y; // + should be – 3: if (x < y) 4: write (a); else 5: write (a + 1); Test Case (x, y) Actual Output Expected Output (1, 1)31 (-1, 0) (0, 0)11 IVMPs Identified: stmt 1, inst 1: ( {x=1, y=1}  {x=0, y=0} ) 3: if (x < y) stmt 2, inst 1: ( {x=1, y=1, a=2}  {x=0, y=0, a=0} ) x = 1 y = 1 branch = F \ -1 \ 0 \ T x = 1 y = 1 branch = F \ 0 \ F a = 2 output = 3 x = 0 y = 0 branch = F x = -1 y = 0 branch = T x = 1 y = 1 branch = F x = 0 y = 0 a = 0 x = -1 y = 0 a = -1 x = 1 y = 1 a = 2 x = 0 y = 0 x = -1 y = 0 x = 1 y = 1 1: 2: 3: 4: 5: VALUE PROFILE a = -1 output = -1 a = 0 output = 1

12. 12 Searching for IVMPs: Example 1: read (x, y); 2: a := x + y; // + should be – 3: if (x < y) 4: write (a); else 5: write (a + 1); Test Case (x, y) Actual Output Expected Output (1, 1)31 (-1, 0) (0, 0)11 IVMPs Identified: stmt 1, inst 1: ( {x=1, y=1}  {x=0, y=0} ) 5: write (a + 1); stmt 2, inst 1: ( {x=1, y=1, a=2}  {x=0, y=0, a=0} ) stmt 5, inst 1: ( {a=2, output=3}  {a=0, output=1} ) a = 2 output = 3 \ 0 \ 1 a = 2 output = 3 x = 0 y = 0 branch = F x = -1 y = 0 branch = T x = 1 y = 1 branch = F x = 0 y = 0 a = 0 x = -1 y = 0 a = -1 x = 1 y = 1 a = 2 x = 0 y = 0 x = -1 y = 0 x = 1 y = 1 1: 2: 3: 4: 5: VALUE PROFILE a = -1 output = -1 a = 0 output = 1

13. 13 Searching for IVMPs: Example 1: read (x, y); 2: a := x + y; // + should be – 3: if (x < y) 4: write (a); else 5: write (a + 1); Test Case (x, y) Actual Output Expected Output (1, 1)31 (-1, 0) (0, 0)11 IVMPs Identified: stmt 1, inst 1: ( {x=1, y=1}  {x=0, y=0} ) stmt 2, inst 1: ( {x=1, y=1, a=2}  {x=0, y=0, a=0} ) stmt 5, inst 1: ( {a=2, output=3}  {a=0, output=1} ) DONE a = 2 output = 3 x = 0 y = 0 branch = F x = -1 y = 0 branch = T x = 1 y = 1 branch = F x = 0 y = 0 a = 0 x = -1 y = 0 a = -1 x = 1 y = 1 a = 2 x = 0 y = 0 x = -1 y = 0 x = 1 y = 1 1: 2: 3: 4: 5: VALUE PROFILE a = -1 output = -1 a = 0 output = 1

14. 14 IVMPs at Non-Faulty Statements Causes of IVMPs at non-faulty statements Statements in same dependence chain Coincidence Consider multiple failing runs Stmt w/ IVMPs in more runs  more likely to be faulty Stmt w/ IVMPs in fewer runs  less likely to be faulty

15. 15 {1, 2} {4, 5} {3} MOST LIKELY TO BE FAULTY LEAST LIKELY TO BE FAULTY Multiple Failing Runs: Example 1: read (x, y); 2: a := x + y; 3: if (x < y) 4: write (a); else 5: write (a + 1); Test Case (x, y)Actual OutputExpected Output (1, 1)31 (0, 1)1 (-1, 0) (0, 0)11 [A] [B] [C] [D] Test Case [A] IVMPs: stmt 1, inst 1: ( {x=1, y=1}  {x=0, y=1} ) stmt 1, inst 1: ( {x=1, y=1}  {x=0, y=0} ) stmt 2, inst 1: ( {x=1, y=1, a=2}  {x=0, y=1, a=1} ) stmt 2, inst 1: ( {x=1, y=1, a=2}  {x=0, y=0, a=0} ) stmt 5, inst 1: ( {a=2, output=3}  {a=0, output=1} ) stmts with IVMPs: {1, 2, 5} 1: read (x, y); 2: a := x + y; 5: write (a + 1); Test Case [B] IVMPs: stmt 1, inst 1: ( {x=0, y=1}  {x=-1, y=0} ) stmt 2, inst 1: ( {x=0, y=1,a=1}  {x=-1, y=0,a=-1} ) stmt 4, inst 1: ( {a=1, output=1}  {a=-1, output=-1} ) stmts with IVMPs: {1, 2, 4} 2: a := x + y; 4: write (a); 1: read (x, y);

16. 16 Ranking Statements using IVMPs Sort in decreasing order of: Break ties using Tarantula technique [Jones et. al., ICSE 2002] The number of failing runs in which the statement is associated with at least one IVMP fraction of failing runs exercising stmt fraction of passing runs exercising stmt fraction of failing runs exercising stmt +

17. 17 Techniques Evaluated Value Replacement technique Consider all available failing runs (ValRep-All) Consider only 2 failing runs (ValRep-2) Consider only 1 failing run (ValRep-1) Tarantula technique (Tarantula) Consider all available test cases Most effective technique known for our benchmarks Only rank statements exercised by failing runs

18. 18 Score for each ranked statement list Represents percentage of statements that need not be examined before error is located Higher score is better Metric for Comparison size of list rank of the faulty stmt 100% x size of list High Suspiciousness Low Suspiciousness High Suspiciousness Low Suspiciousness Higher Score Lower Score

19. 19 Benchmark Programs ProgramLOC# Faulty Ver.Avg. Suite Size (Pool Size) tcas1384117 (1608) totinfo3462315 (1052) sched299920 (2650) sched2297917 (4130) ptok402717 (4130) ptok2483923 (4115) replace5163129 (5542) 129 faulty programs (errors) derived from 7 base programs Each faulty program is associated with a branch-coverage adequate test suite containing at least 5 failing and 5 passing test cases Test suite used by Value Replacement, test pool used by Tarantula

20. 20 Effectiveness Results Number (%) of faulty programs ScoreValRep-AllVal-Rep-2ValRep-1 ≥ 99%23 (17.8%)21 (16.3%)18 (14.0%) ≥ 90%89 (69.0%)84 (65.1%)75 (58.1%) Value Replacement technique

21. 21 Effectiveness Results Comparison to Tarantula Number (%) of faulty programs ScoreValRep-AllVal-Rep-2ValRep-1Tarantula ≥ 99%23 (17.8%)21 (16.3%)18 (14.0%)7 (5.4%) ≥ 90%89 (69.0%)84 (65.1%)75 (58.1%)48 (37.2%)

22. 22 Value Replacement: Summary Highly Effective Precisely locates 39 / 129 errors (30.2%) Most effective previously known: 5 / 129 (3.9%) Limitations Can require significant computation time to search for IVMPs Assumes multiple failing runs are caused by the same error

23. 23 Handling Multiple Errors Effectively locate multiple simultaneous errors [Jeffrey et. al., ICSM 2009] Iteratively compute a ranked list of statements to find and fix one error at a time Three variations of this technique MIN: minimal computation; use same list each time FULL: full computation; produce new list each time PARTIAL: partial computation; revise list each time

24. 24 Multiple-Error Techniques Value Replacement Faulty Program and Test Suite Ranked List of Program Statements Developer Find/Fix Error Done Single Error Faulty Program and Test Suite Ranked List of Program Statements Done Failing Run Remains? No Yes Multiple Errors (MIN) Value Replacement Developer Find/Fix Error

25. 25 Multiple-Error Techniques Faulty Program and Test Suite Ranked List of Program Statements Done Failing Run Remains? No Yes Multiple Errors (FULL) Value Replacement Developer Find/Fix Error Faulty Program and Test Suite Ranked List of Program Statements Done Failing Run Remains? No Yes Multiple Errors (PARTIAL) Partial Value Replacement Developer Find/Fix Error

26. 26 PARTIAL Technique Step 1: Initialize ranked lists and locate first error For each statement s, compute a ranked list by considering only failing runs exercising s Report ranked list with highest suspiciousness value at the front of the list Step 2: Iteratively revise ranked lists and locate each remaining error For each remaining failing run that exercises the statement just fixed, recompute IVMPs Update any affected ranked lists Report ranked list with the most different elements at the front of the list, compared to previously-selected lists

27. 27 PARTIAL Technique: Example 1 2 34 5 Program (2 faulty statements) Failing RunExecution Trace Statements with IVMPs A(1, 2, 3, 5){2, 5} B(1, 2, 3, 5){1, 2} C(1, 2, 4, 5){2, 4, 5} Computed Ranked Lists: (statement suspiciousness ) 1 2 3 4 5 2 3, 5 2, 1 1, 4 1, 3 0 2 2, 1 1, 5 1, 3 0, 4 0 2 1, 4 1, 5 1, 1 0, 3 0 2 3, 5 2, 1 1, 4 1, 3 0 [based on runs A, B, C] [based on runs A, B] [based on run C] [based on runs A, B, C] Report list 1, 2, or 5 (assume 1)  Fix faulty statement 2

28. 28 PARTIAL Technique: Example 1 2 34 5 Program (1 faulty statement) Failing RunExecution Trace Statements with IVMPs C(1, 2, 4, 5){4} Computed Ranked Lists: (statement suspiciousness ) 2 3 4 5 2 2, 1 1, 4 1, 5 1, 3 0 2 2, 1 1, 5 1, 3 0, 4 0 4 1, 1 0, 2 0, 3 0, 5 0 2 2, 1 1, 4 1, 5 1, 3 0 [based on runs A, B, C] (C updated) [based on runs A, B] (no updates) [based on run C] (C updated) [based on runs A, B, C] (C updated) Report list 4  Fix faulty statement 4  Done

29. 29 Techniques Compared (MIN) Only compute ranked list once (FULL) Fully recompute ranked list each time (PARTIAL) Compute IVMPs for subset of failing runs and revise ranked lists each time (ISOLATED) Locate each error in isolation

30. 30 Benchmark Programs Program# 5-Error Faulty Versions Average Suite Size (# Failing Runs / # Passing Runs) tcas2011 (5 / 6) totinfo2022 (10 / 12) sched2029 (10 / 19) sched22030 (9 / 21) ptok232 (8 / 24) ptok21129 (5 / 24) replace2038 (9 / 29) Each faulty program contains 5 seeded errors, each in a different stmt Each faulty program is associated with a stmt-coverage adequate test suite such that at least one failing run exercises each error Experimental Benchmark Programs

31. 31 50 55 60 65 70 75 80 85 90 tcas totinfo sched sched2 ptok ptok2 replace Effectiveness Comparison of Value Replacement Techniques Isolated Full Partial Min Effectiveness Results Avg. Score per Ranked List (%)

32. 32 Efficiency of Value Replacement Searching for IVMPs is time-consuming Lossy techniques Reduce search space for finding IVMPs May result in some missed IVMPs Performed for single-error benchmarks Lossless techniques Only affect the efficiency of implementation Result in no missed IVMPs Performed for multi-error benchmarks 5 failing runs X 50,000 stmt instances per run 15 alt value sets per instance X = 3.75 million value replacement program executions  Over 10 days if each execution requires a quarter-second

33. 33 Lossy Techniques Limit considered statement instances Find IVMP  skip all subsequent instances of the same statement in the current run Don’t find IVMP  skip statement in subsequent runs Limit considered alternate value sets only use min, and max >, as compared to original value orig max <min >min <max > (skip)

34. 34 Lossless Techniques stmt instance 1 stmt instance 2 stmt instance 3 Original Execution (assume 2 alternate value sets at each stmt instance) Regular Value Replacement Executions (value replacements are independent of each other) (portions of original execution are duplicated multiple times) (x 6) (x 4) (x 2) Efficiency Improvements: (1)Fork child process to do each value replacement in original failing execution (2)Perform value replacements in parallel

35. 35 Lossless Techniques With Redundant Execution Removed (no duplication of any portion of original execution) With Parallelization (total time required to perform all value replacements is reduced)

36. 36 Search Reduction by Lossy Techniques Reduction in # of Executions by Lossy Techniques: Single-Error Benchmarks # val replacements needed FullLimited Mean2.0 M0.03 M Max21.5 M0.4 M

37. 37 Search Reduction by Lossy Techniques Reduction in # of Executions by Lossy Techniques: Single-Error Benchmarks On average, total number of executions reduced by a factor of 67

38. 38 Time Required for Reduced Search Time Required to Search using Lossy Techniques: Single-Error Benchmarks Mean55.6 min < 1 min39% of progs < 10 min60% of progs < 100 min87% of progs Max846.5 min

39. 39 Time Required for Reduced Search Time Required to Search using Lossy Techniques: Single-Error Benchmarks Only 13% of faulty programs required more than 100 minutes of IVMP search time

40. 40 0 100 200 300 400 500 600 tcas totinfo sched sched2 ptok ptok2 replace Time to Search in Each Faulty Program using Lossless Techniques: Multi-Error Benchmarks Full Partial Min Time Required with Lossless Techniques Avg. Time (seconds) With Lossless techniques, multiple errors in a program can be located in minutes. With Lossy techniques, some single errors require hours to locate.

41. 41 Execution Suppression Efficient location of memory errors through targeted state alteration [Jeffrey et. al., ICSM 2008] Alter state in a way that will definitely get closer to the goal each time Goal: identify first point of memory corruption in a failing execution

42. 42 Memory Errors and Corruption Memory errors Buffer overflow Uninitialized read Dangling pointer Double free Memory leak Memory corruption Incorrect memory location is accessed, or Incorrect value is assigned to a pointer variable

43. 43 Study of Memory Corruption Traversal of error  First point of memory corruption  Failure ProgramLOCMemory Error Type Analyzed Input Types gzip6.3 KGlobal overflowNo crashCrash 1 man10.8 KGlobal overflowCrash 1 bc10.7 KHeap overflowNo crashCrash 1 pine211.9 KHeap overflowNo crashCrash 1 mutt65.9 KHeap overflowNo crashCrash 1 ncompress1.4 KStack overflowNo crashCrash 1Crash 2 polymorph1.1 KStack overflowNo crashCrash 1Crash 2 xv69.2 KStack overflowNo crashCrash 1 tar28.4 KNULL dereferenceCrash 1 tidy35.9 KNULL dereferenceCrash 1 cvs104.1 KDouble freeCrash 1

44. 44 Observations from Study Total distance from point of error traversal until failure can be large Different inputs triggering memory corruption may result in different crashes or no crashes Distance from error traversal to first memory corruption, is considerably less than distance from first memory corruption to failure

45. 45 Execution Suppression: High-Level Program crash reveals memory corruption Key: assume memory corruption leads to crash Component 1: suppression Iteratively identify first point of memory corruption Omit the effect of certain statements during execution Component 2: variable re-ordering Expose crashes where they may not occur Helpful since key assumption does not always hold

46. 46 Suppression: How it Works While a crash occurs Identify accessed location L directly causing crash Identify last definition D of location L Re-execute program and omit execution of D and anything dependent on it Endwhile Report the statement associated with the most recent D First point of memory corruption

47. 47 Suppression: Example 1: int *p1 = &x[1]; 2: int *p2 = &x[0]; 3: int *q1 = &y[1]; 4: int *q2 = &x[0]; 5: *p1 = readInt(); 6: *p2 = readInt(); 7: *q1 = readInt(); 8: *q2 = readInt(); 9: int a = *p1 + *p2; 10: int b = *q1 + *q2; 11: int c = a + b + 1; 12: intArray[c] = 0; 13: structArray[*p2]->f = 0; 14: free(p2); 15: free(q2); Stmt 4: copy-paste error: “x” should be “y” Stmt 8: clobbers def @ stmt 6 Stmts 9 - 11: propagation Stmt 12: potential buffer overflow Stmt 13: potential overflow or NULL dereference Stmt 15: double free

48. 48 Suppression: Example Stmt 4: The error as well as the first point of memory corruption (Located in 4 executions) 1: int *p1 = &x[1]; 2: int *p2 = &x[0]; 3: int *q1 = &y[1]; 4: int *q2 = &x[0]; 5: *p1 = readInt(); 6: *p2 = readInt(); 7: *q1 = readInt(); 8: *q2 = readInt(); 9: int a = *p1 + *p2; 10: int b = *q1 + *q2; 11: int c = a + b + 1; 12: intArray[c] = 0; 13: structArray[*p2]->f = 0; 14: free(p2); 15: free(q2);

49. 49 Example: Execution 1 of 4 1: int *p1 = &x[1]; 2: int *p2 = &x[0]; 3: int *q1 = &y[1]; 4: int *q2 = &x[0]; 5: *p1 = readInt(); 6: *p2 = readInt(); 7: *q1 = readInt(); 8: *q2 = readInt(); 9: int a = *p1 + *p2; 10: int b = *q1 + *q2; 11: int c = a + b + 1; 12: intArray[c] = 0; 13: structArray[*p2]->f = 0; 14: free(p2); 15: free(q2); Stmt: Loc Defined: OK? 1 p1 2 p2 3 q1 5 *p1 6 *p2 7 *q1 4 q2 8 *p2/*q2 9 a 10 b 11 c 12 CRASH Action: Suppress definition of c at stmt 11 and all of its effects

50. 50 Example: Execution 2 of 4 1: int *p1 = &x[1]; 2: int *p2 = &x[0]; 3: int *q1 = &y[1]; 4: int *q2 = &x[0]; 5: *p1 = readInt(); 6: *p2 = readInt(); 7: *q1 = readInt(); 8: *q2 = readInt(); 9: int a = *p1 + *p2; 10: int b = *q1 + *q2; 11: int c = a + b + 1; 12: intArray[c] = 0; 13: structArray[*p2]->f = 0; 14: free(p2); 15: free(q2); Stmt: Loc Defined: OK? 1 p1 2 p2 3 q1 5 *p1 6 *p2 7 *q1 4 q2 8 *p2/*q2 9 a 10 b 11 12 13 CRASH Action: Suppress def of *p2/*q2 at stmt 8 and all of its effects

51. 51 Example: Execution 3 of 4 1: int *p1 = &x[1]; 2: int *p2 = &x[0]; 3: int *q1 = &y[1]; 4: int *q2 = &x[0]; 5: *p1 = readInt(); 6: *p2 = readInt(); 7: *q1 = readInt(); 8: *q2 = readInt(); 9: int a = *p1 + *p2; 10: int b = *q1 + *q2; 11: int c = a + b + 1; 12: intArray[c] = 0; 13: structArray[*p2]->f = 0; 14: free(p2); 15: free(q2); 1 p1 2 p2 3 q1 5 *p1 6 *p2 7 *q1 4 q2 Action: Suppress q2 at stmt 4 and effects 8 9 10 11 12 13 14 15 CRASH Stmt: Loc Defined: OK?

52. 52 Example: Execution 4 of 4 1: int *p1 = &x[1]; 2: int *p2 = &x[0]; 3: int *q1 = &y[1]; 4: int *q2 = &x[0]; 5: *p1 = readInt(); 6: *p2 = readInt(); 7: *q1 = readInt(); 8: *q2 = readInt(); 9: int a = *p1 + *p2; 10: int b = *q1 + *q2; 11: int c = a + b + 1; 12: intArray[c] = 0; 13: structArray[*p2]->f = 0; 14: free(p2); 15: free(q2); 1 p1 2 p2 3 q1 5 *p1 6 *p2 7 *q1 4 Result: stmt 4 identified 8 9 10 11 12 13 14 15 Stmt: Loc Defined: OK?

53. 53 Example: Summary 1 2 3 4 5 6 7 8 9 10 11 Execution 1 p1: p2: q1: q2: p1: p2: q1: p2, q2: * * * ** a: b: c: 12 CRASH! 1 2 3 4 5 6 7 8 9 10 11 12 p1: p2: q1: q2: p1: p2: q1: p2, q2: * * * ** a: b: 13 CRASH! Execution 2 suppress 1 2 3 4 5 6 7 8 9 10 11 12 13 14 p1: p2: q1: q2: p1: p2: q1: * * * Execution 3 suppress 15 CRASH! 1 2 3 4 5 6 7 8 9 10 11 12 13 14 p1: p2: q1: p1: p2: q1: * * * Execution 4 suppress 15 REPORT 4

54. 54 Variable Re-Ordering Re-order variables in memory prior to execution Try to cause a crash due to corruption, in cases where crash does not occur Can overcome limitations of suppression Do not terminate prematurely when corruption does not cause crash Applicable to executions that do not crash Position address variables after buffers

55. 55 Variable Re-Ordering: Example From program ncompress: void comprexx(char **fileptr) { int fdin; int fdout; char tempname[1024]; strcpy(tempname, *fileptr);... } tempname fdout fdin $ return addr On the call stack: Original Variable Ordering: Variable Re-Ordering: tempname fdout fdin $ return addr Original ordering  no stack smash Re-ordering  stack smash

56. 56 The Complete Algorithm Explanation Execution Suppression Algorithm exec := original failing execution; Do (A) identifiedStmt, exec := run suppression component using exec; (B) reordering, exec := run variable re-ordering component using exec; While (crashing reordering is found); Report identifiedStmt; (A) Runs suppression until no further crashes occur (B) Attempts to expose an additional crash Do/While loop iterates as long as variable re-ordering exposes a new crash

57. 57 Execution Suppression: Evaluation Suppression-only results (no variable re-ordering): ProgramInput Type # Exec. Required Maximum Static Dependence Distance From Located Stmt To… 1 st Memory CorruptionError gzipCrash 1200 manCrash 1212 bcCrash 1201 pineCrash 1205 muttCrash 1301 ncompressCrash 1 Crash 2 2424 0000 0000 polymorphCrash 1 Crash 2 2323 0000 1111 xvCrash 1402 tarCrash 1200 tidyCrash 1200 cvsCrash 1200

58. 58 Execution Suppression: Evaluation Suppression and variable re-ordering results: ProgramInput Type # Crashes Exposed # Var R-O Exec. Maximum Static Dependence Distance From Located Stmt To… 1 st Memory CorruptionError gzipNo Crash015--- manCrash 111801 bcNo Crash1---01 pineNo Crash1---05 muttNo Crash1---01 ncompressNo Crash1500 polymorphNo Crash1601 xvNo Crash113502

59. 59 Memory Errors in Multithreaded Programs Assume programs run on single processor Two main enhancements required for Execution Suppression Reproduce failure on multiple executions Record failure-inducing thread interleaving Replay same interleaving on subsequent executions In general, other factors should be recorded/replayed Identify data race errors Data race: concurrent, unsynchronized access of a shared memory location by multiple threads; at least one write Identified on-the-fly during suppression

60. 60 Identifying Data Races Data races involve WAR, WAW, or RAW dependencies Identified points of suppression are writes Can involve WAR or WAW dependence prior to that point Can involve RAW dependence after that point Monitor for an involved data race on-the-fly during a suppression execution

61. 61 On-the-fly Data Race Detection Identified suppression point (thread T 1 writes to location L) Last access to L by a thread other than T 1 Next read from L by a thread other than T 1 Monitor for synchronization on L Suppression Execution

62. 62 On-the-fly Data Race Detection Last access to L by a thread other than T 1 Monitor for synchronization on L Suppression Execution Next read from L by a thread other than T 1 WAR or WAW data race may be identified at this point

63. 63 On-the-fly Data Race Detection Last access to L by a thread other than T 1 Monitor for synchronization on L Suppression Execution RAW data race may be identified at this point WAR or WAW data race may be identified at this point

64. 64 Potentially-Harmful Data Races Given two memory accesses involved in a data race, force other thread interleavings to see if the state is altered Memory access point 1: access to L from thread T 1 Memory access point 2: access to L from thread T 2 For each ready thread besides T 1, re-execute from this point and schedule it in place of T 1 If value in L is changed at this point, data race is potentially-harmful Harmful Data Race Checking Executions

65. 65 Evaluation with Multithreaded Programs ProgramLOCError Type# Executions Required Precisely Identifies Error? apache191 KData race3yes mysql-1508 KData race3yes mysql-2508 KData race3yes mysql-3508 KUninitialized read2yes prozilla-116 KStack overflow2yes prozilla-216 KStack overflow4yes axel3 KStack overflow3yes Multithreaded Benchmark Programs and Results:

66. 66 Implementing Suppression: General Global variables count: Dynamic instruction count value suppress: Suppression mode flag (boolean) Variables associated with each register and memory word lastDef: count value associated with the instruction last defining it corrupt: whether associated effects need to be suppressed target.lastDef := ++count; At a program instruction (defines target, uses src 1 and src 2 ): Ensure instruction responsible for a crash can be identified: Carry out suppression as necessary: if (current instruction is a suppression point) suppress := true; target.corrupt := true; if (suppress) if (src 1.corrupt or src 2.corrupt) target.corrupt := true; else execute instruction; target.corrupt := false;

67. 67 Software/Hardware Support Software-only implementation can incur relatively high overhead (SW) Reduce overhead with hardware support Existing support in Itanium processors for deferred exception handling: extra bit for registers (HW1) Further memory augmentation: extra bit for memory words (HW1 + HW2) Overheads compared in a simulator

68. 68 Average Overhead: 7.2x (SW) 2.7x (HW1) 1.8x (HW1 + HW2) Performance Overhead Comparison

69. 69 Other Dynamic Error Location Techniques Other state-alteration techniques Delta Debugging [Zeller et al. FSE 2002, TSE 2002, ICSE 2005] Search in space for values relevant to a failure Search in time for failure cause transitions Predicate Switching [Zhang et. al. ICSE 2006] Alter predicate outcome to correct failing output Value Replacement is more aggressive; Execution Suppression is better targeted for memory errors

70. 70 Other Dynamic Error Location Techniques Program slicing-based techniques Pruning Dynamic Slices with Confidence [Zhang et. al. PLDI 2006] Failure-Inducing Chops [Gupta et. al. ASE 2005] Invariant-based techniques Daikon [Ernst et. al. IEEE TSE Feb. 2001] AccMon [Zhou et. al. HPCA 2007]

71. 71 Other Dynamic Error Location Techniques Statistical techniques Cooperative Bug Isolation [Ben Liblit doctoral dissertation, 2005] SOBER [Liu et. al. FSE 2005] Tarantula [Jones et. al. ICSE 2002] COMPUTERESULT Spectra-based techniques Nearest Neighbor [Renieris and Reiss, ASE 2003] FAILPASS SUSPICIOUS

72. 72 Future Directions Enhancements to Value Replacement Improve scalability Study when IVMPs cannot be found at faulty statement Enhancements to Execution Suppression Improve scalability of variable re-ordering Other techniques to expose crashes Handle memory errors that do not involve corruption Applications to Fixing Errors IVMPs can be used in BugFix [Jeffrey et. al., ICPC 2009] Comparatively little research in automated techniques for fixing errors Applications to Tolerating Errors Suppression can be used to recover from failures in server programs [Nagarajan et. al., ISMM 2009] Other applications?