Localizing Errors in Counterexample Traces From Symptom to Cause: Localizing Errors in Counterexample Traces Mayur Naik, Purdue University Thomas Ball, Microsoft Research Sriram K. Rajamani, Microsoft Research

Model Checking + Fully automatic: Does not require the user to provide annotations. + Transparent: Produces a source-level error trace (counterexample). − An error trace represents a symptom of the error as opposed to its cause. − State-of-the-art model checkers report only one error trace.

The Problem How do we localize the cause of the error in an error trace? How do we produce multiple error traces having distinct causes? Note: Problem is relevant to other error-detection techniques as well.

What is a “Cause”? We define a “cause” to be those parts of an error trace not contained in any correct trace. The program fragments containing the cause are rendered unreachable. The model checker is invoked again to produce additional error traces.

Example main() { AcquireLock(); if (...) ReleaseLock(); else { ... } return;

Error #1: Lock acquired in succession main() { AcquireLock(); if (...) ReleaseLock(); else { ... } return;

Correct Trace Computation main() { AcquireLock(); if (...) ReleaseLock(); else { ... } return;

Error Cause Localization main() { AcquireLock(); if (...) ReleaseLock(); else { ... } return;

Error Recovery main() { AcquireLock(); if (...) ReleaseLock(); else { } return; Insert halt Unreachable from entry of main in future runs of the model checker

Error #2: Lock held on exit main() { AcquireLock(); if (...) ReleaseLock(); else { halt; ... } ... return;

Correct Trace Computation main() { AcquireLock(); if (...) ReleaseLock(); else { halt; ... } ... return;

Error Cause Localization main() { AcquireLock(); if (...) ReleaseLock(); else { halt; ... } ... return;

Error Recovery main() { AcquireLock(); if (...) ReleaseLock(); else { halt; ... } ... return; Insert halt Unreachable from entry of main in future runs of the model checker

Final (Error-Free) Program main() { AcquireLock(); if (...) ReleaseLock(); else { halt; ... } return;

Our Results A technique that exploits correct traces for error cause localization. Efficient algorithm for computing correct traces. Experimental results in the context of the SLAM toolkit. repair

Transitions and Edges 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 AcquireLock();

Transitions and Edges <(3, L), (5, U)> 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 AcquireLock(); <(3, L), (5, U)>

Transitions and Edges project(<(3, L), (5, U)>) = (3, 5) 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 AcquireLock(); project(<(3, L), (5, U)>) = (3, 5)

High-Level Algorithm while true do switch ModelCheck(G, ve) of // ve is of the form assert(e) case FAILURE(T): let C = GetCorrectTransitions(G, ve) and K = project(T) \ project(C) in if K = Ø then break for each (vi, vj) in K do insert a halt statement between vi and vj case SUCCESS:

Computing Correct Transitions 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 AcquireLock(); (1,U) (2,L) (3,L) (4,L) (5,U) ve ≡ assert(s==U) (5,L) <(1, U), (2, L)> <(2, L), (4, L)> <(4, L), (5, L)> Reachable state-space computed by model checker Transitions in Error Trace (T)

Computing Correct Transitions 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 AcquireLock(); (1,U) (2,L) (3,L) (4,L) (5,U) ve ≡ assert(s==U) (5,L) <(1, U), (2, L)> <(2, L), (4, L)> <(4, L), (5, L)> Transitions in Error Trace (T) Correct Transitions (C)

Computing Correct Transitions 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 AcquireLock(); (1,U) (2,L) (3,L) (4,L) (5,U) ve ≡ assert(s==U) (5,L) <(1, U), (2, L)> <(2, L), (4, L)> <(4, L), (5, L)> <(3, L), (5, U)> Transitions in Error Trace (T) Correct Transitions (C)

Computing Correct Transitions 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 AcquireLock(); (1,U) (2,L) (3,L) (4,L) (5,U) ve ≡ assert(s==U) (5,L) <(1, U), (2, L)> <(2, L), (4, L)> <(4, L), (5, L)> <(2, L), (3, L)> <(3, L), (5, U)> Transitions in Error Trace (T) Correct Transitions (C)

Computing Correct Transitions 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 AcquireLock(); (1,U) (2,L) (3,L) (4,L) (5,U) ve ≡ assert(s==U) (5,L) <(1, U), (2, L)> <(2, L), (4, L)> <(4, L), (5, L)> <(1, U), (2, L)> <(2, L), (3, L)> <(3, L), (5, U)> Transitions in Error Trace (T) Correct Transitions (C)

Example 1: An omission error (1,U) (2,L) (5,L) (4,L) (6,L) (1,U) (2,L) (5,U) (3,L) (6,U) 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 ... 6 AcquireLock(); ... <(1, U), (2, L)> <(2, L), (4, L)> <(4, L), (5, L)> <(5, L), (6, L)> <(1, U), (2, L)> <(2, L), (3, L)> <(3, L), (5, U)> <(5, U), (6, U)> Transitions in Error Trace (T) Correct Transitions (C)

Error Cause Localization 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 ... 6 AcquireLock(); ... <(1, U), (2, L)> <(2, L), (4, L)> <(4, L), (5, L)> <(5, L), (6, L)> <(1, U), (2, L)> <(2, L), (3, L)> <(3, L), (5, U)> <(5, U), (6, U)> K = project (T) \ project (C) Transitions in Error Trace (T) Correct Transitions (C)

Error Cause Localization 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 ... 6 AcquireLock(); ... <(1, U), (2, L)> <(2, L), (4, L)> <(4, L), (5, L)> <(5, L), (6, L)> <(1, U), (2, L)> <(2, L), (3, L)> <(3, L), (5, U)> <(5, U), (6, U)> K = project (T) \ project (C) Transitions in Error Trace (T) Correct Transitions (C)

Error Cause Localization 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 ... 6 AcquireLock(); ... <(1, U), (2, L)> <(2, L), (4, L)> <(4, L), (5, L)> <(5, L), (6, L)> <(1, U), (2, L)> <(2, L), (3, L)> <(3, L), (5, U)> <(5, U), (6, U)> K = project (T) \ project (C) Transitions in Error Trace (T) Correct Transitions (C)

Error Cause Localization 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 ... 6 AcquireLock(); ... (1,U) (2,L) (4,L) (5,L) (6,L) <(1, U), (2, L)> <(2, L), (4, L)> <(4, L), (5, L)> <(5, L), (6, L)> <(1, U), (2, L)> <(2, L), (3, L)> <(3, L), (5, U)> <(5, U), (6, U)> K = project (T) \ project (C) = { (2, 4), (4, 5) } Transitions in Error Trace (T) Correct Transitions (C)

Experimental Results Name of driver LOC mouse packet filter 984 serial mouse port 7441 keyboard packet filter 1067 IEEE 1394 bus driver 5818 keyboard class driver 13161 i8042 port 22168 packet-based DMA 24971 serial port 30905

Experimental Results Name of driver LOC Number of edges in error trace error cause mouse packet filter 984 73 110 4 serial mouse port 7441 56 1 keyboard packet filter 1067 107 IEEE 1394 bus driver 5818 45 7 44 60 81 3 85 keyboard class driver 13161 158 i8042 port 22168 127 124 5 packet-based DMA 24971 75 serial port 30905 248

Experimental Results Name of driver LOC Number of edges in Error cause localized? error trace error cause mouse packet filter 984 73 No 110 4 Yes serial mouse port 7441 56 1 keyboard packet filter 1067 107 IEEE 1394 bus driver 5818 45 7 44 60 81 3 85 keyboard class driver 13161 158 i8042 port 22168 127 124 5 packet-based DMA 24971 75 serial port 30905 248

Example 2: A variable-value error (2,(S,S)) (4,(S,S)) (5,(S,F)) (1,(S,S)) (6,(S,F)) Error Trace main() { 1 int status = S; 2 if (*) 3 status = foo(); else { 4 foo(); 5 status = S; } 6 assert(status==x); (1,(S,S)) (2,(S,S)) (3,(S,S)) (6,(S,S)) (6,(F,F)) Correct Traces enum { S, F } x = S; int foo() { if (*) x = S; else x = F; return x; } Program state is of the form (status, x)

Example 2: A variable-value error (2,(S,S)) (4,(S,S)) (5,(S,F)) (1,(S,S)) (6,(S,F)) Error Trace (6,(S,S)) (3,(S,S)) Correct Traces (2,(S,S)) (4,(S,S)) (5,(S,S)) (1,(S,S)) (6,(F,F)) main() { 1 int status = S; 2 if (*) 3 status = foo(); else { 4 foo(); 5 status = S; } 6 assert(status==x); enum { S, F } x = S; int foo() { if (*) x = S; else x = F; return x; } Program state is of the form (status, x)

Error Cause Localization (2,(S,S)) (4,(S,S)) (5,(S,F)) (1,(S,S)) (6,(S,F)) Error Trace (6,(S,S)) (3,(S,S)) Correct Traces (2,(S,S)) (4,(S,S)) (5,(S,S)) (1,(S,S)) (6,(F,F)) main() { 1 int status = S; 2 if (*) 3 status = foo(); else { 4 foo(); 5 status = S; } 6 assert(status==x); enum { S, F } x = S; int foo() { if (*) x = S; else x = F; return x; } K = project (T) \ project (C) = Ø

High-Level Algorithm while true do switch ModelCheck(G, ve) of // ve is of the form assert(e) case FAILURE(T): let C = GetCorrectTransitions(G, ve) and K = project(T) \ project(C) in if K = Ø then break for each (vi, vj) in K do insert a halt statement between vi and vj case SUCCESS:

High-Level Algorithm while true do switch ModelCheck(G, ve) of // ve is of the form assert(e) case FAILURE(T): let C = GetCorrectTransitions(G, ve) and K = project(T \ C) in if K = Ø then break for each (vi, vj) in K do insert a halt statement between vi and vj case SUCCESS:

Example 2: A variable-value error (2,(S,S)) (4,(S,S)) (5,(S,F)) (1,(S,S)) (6,(S,F)) Error Trace (6,(S,S)) (3,(S,S)) Correct Traces (2,(S,S)) (4,(S,S)) (5,(S,S)) (1,(S,S)) (6,(F,F)) main() { 1 int status = S; 2 if (*) 3 status = foo(); else { 4 foo(); 5 status = S; } 6 assert(status==x); enum { S, F } x = S; int foo() { if (*) x = S; else x = F; return x; } Program state is of the form (status, x)

Error Cause Localization (6,(S,S)) (3,(S,S)) Correct Traces (2,(S,S)) (4,(S,S)) (5,(S,S)) (1,(S,S)) (6,(F,F)) main() { 1 int status = S; 2 if (*) 3 status = foo(); else { 4 foo(); 5 status = S; } 6 assert(status==x); (1,(S,S)) (2,(S,S)) (4,(S,S)) (5,(S,F)) (6,(S,F)) Error Trace enum { S, F } x = S; int foo() { if (*) x = S; else x = F; return x; } K = project (T \ C) = { (4, 5), (5, 6) }

Example 1: An omission error (1,U) (2,L) (5,L) (4,L) (6,L) (1,U) (2,L) (5,U) (3,L) (6,U) 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 ... 6 AcquireLock(); ... <(1, U), (2, L)> <(2, L), (4, L)> <(4, L), (5, L)> <(5, L), (6, L)> <(1, U), (2, L)> <(2, L), (3, L)> <(3, L), (5, U)> <(5, U), (6, U)> Transitions in Error Trace (T) Correct Transitions (C)

Error Cause Localization 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 ... 6 AcquireLock(); ... <(1, U), (2, L)> <(2, L), (4, L)> <(4, L), (5, L)> <(5, L), (6, L)> <(1, U), (2, L)> <(2, L), (3, L)> <(3, L), (5, U)> <(5, U), (6, U)> K = project (T \ C) Transitions in Error Trace (T) Correct Transitions (C)

Error Cause Localization 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 ... 6 AcquireLock(); ... <(1, U), (2, L)> <(2, L), (4, L)> <(4, L), (5, L)> <(5, L), (6, L)> <(1, U), (2, L)> <(2, L), (3, L)> <(3, L), (5, U)> <(5, U), (6, U)> K = project (T \ C) Transitions in Error Trace (T) Correct Transitions (C)

Error Cause Localization 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 ... 6 AcquireLock(); ... <(1, U), (2, L)> <(2, L), (4, L)> <(4, L), (5, L)> <(5, L), (6, L)> <(1, U), (2, L)> <(2, L), (3, L)> <(3, L), (5, U)> <(5, U), (6, U)> K = project (T \ C) Transitions in Error Trace (T) Correct Transitions (C)

Error Cause Localization 1 AcquireLock(); 2 if (...) 3 ReleaseLock(); else { 4 ... } 5 ... 6 AcquireLock(); ... <(1, U), (2, L)> <(2, L), (4, L)> <(4, L), (5, L)> <(5, L), (6, L)> <(1, U), (2, L)> <(2, L), (3, L)> <(3, L), (5, U)> <(5, U), (6, U)> K = project (T \ C) = { (2, 4), (4, 5), (5, 6) } Transitions in Error Trace (T) Correct Transitions (C)

Limitations Control-based approach fails to localize the cause when every edge in the error trace is contained in some correct trace. Transition-based approach localizes the cause to a suffix of the error trace. Model Imprecision: Infeasible paths can misguide error cause localization using either approach.

Related Work Multiple Counterexamples Error Cause Localization Verisim [Bhargavan et al., TSE ’02] Error Cause Localization Explaining counterexamples [Jin et al., TACAS ’02] Explaining type errors [Wand, POPL ’86; Johnson & Walz, POPL ’86; Beaven & Stansifer, LOPLAS ’93; Duggan & Bent, SCP ’96; Chitil, ICFP ’01; Tip & Dinesh, TOSEM ’01] Program Slicing [Weiser, TSE ’84] Algorithmic Debugging [Shapiro, Ph.D. thesis ’82] Delta Debugging [Zeller, FSE ’99] Anomaly Detection Static: Meta-Level Compilation [Hallem et al., PLDI ’02] Dynamic: Daikon [Ernst et al., TSE ’01], DIDUCE [Hangal & Lam, ICSE ’02]

Conclusions We have presented a technique for localizing the causes of errors in counterexample traces. A combination of the control-based and transition-based approaches appears promising. Our technique is quite general and should be applicable to error detection tools based on data-flow analysis as well.