1 Testing, Abstraction, Theorem Proving: Better Together! Greta Yorsh joint work with Thomas Ball and Mooly Sagiv

2Motivation Traditionally, Find errors using testing Prove absence of errors using static analysis –abstract interpretation –theorem proving Recently, finding errors by a combination of static and dynamic analyses Our method: leverage on existing tests to prove absence of errors Static Analysis Testing 1. 2.

3 Main Result A new method for static analysis –leverages on concrete executions –computes program invariants –sound –complete with respect to abstract interpreter –terminates

4 The idea test set executionMonitor existing test set execution AbstractionAbstraction to generalize from a test set Theorem proverTheorem prover to check soundness

5 T’ abstraction  program P Execute Abstract CTCT Check invariant(A T,P) ATAT Check safety properties yes potential error Fabricate tests test set T no verified The Core Algorithm

6 void foo(int x, int y){ int *px = NULL; A: x = x + 1; B: if (x < 4) C: px = &x; D: if (px == &y) E: x = x + 1; F: if (x < 5) G: *px = *px + 1; H: return; } [pc, x, y, px ] Concrete State pc, x<5, px== NULL Abstraction predicates: foo(2,0)foo(6,0)foo(11,0)Test set T :

7 void foo(int x, int y){ int *px = NULL; A: x = x + 1; B: if (x < 4) C: px = &x; D: if (px == &y) E: x = x + 1; F: if (x < 5) G: *px = *px + 1; H: return; } foo(3,0)foo(2,0)foo(6,0)foo(11,0) pc, x<5, px== NULL Abstraction predicates: (A,t,t) (B,t,t) (C,t,t) (D,t,f) (F,t,f) (G,t,f) (H,t,f) (A,f,t) (B,f,t) (D,f,t) (F,f,t) (H,f,t) Check invariant(A T,foo) Test set T : [B,4,0, NULL ] [D,4,0, NULL ] Fabricated States

8 b d A,t,t C,t,t D,t,f F,t,f G,t,f H,t,f F,t,t E,t,f G,t,t F,f,f H,f,f A,f,t B,f,t D,f,t F,f,t H,f,t foo(2,0)foo(11,0) foo(3,0) B,t,t D,t,t foo(6,0) void foo(int x, int y){ int *px = NULL; A: x = x + 1; B: if (x < 4) C: px = &x; D: if (px == &y) E: x = x + 1; F: if (x < 5) G: *px = *px + 1; H: return; } Finding a bug [pc, x, y, px ] (pc, x<5, px== NULL ) Concrete State Abstract State

9 T’ abstraction  program P Execute Abstract CTCT Check invariant(A T,P) ATAT Check safety properties yes potential error Fabricate tests test set T no verified The Core Algorithm

10 What does it require? Execution from fabricate states –fault injection or debugger Check invariants using a theorem prover Fabricate states using counter-examples from the theorem prover

11 Using Theorem Provers Calls to a theorem prover –can be expensive –timeout potentially causes loss of precision –not always generate concrete counter-example Our method is oriented towards finding a proof rather than detecting errors Checking invariants requires less theorem prover calls than computing invariants

12 void foo(int x, int y){ int *px = NULL; A: x = x + 1; B: if (x < 5) C: px = &x; D: if (px == &y) E: x = x + 1; F: if (x < 5) G: *px = *px + 1; H: return; } foo(3,0)foo(2,0)foo(6,0)foo(11,0) pc, x<5, px== NULL Abstraction predicates: Test set T : Check invariant(A T,foo) [D,4,0,&y ] [E,4,0,&y ]

13 e’ A,t,t B,t,t C,t,t F,t,f G,t,f H,t,f F,f,f H,f,f A,f,t B,f,t D,f,t F,f,t H,f,t fixed_foo(2,0)fixed_foo(11,0) d’ D,t,f E,t,f fixed_foo(6,0) pc, x<5, px==NULL void foo(int x, int y){ int *px = NULL; A: x = x + 1; B: if (x < 5) C: px = &x; D: if (px == &y) E: x = x + 1; F: if (x < 5) G: *px = *px + 1; H: return; } Finding a proof

14 Bisimulation – every error is real Weak reachability – every abstract state represents a feasible concrete state Simulation –error in concrete system is error in abstract system –avoids refinements unnecessary to prove the property (our method) Checking Invariants BisimulationWeak reachabilitySimulation less precise more precise (our method)

15 2-Process Bakery Proof by Bisimulation –results from [C. Pasareanu et. al. - CAV’05] –5 refinement steps Our method –invariant with 17 abstract states –no refinement

16 A,A,t B,A,t A,A,f C,A,t A,B,t B,B,t D,A,t C,A,f C,B,t D,B,t C,C,t D,C,t A,C,t B,C,t C,C,f C,D,f C,B,f 2-Process Bakery An invariant with 17 abstract states

17 void foo(int x, int y){ int *px = NULL; A: x = x + 1; B: if (x < 5) C: px = &x; D: if (px == &y) E: x = x + 1; F: if (x < 5) G: *px = *px + 1; H: return; } foo(3,0)foo(2,0)foo(6,0)foo(11,0) pc, x<10, px== NULL Abstraction predicates: Test set T :

18 A,t,t B,t,t C,t,t D,t,f F,t,f G,t,f H,t,f E,t,f F,f,f H,f,f H,t,t fixed_foo(2,0) fixed_foo(6,0) F,t,t G,t,t f’ D,t,t g’ A,f,t B,f,t D,f,t F,f,t H,f,t fixed_foo(11,0) pc, x<10, px==NULL void foo(int x, int y){ int *px = NULL; A: x = x + 1; B: if (x < 5) C: px = &x; D: if (px == &y) E: x = x + 1; F: if (x < 5) G: *px = *px + 1; H: return; } Finding a false error

19 abstraction  T’ program P Execute Abstract CTCT Check invariant(A T,P) ATAT Check safety properties yes potential error Fabricate tests test set T no verified The Algorithm classify errors false alarm refine abstraction real error abstraction  ’

20 Abstract Interpretation Potentially more efficient –execution and abstraction of concrete states is fast –avoid abstract transformers for parametric domains –no static abstract transition system –one concrete state can “discover” many abstract states Potentially more precise –the result is the least fixpoint w.r.t.  (if all theorem prover calls were conclusive)

21 Hybrid Approach It is sound to stop the concrete execution at any moment and check invariant(A,P) Alternate between concrete execution and abstract interpretation Tune the performance of the analysis execution time vs. theorem proving time –timeout new abstract state not covered – while (x < 10 6 ) x++;

22 a0a0...

23 Further Research Directions Robust implementation and evaluation –based on a SAT solver Fabricated states –cover more abstract states –turn fabricated traces into concrete traces –guide refinement Benefits for testing –new notion of coverage related to errors –eliminate redundant tests –infer procedures pre- and post-conditions –unit tests generation

24 The Most Related Work [T.Reps et al. – VMCAI’04 ] Best abstract transformers for statements [T.Ball – FMCO’04] Weak reachability [D.Lee, M.Yannakakis – STOC’92] [C. Pasareanu et. al. - CAV’05] Bisimulation

25Summary A new method for static analysis –computing invariants –leverage existing test suite –Abstraction to generalize from a test –Theorem prover to check soundness –Model generator to create new tests Potentially faster and more precise than abstract interpretation Explain abstract interpretation in terms of concrete execution and abstraction