Lazy Annotation for Program Testing and Verification (Supplementary Materials) Speaker: Chen-Hsuan Adonis Lin Advisor: Jie-Hong Roland Jiang December 3,

Outline How to compute Interpolants of program sequence Concolic Approach (without learning) Dart: Directed Automated Random Testing December 3,

Strongest and Weakest Interpolants If I and I′ are both interpolants for (F,G), then so are I ∧ I′ and I ∨ I′ Let F ∧ G be unsatisfiable. The strongest interpolant for (F, G), denoted SI (F, G), is the unique interpolant for (F, G) that implies any other interpolant. The weakest interpolant for (F,G), denoted WI(F,G), is the unique interpolant that is implied by any other interpolant SI (F, G) implies WI (F, G) December 3,

Interpolants of Sequences We want to handle program paths, therefore a generalization of interpolant is needed. Given a sequence of formulas Γ = A 1,A 2,…,A n, we say that Ā 0, Ā 1,…, Ā n is in an interpolant for Γ when: Ā 0 = TRUE and Ā n = FALSE, For all 1≤i≤n, Ā i-1 ∧ A i implies Ā i, and For all 1≤i≤n, Ā i is in L(A 1,…,A i ) ∩ L(A i+1,…,A n ) If Γ is quantifier-free we can derive a quantifier- free interpolant for Γ (from the refutation of Γ ) December 3,

Interpolants for Sequences (con’t) An intuition: So this is a structured refutation of A 1, …, A k (Ā i ∧ A i+1 ) implies Ā i+1 December 3, A1A1 A2A2 A3A3 AkAk... Ā1Ā1 Ā2Ā2 Ā3Ā3 Ā k-1... TrueFalse 

Iterative Computation of Interpolants Given a formula F = F 1 ∧... ∧ F n, determine whether F is unsatisfiable, and if so, find interpolants for the pairs (F..i, F i+1.. ), i ∈ {1,...,n}, where F..i := F 1 ∧... ∧ F i and F i+1.. :=F i+1 ∧... ∧ F n Each formula F i models a program instruction A formula F = F 1 ∧... ∧ F n models a trace through a program In order to check if the trace is feasible or spurious, one can check if F is satisfiable or unsatisfiable December 3,

Iterative Computation of Interpolants (con’t) Definition (Tracking Property) Let F 1 ∧... ∧ F n be unsatisfiable, and let K i be interpolants for (F..i, F i+1.. ). We say that the family {K i } satisfies the tracking property if ( K i ∧ F i+1 |= K i+1 ) Proposition: Let F 1 ∧ F 2 ∧... ∧ F n be unsatisfiable. Let {I i } and {J i } be families of predicates defined according to the following procedures: I 0 := true, I i+1 := any interpolant for (I i ∧ F i+1, F i+2.. ), where i=0,..., n−1 J n := false, J i−1 := any interpolant for (F..i−1, ¬(F i → J i )), where i=n,..., 1 {I i } and {J i } are interpolants for (F..i,F i+1.. ) and satisfy the tracking property December 3,

Iterative Computation of Interpolants (con’t) Ｉ nterpolants satisfying the tracking property “explain” the infeasibility of a trace by providing Hoare annotations Evaluate strongest interpolants (I i ) {true} X := true {X} Y := X {X ∧ Y } assume(¬Y ∧ Z) {false}   Evaluate weakest interpolants (J i ) {true} X := true {X ∨ ¬Z} Y := X {Y ∨ ¬Z} assume(¬Y ∧ Z) {false}  By definition, I i |= J i ; Ex: (X ∧ Y) |= (Y ∨ ¬Z) December 3,

Iterative Computation of Interpolants (con’t) Evaluate strongest interpolants (I i ) {true} X := true {X} Y := X {X ∧ Y } assume(¬Y ∧ Z) {false Evaluate weakest interpolants (J i ) {true} X := true {X ∨ ¬Z} Y := X {Y ∨ ¬Z} assume(¬Y ∧ Z) {false} Intuitively, the strongest interpolants at node n records all facts that are established by the path leading up to n Ex: the strongest interpolant at node 2 is {X ∧ Y} Intuitively, the weakest interpolant at n represents the disjunction of all conditions that make the trace infeasible if they hold at n Ex: the weakest interpolant at node 2 is {Y ∨ ¬Z} December 3,

Outline How to compute Interpolants of program sequence Concolic Approach (without learning) Dart: Directed Automated Random Testing December 3,

Motivation of software testing Today, QA is mostly testing “50% of my company employees are testers, and the rest spends 50% of their time testing!” -- Bill Gates 1995 December 3,

Concolic Approach Combine concrete and symbolic execution for unit testing (Concrete + Symbolic = Concolic) DART: Directed Automated Random Testing Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation Authors Patrice Godefroid (Bell Labs) Nils Klarlund (Bell Labs) Koushik Sen (CS, UIUC) December 3,

Example (C code) int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if (z==y) { if (y == x+10) abort(); /* error */ } (1) Interface extraction: parameters of toplevel function external variables return values of external functions main(){ int tmp1 = randomInt(); int tmp2 = randomInt(); test_me(tmp1,tmp2); } (2) Generation of test driver for random testing: Problem: probability of reaching abort() is extremely low! December 3,

DART: Directed Search main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if (z==y) { if (y == x+10) abort(); /* error */ } Concrete Execution Symbolic Execution Path Constraint x = 36, y = 99 create symbolic variables x, y December 3,

DART: Directed Search main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if (z==y) { if (y == x+10) abort(); /* error */ } Concrete Execution Symbolic Execution Path Constraint create symbolic variables x, y x = 36, y = 99, z = 72 z = 2 * x December 3,

DART: Directed Search main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if (z==y) { if (y == x+10) abort(); /* error */ } Concrete Execution Symbolic Execution Path Constraint create symbolic variables x, y x = 36, y = 99, z = 72 z = 2 * x 2 * x != y Solve: 2 * x == y Solution: x = 1, y = 2 December 3,

DART: Directed Search main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if (z==y) { if (y == x+10) abort(); /* error */ } Concrete Execution Symbolic Execution Path Constraint x = 1, y = 2 create symbolic variables x, y December 3,

DART: Directed Search main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if (z==y) { if (y == x+10) abort(); /* error */ } Concrete Execution Symbolic Execution Path Constraint create symbolic variables x, y x = 1, y = 2, z = 2 z = 2 * x December 3,

DART: Directed Search main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if (z==y) { if (y == x+10) abort(); /* error */ } Concrete Execution Symbolic Execution Path Constraint create symbolic variables x, y x = 1, y = 2, z = 2 z = 2 * x 2 * x == y December 3,

DART: Directed Search main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if (z==y) { if (y == x+10) abort(); /* error */ } Concrete Execution Symbolic Execution Path Constraint create symbolic variables x, y 2 * x == y x = 1, y = 2, z = 2 z = 2 * x y != x + 10 Solve: (2 * x == y) Æ (y == x +10) Solution: x = 10, y = 20 December 3,

DART: Directed Search main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if (z==y) { if (y == x+10) abort(); /* error */ } Concrete Execution Symbolic Execution Path Constraint x = 10, y = 20 create symbolic variables x, y December 3,

DART: Directed Search main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if (z==y) { if (y == x+10) abort(); /* error */ } Concrete Execution Symbolic Execution Path Constraint create symbolic variables x, y x = 10, y = 20, z = 20 z = 2 * x December 3,

DART: Directed Search main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if (z==y) { if (y == x+10) abort(); /* error */ } Concrete Execution Symbolic Execution Path Constraint create symbolic variables x, y x = 10, y = 20, z = 20 z = 2 * x 2 * x == y December 3,

DART: Directed Search main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) {return 2 * x; } void test_me(int x, int y) { int z = double(x); if (z==y) { if (y == x+10) abort(); /* error */ } Concrete Execution Symbolic Execution Path Constraint create symbolic variables x, y 2 * x == y y == x +10 z = 2 * x x = 10, y = 20, z = 20 Program Error December 3,

Concolic Testing: A Middle Approach + Complex programs + Efficient - Less coverage + No false positive - Simple programs - Not efficient + High coverage - False positive Random Testing Symbolic Testing Concolic Testing + Complex programs +/- Somewhat efficient + High coverage + No false positive December 3,

Limitations: A Comparative View Concolic: Broad, shallowRandom: Narrow, deep December 3,

Hybrid Concolic Testing Interleave Random Testing and Concolic Testing to increase coverage Deep, broad, hybrid Search December 3,

Thanks for your attention December 3,