50.530: Software Engineering Sun Jun SUTD

Week 12: Software Model Checking

Software Model Checking Determining whether a program satisfies a property by the means of exhaustive searching. Program Model Checker Counterexample! Property What is “property”?

How Model Checking Works? Program Property System behaviors

Model Checking Works Three researchers won Turing Award 2007 for their pioneer work on model checking! (part of) Intel i7 processor is verified by symbolic model checking without a single test case! 8 cores, millions of registers; functional verification! The Slam project from Microsoft successfully detected many bugs in many driver software! Dozens of K lines of C codes; debugging.

Fundamentals of Model checking

Model: Kripke Structure A Kripke structure is a tuple (S, R, L, I) where S is a set of states; R is a set of transitions; I is the nonempty set of initial states; L labels each state by a set of atomic propositions.

Model Example: Microwave Oven 1 start oven open door open door cook {start,error} close door 2 {close} 3 4 {close, heat} done open door close door start oven start cooking reset warmup 5 6 7 {start,close,heat} {start,close,error} {start,close} The transition labels are not part of the Kripke Structure.

A program can be transformed to a Kripke Structure. Model Example A program can be transformed to a Kripke Structure. 0,0,-1 1,0,-1 4,0,-1 5,0,-1 {error} L0 x = 0; L1 while (x < n) { L2 x++; L3 } L4 if (x <= 0) { L5 error(); L6 } 0,0,0 1,0,0 4,0,0 5,0,0 {error} 0,0,1 1,0,1 2,0,1 3,1,1 …… 0,0,2 1,0,2 2,0,2 3,1,2 …… …………………………………………………………. Each state is represented by the (l,x,n) where l represent the line number; x is the value of variable x; and n is the value of n. The set of labels are: {error}. Question: how many states are there?

Property: Temporal Logic Temporal logic (CTL, LTL, CTL* among many others) extends propositional logic with temporal operators. Proposed to specify properties about programs (in particular, program paths). Turing award 1996 for his work on introducing temporal logic.

Linear Temporal Logic LTL is built up from a finite set of propositions, the logical operators ¬ and ∨, and the temporal modal operators (X, G, and F). p: p holds at the current state X p: p holds at the state after one transition G p: p holds on every state in the path F p: p holds on some future state in the path p … p … p p p p … p … …

LTL Examples G !error G (!heat ∨ close) G (error => F heat) an error should never occur. G (!heat ∨ close) it is never that case that the microwave oven is heating and not closed. G (error => F heat) from a state labelled with error, it will eventually reach a state labeled with heat.

LTL Verification A trace of a Kripke Structure is a sequence of labels obtained by traversing through a path in the structure. A Kripke Structure satisfies an LTL formula iff every path in the structure satisfies the formula. G !error ? G (!heat ∨ close) ? G (error => F heat) ?

LTL Verification Algorithm Example: G p Model checking G p works by traversing through every state of the Kripke Structure (typically using BFS or DFS) Example: GF p Model checking GF p works by finding a loop in the Kripke Structure such that no state in the loop is labelled with p Standard loop finding algorithms are like Nested DFS, Tarjan’s Strongly Connected Component algorithm. a deadlocking state not labeled with p

Counterexamples Example: G p A counterexample is a finite path in the Kripke structure which ends with a state not satisfying p. Example: GF p A counterexample is a path which leads to a loop such that p is never satisfied during the loop.

Counterexample Examples G !error <3, 1, 2> GF heat <3,1>*

State Space Explosion 0,0,-1 1,0,-1 4,0,-1 5,0,-1 {error} L0 x = 0; L1 while (x < n) { L2 x++; L3 } L4 if (x <= 0) { L5 error(); L6 } 0,0,0 1,0,0 4,0,0 5,0,0 {error} 0,0,1 1,0,1 2,0,1 3,1,1 …… 0,0,2 1,0,2 2,0,2 3,1,2 …… …………………………………………………………. State Space Explosion is perhaps the most important problem of model checking.

Parallel Composition The following models a traffic light system. The light model The car model

Parallel composition often leads to state space explosion. The overall model (where one transition of the light model and one of the car model always occurs synchronously) Parallel composition often leads to state space explosion.

Abstraction: Example Concrete Kripke Structure Abstract Kripke Structure {p,q} {p} {q} {p} 1 2 6 {p} 3 5 4 5 1 2 3 6 4 {p,q} {p} {p,q}

Abstraction: Example 1,0 4,0 L0 x = 0; L1 while (*) { L2 x++; L3 } L4 if (x < 0) { L5 error(); L6 } 0,0 2,0 3,1 1,1 4,1 2,1 1,2 4,2 3,2 2,2 1,3 4,3 3,3 …… Abstraction: For each control location, let’s group the states into two groups. One contains all states which satisfy x >=0. One contains all states which satisfy x < 0.

Abstraction: Example 1,0 4,0 L0 x = 0; L1 while (*) { L2 x++; L3 } L4 if (x < 0) { L5 error(); L6 } 0,0 2,0 3,1 1,1 4,1 2,1 1,2 4,2 3,2 2,2 1,3 4,3 3,3 0, x>=0 …… 4, x>=0 1, x>=0 2, x>=0 3, x>=0

Abstraction: Definition A Kripke Structure A = (Sa, Ra, La, Ia) is an abstraction of a Kripke Structure C = (S, R, L, I) if Sa is a set of subset of S. Ra contains a transition (s,s’) where s and s’ are in Sa if and only if there exists x in s and x’ in s’ such that (x,x’) is in R. La(s) for any s in Sa is the union of L(x) for all x in s. Ia is a subset of S containing I.

Exercise 1 Abstract the model by grouping state green and yellow into one.

Theorem Theory: If A satisfies an LTL formula, then C satisfies the formula too. Proof: Every trace of C is a trace of A. Ergo.

Exercise 2 Is G !(r && d) satisfied or not based on the following model? Is it satisfied based on the model you construct in Exercise 1?

Abstract Programs It does not make sense to construct the concrete Kripke Structure first and then the abstraction. Right, we need a systematic way of generating abstraction from the program syntax, and never construct the concrete Kripke Structure.

AUTOMATIC PREDICATE ABSTRACTION OF c PROGRAMS Thomas Ball et al. PLDI 2001, most influential paper award AUTOMATIC PREDICATE ABSTRACTION OF c PROGRAMS

Predicate Abstraction Ordinary C programs Boolean C programs Given n predicates and a C program, C2BP automatically construct a C program which only contains n Boolean variables, each of which corresponds to a predicate. It is guaranteed that the Kripke Structure of the Boolean program is an abstraction of that of the original program.

Predicate Abstraction: Example L0 x = 0; L1 while (*) { L2 x++; L3 } L4 if (x < 0) { L5 error(); L6 } L0 b=true; L1 while (*) { L2 if (b) {b=true} else {b=*}; L3 } L4 if (*) { assume(!b); L5 error(); L6 } where the set of predicates is {x>=0}; assume(b) means that we assume that b is true there and we would ignore the cases where b is not true.

Predicate Abstraction: Example 0,0 1,0 4,0 L0 x = 0; L1 while (*) { L2 x++; L3 } L4 if (x < 0) { L5 error(); L6 } 2,0 3,1 1,1 4,1 2,1 1,2 4,2 3,2 2,2 1,3 4,3 3,3 L0 b=true; L1 while (*) { L2 if (b) {b=true} else {b=*}; L3 } L4 if (*) { assume(!b); L5 error(); L6 } …… 0,b 4,b 1,b 2,b 3,b

Predicate Abstraction: Assignment Let b1,b2,…,bk be the Boolean variables corresponding to the predicates p1,p2,…,pk. A cube is a formula c1 ⋀ c2 ⋀ … ⋀ ck (where ci is either !bi or bi). Ideally, an assignment x := exp is translated into if (p) {b := true} if (n) {b := false} if (u) { b = *} for any b; for any cube p such that {p} x := exp {b} holds; for any cube n such that {n} x := exp {!b} holds; and for any cube u such that neither {u} x := exp {!b} nor {u} x := exp {b} holds. In reality, we often abstract this so that we don’t have to check all cubes.

Predicate Abstraction: Conditional A conditional if (cond) { … } else { … } is translated to if (*) { assume(c); //c is any bi or !bi such that cond => c … } else { assume(nc); //nc is any bi or !bi such that !cond => nc

Predicate Abstraction: While A while loop is interpreted as a goto statement plus a conditional The goto statement is simply copied The conditional statement is translated as explained in the last slide. while(cond) { … } while(*) { assume(cond); do { L1: … } while(cond) if(cond) goto L1

Exercise 3 Assume that we know Question: Is error reachable? lock() 1: if (*) { 2: do { 3: got_lock = 0; 4: if (*) { 5: lock(); 6: got_lock ++; 7: } 8: if (got_lock) { 9: unlock(); 10: } 11: } while (*) ; 12: } 13: do { 14: lock(); 15: old = new; 16: if (*) { 17: unlock(); 18: new ++; 19: } 20: } while ( new != old); 21: unlock (); Assume that we know Question: Is error reachable? lock() unlock() {error}

Exercise 3 1: if (*) { 3: got_lock = 0; 4: if (*) { 5: lock(); 6: got_lock ++; 7: } 8: if (got_lock) { 9: unlock(); 10: } 11: if (*) {goto 3;} 12: } 14: lock(); 15: old = new; 16: if (*) { 17: unlock(); 18: new ++; 19: } 20: if (new != old) {goto 14;} 21: unlock (); Assume that we know Property: G !((pc=5⋁pc=14) ⋀ locked) && G !((pc=9⋁pc=17⋁pc=21) ⋀!locked) lock() unlock() {error}

Exercise 3 1: if (*) { 3: got_lock = 0; 4: if (*) { 5: lock(); 6: got_lock ++; 7: } 8: if (got_lock) { 9: unlock(); 10: } 11: if (*) {goto 3;} 12: } 14: lock(); 15: old = new; 16: if (*) { 17: unlock(); 18: new ++; 19: } 20: if (new != old) {goto 14;} 21: unlock (); Construct a Boolean program using predicates {locked} where locked is predicate denoting whether it is locked. Assume lock() is implemented by simply assigning locked to true; unlock() is implemented by simply assigning locked to false.

Counter-example guided abstraction-refinement Clarke et al. Journal of the ACM 2003 Counter-example guided abstraction-refinement

Theorem Theory: If A satisfies an LTL formula, then C satisfies the formula too. What if A does not?

Exercise 4 Abstract the light model by grouping state green and yellow into one state and construct the parallel composition with the car model.

Parallel Composition Then construct the parallel composition and check if the composition satisfies the property.

Spurious Counterexamples If a counterexample is found while model checking A, it doesn’t mean that C doesn’t satisfy the property, i.e., the counterexample could be spurious. e.g., is G !(r && d) satisfied by this model?

Spurious Counterexamples Is G !(r && d) satisfied by this model? A spurious example: <rs, ws, rd>

Analyzing Spurious Counterexample Step 1: from rs to ws

Analyzing Spurious Counterexample Step 2: from ws to rd This step is broken in the concrete system! An abstraction where green and yellow are separated will not have this spurious counterexample!

If a counterexample is spurious, the counterexample must be broken at some step! We can always get rid of a spurious counterexample by refining the abstraction!

The Problem The most abstract The least abstract Very small and easy to check; Lots of spurious counterexamples Very big and hard to check; No spurious counterexamples

Can we find the right abstraction so that it is not very big and we can find a real counterexample or show there is none?

CEGAR Construct the initial abstraction Model check the abstraction If a counterexample is found If no counterexample is found Check spuriousness Report “system verified” If it is not spurious If it is spurious Report counterexample Refine the abstraction

Property: G !(pc=2⋀locked) CEGAR: Example 1. do { 2. lock(); 3. old=new; 4. if (*) { 5. unlock(); 6. new++; 7. } 8. } while (new != old) Property: G !(pc=2⋀locked)

Initial Abstraction We should group two states only if they have the same truth value for all atomic subformulae in the property, and they are at the same control location. In other words, we apply predicate abstraction with an initial set of predicates containing only the atomic subformulae of the property. For instance, {x > 0, x+y=4} if the property is (G x > 0 => F x+y=4).

Property: G !(pc=2⋀locked) CEGAR: Example 1. do { 2. lock(); 3. old=new; 4. if (*) { 5. unlock(); 6. new++; 7. } 8. } while (new != old) 1. 2. locked=true; 3. skip; 4. if (*) { 5. locked=false; 6. skip; 7. } 8. if (*) {goto 2;} abstract with {locked} Property: G !(pc=2⋀locked)

Is the property satisfied, with this abstraction? CEGAR: Example 1. 2. locked=true; 3. skip; 4. if (*) { 5. locked=false; 6. skip; 7. } 8. if (*) {goto 2;} {!locked} 2 {locked} 3 {locked} 4 {locked} 5 {!locked} Is the property satisfied, with this abstraction? 6 {!locked} 7 {!locked} 8 8 2 {locked} {locked}

Check Spuriousness Given a counterexample, i.e. a path of the Boolean program, we can check whether it is spurious using symbolic execution.

Spuriousness 1. do { 2. lock(); 3. old=new; 4. if (*) { 5. unlock(); 6. new++; 7. } 8. } while (new != old) Symbolic execution: locked=false ⋀ //initial condition locked1=true* ⋀ //line 2 old = new ⋀ //line 3 new!=old ⋀ //condition from line 8 *for simplicity, assume that lock() is locked = true and unlock() is locked = false. 2 3 4 {!locked} {locked} 8 Unsat

Abstraction Refinement l1 l2 l3 l4 If the counterexample is spurious, it must be broken somewhere. Abstraction refinement is to find a new predicate such that the spurious counterexample is removed.

Refinement: Weakest Precondition prog1 prog2 prog3 l1 l2 l3 l4 wp(prog3, l4) would be such a predicate.

Refinement: Weakest Precondition Example: What is the weakest precondition at line 8 for reaching line 2 (after line 8) with the following post-condition? locked = true Answer: locked=true ⋀ new!=old Since locked=true is already used for abstraction, the new predicate is new!=old. 1. 2. lock(); 3. old=new; 4. if (*) { 5. unlock(); 6. new++; 7. } 8. if (new != old) { }

Refinement: Interpolant An interpolant based on the path condition of the spurious counterexample would be such a predicate. The interpolant at line 8 is old=new. locked=false ⋀ //initial condition locked1=true* ⋀ //line 2 old = new ⋀ //line 3 new!=old ⋀ //condition from line 8

Property: G !(pc=2⋀locked) Refinement: Example Abstract with {locked, new!=old}. Let ne be a Boolean which is true iff new!=old. 1. do { 2. lock(); 3. old=new; 4. if (*) { 5. unlock(); 6. new++; 7. } 8. } while (new != old) 1. 2. locked=true; 3. ne=false; 4. if (*) { 5. locked=false; 6. if(!ne) {ne=true} else {ne=*;}; 7. } 8. if (ne) {goto 2;} Property: G !(pc=2⋀locked)

Property: G !(pc=2⋀locked) Exercise 5 Draw the Kripke Structure of the following program and check whether the property is satisfied or not. 1. 2. locked=true; 3. ne=false; 4. if (*) { 5. locked=false; 6. if(!ne) {ne=true} else {ne=*;}; 7. } 8. if (ne) {goto 2;} Property: G !(pc=2⋀locked)

Recap

State-of-the-Art SLAM2 Part of Static Driver Verifier (SDV) 2, released with the Windows 7 WDK. Is capable of verifying (falsifying) programs with dozens of thousands of lines of codes. For SDV 2.0, the true bugs/total bugs ratio is 90-98% on Windows 7 Microsoft drivers, depending on the class of driver. The number of non-useful results (timeouts, “don’t know” results) for drivers shipped as WDK samples, is 3.5% for WDM drivers and 0.02% for KMDF drivers.