1. 50.530: Software Engineering
Sun Jun
SUTD

2. Week 12: Software Model Checking

3. Software Model Checking
Determining whether a program satisfies a property by the means of exhaustive searching.
Program
Model Checker
Counterexample!
Property
What is “property”?

4. How Model Checking Works?
Program
Property
System behaviors

5. Model Checking Works
Three researchers won Turing Award 2007 for their pioneer work on model checking!
(part of) Intel i7 processor is verified by symbolic model checking without a single test case!
8 cores, millions of registers; functional verification!
The Slam project from Microsoft successfully detected many bugs in many driver software!
Dozens of K lines of C codes; debugging.

6. Fundamentals of Model checking

7. Model: Kripke Structure
A Kripke structure is a tuple (S, R, L, I) where
S is a set of states;
R is a set of transitions;
I is the nonempty set of initial states;
L labels each state by a set of atomic propositions.

8. Model Example: Microwave Oven1
start oven
open door
open door
cook
{start,error}
close door 2
{close} 3 4
{close, heat}
done
open door
close door
start oven
start cooking
reset
warmup 5 6 7
{start,close,heat}
{start,close,error}
{start,close}
The transition labels are not part of the Kripke Structure.

9. A program can be transformed to a Kripke Structure.
Model Example
A program can be transformed to a Kripke Structure.
0,0,-1
1,0,-1
4,0,-1
5,0,-1
{error}
L0 x = 0;
L1 while (x < n) {
L x++;
L3 }
L4 if (x <= 0) {
L error();
L6 }
0,0,0
1,0,0
4,0,0
5,0,0
{error}
0,0,1
1,0,1
2,0,1
3,1,1 ……
0,0,2
1,0,2
2,0,2
3,1,2 ……
………………………………………………………….
Each state is represented by the (l,x,n) where l represent the line number; x is the value of variable x; and n is the value of n. The set of labels are: {error}.
Question: how many states are there?

10. Property: Temporal Logic
Temporal logic (CTL, LTL, CTL* among many others) extends propositional logic with temporal operators.
Proposed to specify properties about programs (in particular, program paths).
Turing award 1996 for his work on introducing temporal logic.

11. Linear Temporal Logic
LTL is built up from a finite set of propositions, the logical operators ¬ and ∨, and the temporal modal operators (X, G, and F).
p: p holds at the current state
X p: p holds at the state after one transition
G p: p holds on every state in the path
F p: p holds on some future state in the path p … p … p p p p … p … …

12. LTL Examples G !error G (!heat ∨ close) G (error => F heat)
an error should never occur.
G (!heat ∨ close)
it is never that case that the microwave oven is heating and not closed.
G (error => F heat)
from a state labelled with error, it will eventually reach a state labeled with heat.

13. LTL Verification
A trace of a Kripke Structure is a sequence of labels obtained by traversing through a path in the structure.
A Kripke Structure satisfies an LTL formula iff every path in the structure satisfies the formula.
G !error ?
G (!heat ∨ close) ?
G (error => F heat) ?

14. LTL Verification Algorithm
Example: G p
Model checking G p works by traversing through every state of the Kripke Structure (typically using BFS or DFS)
Example: GF p
Model checking GF p works by finding
a loop in the Kripke Structure such that no state in the loop is labelled with p
Standard loop finding algorithms are like Nested DFS, Tarjan’s Strongly Connected Component algorithm.
a deadlocking state not labeled with p

15. Counterexamples Example: G p
A counterexample is a finite path in the Kripke structure which ends with a state not satisfying p.
Example: GF p
A counterexample is a path which leads to a loop such that p is never satisfied during the loop.

16. Counterexample Examples
G !error
<3, 1, 2>
GF heat
<3,1>*

17. State Space Explosion 0,0,-1 1,0,-1 4,0,-1 5,0,-1 {error} L0 x = 0;
L1 while (x < n) {
L x++;
L3 }
L4 if (x <= 0) {
L error();
L6 }
0,0,0
1,0,0
4,0,0
5,0,0
{error}
0,0,1
1,0,1
2,0,1
3,1,1 ……
0,0,2
1,0,2
2,0,2
3,1,2 ……
………………………………………………………….
State Space Explosion is perhaps the most important problem of model checking.

18. Parallel Composition The following models a traffic light system.
The light model
The car model

19. Parallel composition often leads to state space explosion.
The overall model (where one transition of the light model and one of the car model always occurs synchronously)
Parallel composition often leads to state space explosion.

20. Abstraction: Example
Concrete Kripke Structure Abstract Kripke Structure
{p,q}
{p}
{q}
{p} 1 2 6
{p} 3 5 4 5 1 2 3 6 4
{p,q}
{p}
{p,q}

21. Abstraction: Example 1,0 4,0 L0 x = 0; L1 while (*) { L2 x++; L3 }
L4 if (x < 0) {
L error();
L6 }
0,0
2,0
3,1
1,1
4,1
2,1
1,2
4,2
3,2
2,2
1,3
4,3
3,3 ……
Abstraction: For each control location, let’s group the states into two groups.
One contains all states which satisfy x >=0.
One contains all states which satisfy x < 0.

22. Abstraction: Example 1,0 4,0 L0 x = 0; L1 while (*) { L2 x++; L3 }
L4 if (x < 0) {
L error();
L6 }
0,0
2,0
3,1
1,1
4,1
2,1
1,2
4,2
3,2
2,2
1,3
4,3
3,3
0, x>=0 ……
4, x>=0
1, x>=0
2, x>=0
3, x>=0

23. Abstraction: Definition
A Kripke Structure A = (Sa, Ra, La, Ia) is an abstraction of a Kripke Structure C = (S, R, L, I) if
Sa is a set of subset of S.
Ra contains a transition (s,s’) where s and s’ are in Sa if and only if there exists x in s and x’ in s’ such that (x,x’) is in R.
La(s) for any s in Sa is the union of L(x) for all x in s.
Ia is a subset of S containing I.

24. Exercise 1
Abstract the model by grouping state green and yellow into one.

25. Theorem
Theory: If A satisfies an LTL formula, then C satisfies the formula too.
Proof: Every trace of C is a trace of A. Ergo.

26. Exercise 2
Is G !(r && d) satisfied or not based on the following model? Is it satisfied based on the model you construct in Exercise 1?

27. Abstract Programs
It does not make sense to construct the concrete Kripke Structure first and then the abstraction.
Right, we need a systematic way of generating abstraction from the program syntax, and never construct the concrete Kripke Structure.

28. AUTOMATIC PREDICATE ABSTRACTION OF c PROGRAMS
Thomas Ball et al. PLDI 2001, most influential paper award
AUTOMATIC PREDICATE ABSTRACTION OF c PROGRAMS

29. Predicate Abstraction
Ordinary C programs
Boolean C programs
Given n predicates and a C program, C2BP automatically construct a C program which only contains n Boolean variables, each of which corresponds to a predicate. It is guaranteed that the Kripke Structure of the Boolean program is an abstraction of that of the original program.

30. Predicate Abstraction: Example
L0 x = 0;
L1 while (*) {
L x++;
L3 }
L4 if (x < 0) {
L error();
L6 }
L0 b=true;
L1 while (*) {
L if (b) {b=true} else {b=*};
L3 }
L4 if (*) {
assume(!b);
L error();
L6 }
where
the set of predicates is {x>=0};
assume(b) means that we assume that b is true there and we would ignore the cases where b is not true.

31. Predicate Abstraction: Example
0,0
1,0
4,0
L0 x = 0;
L1 while (*) {
L x++;
L3 }
L4 if (x < 0) {
L error();
L6 }
2,0
3,1
1,1
4,1
2,1
1,2
4,2
3,2
2,2
1,3
4,3
3,3
L0 b=true;
L1 while (*) {
L if (b) {b=true} else {b=*};
L3 }
L4 if (*) {
assume(!b);
L error();
L6 } ……
0,b
4,b
1,b
2,b
3,b

32. Predicate Abstraction: Assignment
Let b1,b2,…,bk be the Boolean variables corresponding to the predicates p1,p2,…,pk. A cube is a formula c1 ⋀ c2 ⋀ … ⋀ ck (where ci is either !bi or bi). Ideally, an assignment x := exp is translated into if (p) {b := true} if (n) {b := false} if (u) { b = *} for any b; for any cube p such that {p} x := exp {b} holds; for any cube n such that {n} x := exp {!b} holds; and for any cube u such that neither {u} x := exp {!b} nor {u} x := exp {b} holds.
In reality, we often abstract this so that we don’t have to check all cubes.

33. Predicate Abstraction: Conditional
A conditional if (cond) { … } else { … } is translated to if (*) { assume(c); //c is any bi or !bi such that cond => c … } else { assume(nc); //nc is any bi or !bi such that !cond => nc

34. Predicate Abstraction: While
A while loop is interpreted as a goto statement plus a conditional
The goto statement is simply copied
The conditional statement is translated as explained in the last slide.
while(cond) { … }
while(*) {
assume(cond);
do {
L1: …
} while(cond)
if(cond) goto L1

36. Exercise 3 Assume that we know Question: Is error reachable? lock()
1: if (*) {
2: do {
3: got_lock = 0;
4: if (*) {
5: lock();
6: got_lock ++;
7: }
8: if (got_lock) {
9: unlock();
10: }
11: } while (*) ;
12: }
13: do {
14: lock();
15: old = new;
16: if (*) {
17: unlock();
18: new ++;
19: }
20: } while ( new != old);
21: unlock ();
Assume that we know Question: Is error reachable?
lock()
unlock()
{error}

37. Exercise 3
1: if (*) {
3: got_lock = 0;
4: if (*) {
5: lock();
6: got_lock ++;
7: }
8: if (got_lock) {
9: unlock();
10: }
11: if (*) {goto 3;}
12: }
14: lock();
15: old = new;
16: if (*) {
17: unlock();
18: new ++;
19: }
20: if (new != old) {goto 14;}
21: unlock ();
Assume that we know Property: G !((pc=5⋁pc=14) ⋀ locked) && G !((pc=9⋁pc=17⋁pc=21) ⋀!locked)
lock()
unlock()
{error}

38. Exercise 3
1: if (*) {
3: got_lock = 0;
4: if (*) {
5: lock();
6: got_lock ++;
7: }
8: if (got_lock) {
9: unlock();
10: }
11: if (*) {goto 3;}
12: }
14: lock();
15: old = new;
16: if (*) {
17: unlock();
18: new ++;
19: }
20: if (new != old) {goto 14;}
21: unlock ();
Construct a Boolean program using predicates {locked} where locked is predicate denoting whether it is locked.
Assume lock() is implemented by simply assigning locked to true; unlock() is implemented by simply assigning locked to false.

39. Counter-example guided abstraction-refinement
Clarke et al. Journal of the ACM 2003
Counter-example guided abstraction-refinement

40. Theorem
Theory: If A satisfies an LTL formula, then C satisfies the formula too.
What if A does not?

41. Exercise 4
Abstract the light model by grouping state green and yellow into one state and construct the parallel composition with the car model.

42. Parallel Composition
Then construct the parallel composition and check if the composition satisfies the property.

43. Spurious Counterexamples
If a counterexample is found while model checking A, it doesn’t mean that C doesn’t satisfy the property, i.e., the counterexample could be spurious.
e.g., is G !(r && d) satisfied by this model?

44. Spurious Counterexamples
Is G !(r && d) satisfied by this model? A spurious example: <rs, ws, rd>

45. Analyzing Spurious Counterexample
Step 1: from rs to ws

46. Analyzing Spurious Counterexample
Step 2: from ws to rd
This step is broken in the concrete system!
An abstraction where green and yellow are separated will not have this spurious counterexample!

47. If a counterexample is spurious, the counterexample must be broken at some step!
We can always get rid of a spurious counterexample by refining the abstraction!

48. The Problem The most abstract The least abstract
Very small and easy to check; Lots of spurious counterexamples
Very big and hard to check; No spurious counterexamples

49. Can we find the right abstraction so that it is not very big and we can find a real counterexample or show there is none?

50. CEGAR Construct the initial abstraction Model check the abstraction
If a counterexample is found
If no counterexample is found
Check spuriousness
Report “system verified”
If it is not spurious
If it is spurious
Report counterexample
Refine the abstraction

51. Property: G !(pc=2⋀locked)
CEGAR: Example
1. do {
lock();
old=new;
if (*) {
unlock();
new++; }
8. } while (new != old)
Property: G !(pc=2⋀locked)

52. Initial Abstraction We should group two states only if
they have the same truth value for all atomic subformulae in the property, and
they are at the same control location.
In other words, we apply predicate abstraction with an initial set of predicates containing only the atomic subformulae of the property.
For instance, {x > 0, x+y=4} if the property is (G x > 0 => F x+y=4).

53. Property: G !(pc=2⋀locked)
CEGAR: Example
1. do {
lock();
old=new;
if (*) {
unlock();
new++; }
8. } while (new != old) 1.
locked=true;
skip;
if (*) {
locked=false;
skip; }
8. if (*) {goto 2;}
abstract with {locked}
Property: G !(pc=2⋀locked)

54. Is the property satisfied, with this abstraction?
CEGAR: Example 1.
locked=true;
skip;
if (*) {
locked=false;
skip; }
8. if (*) {goto 2;}
{!locked} 2
{locked} 3
{locked} 4
{locked} 5
{!locked}
Is the property satisfied, with this abstraction? 6
{!locked} 7
{!locked} 8 8 2
{locked}
{locked}

55. Check Spuriousness
Given a counterexample, i.e. a path of the Boolean program, we can check whether it is spurious using symbolic execution.

56. Spuriousness
1. do {
lock();
old=new;
if (*) {
unlock();
new++; }
8. } while (new != old)
Symbolic execution: locked=false ⋀ //initial condition locked1=true* ⋀ //line 2 old = new ⋀ //line 3 new!=old ⋀ //condition from line 8 *for simplicity, assume that lock() is locked = true and unlock() is locked = false. 2 3 4
{!locked}
{locked} 8
Unsat

57. Abstraction Refinementl1 l2 l3 l4
If the counterexample is spurious, it must be broken somewhere. Abstraction refinement is to find a new predicate such that the spurious counterexample is removed.

58. Refinement: Weakest Precondition
prog1
prog2
prog3 l1 l2 l3 l4
wp(prog3, l4) would be such a predicate.

59. Refinement: Weakest Precondition
Example:
What is the weakest precondition at line 8 for reaching line 2 (after line 8) with the following post-condition?
locked = true
Answer: locked=true ⋀ new!=old
Since locked=true is already used for abstraction, the new predicate is new!=old. 1.
lock();
old=new;
if (*) {
unlock();
new++; }
8. if (new != old) { }

60. Refinement: Interpolant
An interpolant based on the path condition of the spurious counterexample would be such a predicate. The interpolant at line 8 is old=new.
locked=false ⋀ //initial condition
locked1=true* ⋀ //line 2
old = new ⋀ //line 3
new!=old ⋀ //condition from line 8

61. Property: G !(pc=2⋀locked)
Refinement: Example
Abstract with {locked, new!=old}. Let ne be a Boolean which is true iff new!=old.
1. do {
lock();
old=new;
if (*) {
unlock();
new++; }
8. } while (new != old) 1.
locked=true;
ne=false;
if (*) {
locked=false;
if(!ne) {ne=true} else {ne=*;}; }
8. if (ne) {goto 2;}
Property: G !(pc=2⋀locked)

62. Property: G !(pc=2⋀locked)
Exercise 5
Draw the Kripke Structure of the following program and check whether the property is satisfied or not. 1.
locked=true;
ne=false;
if (*) {
locked=false;
if(!ne) {ne=true} else {ne=*;}; }
8. if (ne) {goto 2;}
Property: G !(pc=2⋀locked)

63. Recap

64. State-of-the-Art SLAM2
Part of Static Driver Verifier (SDV) 2, released with the Windows 7 WDK.
Is capable of verifying (falsifying) programs with dozens of thousands of lines of codes.
For SDV 2.0, the true bugs/total bugs ratio is 90-98% on Windows 7 Microsoft drivers, depending on the class of driver.
The number of non-useful results (timeouts, “don’t know” results) for drivers shipped as WDK samples, is 3.5% for WDM drivers and 0.02% for KMDF drivers.