1. Going from Concrete to Symbolic Model Checking via Predicate Abstraction
Willem Visser
Corina Pasareanu and Radek Pelanek
Automated Software Engineering Group NASA Ames Research Center

2. Overview Abstraction Lightweight framework for testing
Classic over-approximation based
Counter-example based refinement
Under-approximation based
Refinement based on abstraction’s exactness
Lightweight framework for testing
Test generation environment built around JPF with symbolic execution
Measure predicate coverage
Evaluate against other test-case generation methods
Java Container classes

3. Predicate Abstraction
1: x = 2;
2: while (x>0)
3: x = x - 1;
4: assert false;
Abstraction Mapping
p = (x>0)
1: p = T;
2: while (p)
3: p = !p ? F : T | F;

4. Abstraction Mapping
For a,a’ in 2{preds}: if wp(a’,T) /\ a add transition a → a’
may transition
must transition a a’ T
wp(a’,T)
a → a’ a’ T
wp(a’,T) a
a → a’

5. Example Abstraction 1: p = T; 1: x = 2; 2: while (p) 2: while (x>0)
3: x = x - 1;
4: assert false;
Abstraction Mapping
p = (x>0)
1: p = T;
2: while (p)
3: p = !p ? F : T | F;
{x – 1 > 0} x = x – 1 {p}
{x – 1 <= 0} x = x – 1 {!p}
wp(!p,x=x-1) /\ p
add p → !p
wp(p,x=x-1) /\ p
add p → p
wp(!p,x=x-1) /\ !p
add !p → !p
wp(p,x=x-1) /\ !p
!p → wp(!p,x=x-1)
!p→ !p is must trans

6. Infeasible Counter Example
Refinement
1: p = T;
2: while (p)
3: p = !p ? F : T | F;
4: assert false;
Infeasible Counter Example
1,2,3(F),2,4
1: x=2 {x>0};
2: x=2 {x>0};
3: x=1 {x<=0}
{x > 1} x = x -1 {x > 0}
may
must
X>0
X<=0
X>0
X>1
X<=1
X>0
X<=1
X<=0

7. Let’s Go Outside the Box
Rather than over-approximate and refine, we under-approximate and refine
Clearly complements existing techniques
If we restrict ourselves only to feasible behaviors when under-approximating then all safety property violations will be preserved
Build on top of classic explicit-state model checking infrastructure

8. Classic Explicit-State Search
PROCEDURE dfs() {
s = top(Stack);
FOR all transitions t enabled in s DO
s' = successor(s) after executing t;
IF s' NOT IN VisitedStates THEN
Enter s' into VisitedStates;
Push s' onto Stack;
dfs();
END
END;
Pop s from Stack; }
INIT {
Enter s0 into VisitedStates;
Push s0 onto Stack;

9. Explicit-State (1-step) αSearch
PROCEDURE dfs() {
s = top(Stack);
FOR all transitions t enabled in s DO
s' = successor(s) after executing t;
IF α(s‘) NOT IN VisitedStates THEN
Enter α(s‘) into VisitedStates;
Push s' onto Stack;
dfs();
END
END;
Pop s from Stack; }
INIT {
Enter α(s0) into VisitedStates;
Push s0 onto Stack;

10. αSearch Map concrete states to abstract states for state storing
1: x = 2;
2: while (x>0)
3: x = x - 1;
4: assert false;
1,p
2,p
3,p
Abstraction Mapping
p = (x>0)
Under-approximation of the behaviors
Always traverse only feasible paths

11. Concrete, May & Must May Transitions p = (x < 2)
D,1
D,0
E,1
E,2
B,1
C,0
A,p
D,p
E,p
E,!p
B,p
C,p
May Transitions p = (x < 2)
A,p
D,p
B,p
C,p
Must Transitions p = (x < 2)
Concrete

12. Concrete & αSearch Abstraction Search p = (x<2)
A,p
A,0
D,1
D,0
E,1
E,2
B,1
C,0
A,0
B,p
C,p
B,1
C,0
Abstraction Search
p = (x<2)
D,p
D,1
D,0
Transition not “exact”
E,!p
E,2

13. Refinement & αSearch A,0 D,1 D,0 E,1 E,2 B,1 C,0 A,p,q A,p,!q A,!p,!q
After Refinement Step
p = (x<2); q = x < 1
D,1
D,0
E,2
E,1

14. Example 1: x = 2; 2: while (x>0) 3: x = x - 1; 4: assert false;
Abstraction Mapping
p = (x>0)

15. Refinement
Check if the induced abstract transition is a must transition?
If not, add new predicates
Only 1 DP call
1,p
2,p
3,p
x = x – 1
{x – 1 > 0}
wp(p,x=x-1)
{x > 0}
Add x > 1 to abstraction predicates and repeat search
Globally for all transitions
Locally only for the transition (location) it refines

16. Predicate Abstraction αSearch… …
Showing property holds
Over-approximation based
Counter-example driven refinement
Expensive computation to calculate abstraction
Finding defects
Under-approximation based
Abstraction driven refinement
Trivial computation to calculate abstraction mapping

17. Issue αSearch tries to compute a
finite reachable bisimulation quotient
this is only possible if
a finite reachable bisimulation quotient exists
unreachable
reachable
wp(p,T) T p
if new predicates are infinitely required to refine the unreachable area the algorithm will not terminate

18. … Example x = 0; y = 0; while (y >= 0) y = x + y; y >= 0
p,q,r
y >= 0
x+y >= 0
2x+y >= 0 …
p,q
y >= 0
x+y >= 0
y >= 0 p
The refinement only refines the unreachable state space!

19. Modified Bakery Search Order Matters!! while true { x = y; x = x + 1;
wait (x<=y);
x = 0; }
while true {
y = x;
y = y + 1;
wait (y<x);
y = 0; }
1st iteration
18 concrete states and 12 abstract
x+1 <= y, x <= y+1 and y >= 0
2nd iteration
26 concrete state and 19 abstract
x+2<=y, y>=1 and x <= 1
3rd iteration
44 concrete states and 32 abstract
y <= 1, x <= 0 and y >= 2
4th iteration
48 concrete, 36 abstract
BFS
1st iteration
14 concrete states and 20 abstract
x+1 <= y, x <= y+1, y >= 0 and x <= 0
2nd iteration
29 concrete state and 21 abstract
x+2<=y, y<= 0, x <= -1 and x <= 1
3rd iteration
45 concrete states and 33 abstract
DFS
Search Order Matters!!

20. Symbolic Execution and αSearch
Current implementation is for a simple input language
oCaml using Simplify as a decision procedure
We would like to integrate the technique in Java Pathfinder (JPF) that supports symbolic execution (using the Omega Library)
To allow application to programs with complex data structures (objects)

21. From Concrete to Symbolic
X=1, Y = 0
X > Y
Concrete Behavior
Symbolic Behavior

22. Possible Approach Execute the concrete program on valid inputs
Collect all predicates in path condition
Solve constraints over all combinations of these predicates
Use results as inputs for step 1
When no new predicates are found, or, if an error is found, terminate

23. Example public static void method(int x, int y) {
method(1,1) + {true}
public static void method(int x, int y) {
if ((x > 0) && (y < 10)) {
if (y < 5) { … } else { … }
} else {
if (x > 0) { … } else { … }
} }
x > 0 && y < 10
y < 5
end
p1,!p2 → method(1,6)
!p1,p2 → method(-1,1)
!p1,!p2 → method(-1,6)
Solve Constraints
p1 = x > 0 && y < 10
p2 = y < 5
method(1,1) + {p1,p2}

24. Example (2) public static void method(int x, int y) {
method(1,6) + {p1,!p2}
public static void method(int x, int y) {
if ((x > 0) && (y < 10)) {
if (y < 5) { … } else { … }
} else {
if (x > 0) { … } else { … }
} }
x > 0 && y < 10
y < 5
end
p1 = x > 0 && y < 10
!p2 = y < 5
method(1,6) + {p1,!p2}

25. Example (4) public static void method(int x, int y) {
method(-1,1) + {!p1,p2}
public static void method(int x, int y) {
if ((x > 0) && (y < 10)) {
if (y < 5) { … } else { … }
} else {
if (x > 0) { … } else { … }
} }
x > 0 && y < 10
x > 0
end
!p1 = x > 0 && y < 10
!p3 = x > 0
Solve Constraints
!p1,p3 → method(1,11)
method(-1,1) + {!p1,p2,!p3}

26. Example (3) public static void method(int x, int y) {
method(1,11) + {!p1,p3}
public static void method(int x, int y) {
if ((x > 0) && (y < 10)) {
if (y < 5) { … } else { … }
} else {
if (x > 0) { … } else { … }
} }
x > 0 && y < 10
x > 0
end
!p1 = x > 0 && y < 10
p3 = y < 5
method(1,11) + {!p1,p3}

27. End of Part One
Showed under-approximation based search with refinement
Backward weakest precondition based
Forward symbolic execution based
Part Two
Rather than automated refinement we use user-provided abstractions
Motivation is to generate test-cases to achieve high behavioral coverage for Java container classes

28. Explicit-State (1-step) αSearch
PROCEDURE dfs() {
s = top(Stack);
FOR all transitions t enabled in s DO
s' = successor(s) after executing t;
IF α(s‘) NOT IN VisitedStates THEN
Enter α(s‘) into VisitedStates;
Push s' onto Stack;
dfs();
END
END;
Pop s from Stack; }
INIT {
Enter α(s0) into VisitedStates;
Push s0 onto Stack;

29. General Idea SUT ENV (m,n) API m is the seq. length of API calls … &
n is the number of values
used in the parameters of the calls
API …
put(v)
del(v)
Evaluate different techniques for selecting
test-cases from ENV(m,n)
to obtain maximum coverage

30. Predicate Coverage Cover all combinations of a given set of predicates
at each branch in the code
Red-Black Tree Predicates
root = null,
e.left = null,
e.right = null,
e.parent = null,
e.color = BLACK

31. Techniques Considered
Random selection
Classic model checking
State matching on complete state
Abstraction search
State matching on abstract (partial) state
Symbolic Execution
Complete matching using subsumption checks
Abstract matching

32. minor instrumentation
Framework
SUT
with
minor instrumentation
ENV
Coverage Manager
Abstraction Mapping +
State Storage
TestListener
JPF

33. Sample Output Test case number 77 for '15,L+R+P-REDroot':
Branch Number
Predicate Values
Unique ID for the test
Test case number 77 for '15,L+R+P-REDroot':
put(0);put(4);put(5);put(1);put(2);put(3);remove(4);
Test-case to achieve above coverage
Test case number 7 for '32,L-R-P+RED':
X2(0) == X1(0) &&
X2(0) < X0(1) &&
X1(0) < X0(1)
put(X0);put(X1);remove(X2);
put(1);put(0);remove(0);
Concrete
Symbolic
Path Condition
with solutions
Symbolic TC

34. Environment Skeleton M : sequence length N : parameter values
A : abstraction used
for (int i = 0; i < M; i++) {
int x = Verify.random(N - 1);
switch (Verify.random(1)) {
case 0: put(x); break;
case 1: remove(x); break;
} }
Verify.ignoreIf(checkAbstractState(A));

35. Symbolic Environment Skeleton
M : sequence length
A : abstraction used
for (int i = 0; i < M; i++) {
SymbolicInteger x = new SymbolicInteger(“X“+i);
switch (Verify.random(1)) {
case 0: put(x); break;
case 1: remove(x); break;
} }
Verify.ignoreIf(checkAbstractState(A));

36. Abstraction Search
Map state to an abstract version and backtrack if the abstract state was seen before, i.e. discard test-case
Mapping can be lossy or not
Abstraction mappings can be created by the user/tester
Default abstraction mappings are provided

37. Default Mappings Structure of the heap of the program
e.g. structure of the containers
Structure augmented with non-data fields
Structure augmented with symbolic constraints on the data in the structure
This requires checking constraint subsumption

38. Linearization Comparing Structures1 1 2 5 2 5 3 4 3 4 1 1 2 2 5 3 4 3 4 5

39. Linearization + Mapping
1b 2b 3r r b -1 -1
1b 2r 3r r r -1 -1 1 1 2 5 2 5 3 4 3 4
Linearization takes a mapping object as parameter to indicate how each node in the heap should be linearized.
In the example above each node gets, besides the unique identifier, a mapping of “r” if the original structure had a red node and “b” if the original structure had a black node in that position. If we also added the key values for each node the linearization might have looked something like:
1b6 2b4 3r r b

40. + Symbolic Execution Symbolic State x1 > x2 & x2 > x3 &
Symbolic Constraints
Shape

41. + + Subsumption Checking If only it was this simple! x1 > x2 &

42. Getting Ready for Checking Existential EliminationPC
s1 < s2 & s4 > s3 &
s4 < s1 & s4 < s5 &
s7 < s2 & s7 > s1 + s4 x2 x5 s2 s3 x3 x4 s5
 s1,s2,s3,s4,s5 such that
x1 = s1 & x2 = s4 & x3 = s3 & x4 = s5 & x5 = s2 & PC
x1 > x2 & x2 > x3 &
x2 < x4 & x5 > x1

43. Bidirectional Subsumption Checking
If new => old
backtrack
If old => new
new is more general than old
replace old with new
to increase chances of getting a match in the future
Continue on path from new, i.e. don’t backtrack
Ultimately for each shape we want to use disjunction of constraints
Small technicality prevents us – bug in omega lib

44. Evaluation Red-Black Trees Out of Memory runs are not reported
Breadth-first Search unless stated
Sequence Length = Values for the non-symbolic searches
First compare under Branch Coverage

45. Exhaustive Techniques Branch Coverage
Seq
Cov
Len
Time
Mem
Full MC 7 39
4.3
536
584
S+C+V
10.635
17.47
Sym – S+Sub
14.201
16.95
Optimal Branch Coverage is 39

46. Under-Approximation Techniques Branch Coverage
Seq
Cov
Len
Time
Mem S 21 39
6.1
57.353
72.07
S+C 18
5.8
32.577
21.16
Sym - S 7
4.3
10.054
15.43
Sym – S+C
11.998
10.76
Random 9
40.429
3.06
Optimal Branch Coverage is 39

47. Exhaustive Techniques Predicate Coverage
Seq
Cov
Len
Time
Mem
Full MC 7 79
5.2
543
309
S+C+V 10 95
5.7
350
228
Sym – S+Sub 11
102
6.1
222
117
Optimal Predicate Coverage is 106

48. Under-Approximation Techniques Predicate Coverage
Seq
Cov
Len
Time
Mem S
25d
106
21.7 90
13.31
S+C 30
8.3
354
100
Sym - S 12
6.1
230
123.27
Sym – S+C
104
6.2
356
138
Random 60
30.1
61.459
7.74
Optimal Predicate Coverage is 106

49. Observations
For a simple coverage such as branch coverage, all the techniques work well, including the exhaustive ones
But making the coverage more “behavioral”, even by a small increment, kills off the exhaustive techniques

50. Observations Full Blown Model Checking doesn’t work here
Its close cousin, that only looks at the relevant state at the relevant time, scales much better
Branch - full coverage after:
MC: 536s & 584Mb
Complete: 10s & 17Mb
Predicate – best coverage after:
MC: 79 covered with 543s & 309Mb
Complete: 95 covered with 350s & 228Mb

51. Observations
Symbolic techniques have a slight edge over concrete ones for exhaustive analysis
Comparing for Predicate Coverage (10)
Full Concrete(95): 350s & 228Mb
Full Symbolic(95): 123s & 62Mb
Current results indicate symbolic under-approximation based search is less efficient than concrete
Further experimentation required

52. Observations Random Search?
Seems to work rather well here
It will always have an edge on memory, since it uses almost none
It will most likely have an edge on speed, since it needs to do little additional work – it will however redo work often
It will in general do worse on test-case length, since it requires longer sequences to achieve more complex coverage

53. Observations Search Order Matters for the lossy techniques
BFS is inherently better than DFS
On occasion though it is the other way round

54. Conclusions & Future Work
Showed how predicate abstraction can be used for an under-approximation based search with refinement
Showed how a lightweight variant, where the abstraction mapping is given and no refinement is done, can be used for bug-finding and test-case generation
Goal: Derive predicates for analyzing containers automatically through the use of symbolic execution during refinement
Can we derive shape predicates automatically?