1. Model Checking Java Programs using Structural Heuristics
Alex Groce
Carnegie Mellon University
Willem Visser
NASA Ames Research Center

2. Model Checking Explores graph of reachable system states
Checking for local assertions, invariants and general temporal (logic) properties
Symbolic model checking
Explicit-state model checking

3. Java PathFinder Java Code Bytecode JAVAC JVM Model Checker Special JVM
void add(Object o) {
buffer[head] = o;
head = (head+1)%size; }
Object take() { …
tail=(tail+1)%size;
return buffer[tail];
Java Code
0: iconst_0
1: istore_2
2: goto #39
5: getstatic
8: aload_0
9: iload_2
10: aaload
Bytecode
JAVAC
JVM
Model
Checker
Special
JVM

4. Depth-first Search push initial state on Stack while (Stack not empty)
s = top(Stack)
if s has no more successors
pop the Stack
else
s’ = next successor of s
if s’ not already visited
mark s’ visited
if s’ is a goal state
then terminate
push s’ on Stack

5. Problems with DFS Produces lengthy counterexamples
If state-space is too large to fully explore
May expend all resources on a single path when shallow counterexamples exist
Failed runs give little information because states explored may be very “similar”

6. Directed Model Checking
Model checking as a search in a state space
Why not use heuristics to guide the search?
Need to know what we’re looking for
Can we find good heuristics for model checking?
Bug-finding rather than verification

7. Best-first Search priority queue Q = {initial state}
while (Q not empty)
s = state in Q with lowest f
remove s from Q
for each successor state s’ of s
if s’ not already visited
mark s’ visited
if s’ is a goal state
then terminate
f = h(s’)
store (s’, f) in Q

8. Two Kinds of Heuristics
Property-specific heuristics
Directed at a specific error
Number of unblocked threads as a measure of distance to deadlock
Static analysis for distance to an assertion check
Focus of most previous work in field

9. Two Kinds of Heuristics
Structural heuristics
Designed to explore the structure of a program in a systematic fashion
But what do we mean by structure?

10. Structural Heuristics
One obvious kind of structure in a program:
Control flow
Reachable control flow rather than just CFG
Motivation for branch coverage metrics used in software testing

11. Branch Coverage Instrument model checker to calculate branch coverage
Using a simple coverage measure as a heuristic doesn’t work well
Easily falls into local minima (once any branches are taken, every state on that path has “better” coverage)
Doesn’t distinguish between branches explored once and branches explored many times

12. The Branch Counting Heuristic
Count the number of times each branch has been taken
Heuristic value is then:
Branches never before taken get lowest value
Non-branching transitions are next lowest
Otherwise, score is equal to the count
(lower values are explored first)

13. Each CFG state is a basic block that increments
Three Searches
DFS
Branch Counting
CFG
Each CFG state is a basic block that increments
some variable x.
ERROR
BFS

14. Three Searches
DFS
Branch Counting
CFG
BFS

15. Three Searches
DFS
Branch Counting
CFG
BFS

16. Three Searches
DFS
Branch Counting
CFG
BFS

17. Three Searches
DFS
Branch Counting
CFG
Heuristic
avoids taking
BFS

18. Three Searches
DFS
Branch Counting
CFG
BFS

19. Three Searches DFS Branch Counting CFG Expands 15 states BFS
Terminates
only with
depth limit
Expands 25 states

20. Experimental Results DEOS real-time operating system example
This version uses an integer valued counter, without abstraction

21. Results for DEOS
All experiments performed on a 1.4GHz Athlon, limiting Java heap size to 512MB, all times are in seconds

22. The Interleaving Heuristic
An important (and very hard to find) class of errors in Java is concurrency errors
What kind of structure could we explore to catch these?
Thread-interdependency

23. The Interleaving Heuristic
Not clear how to heuristically define actual thread-interdependence
So we use an approximation:
Executions in which context is switched more often are given better heuristic values
Explores executions unlikely to appear in testing (JVM/JITs schedule quite differently)

24. The Interleaving Heuristic
Keep track on each path of which threads are executed at each transition
Give lower (better) heuristic score to paths in which the most recently executed thread has been run less frequently
Slightly more complicated in practice, counting live threads

25. Limiting the Queue
With heuristics we are more interested in finding bugs than in verification
So, we apply a technique from heuristic search literature:
Limit the size of the priority queue!
When queue has more than k states in it, remove all but k states with best heuristic values

26. Experimental Results Dining Philosophers Comparison to other results:
Godefroid and Khurshid in TACAS ’02 paper apply genetic algorithms to dining philosophers
Best result reported is 17 philosophers, 177 seconds, 50% success rate (on a slower machine)
HSF-SPIN
Not clear how to compare (times not given)
Best result they show is 16 philosophers, and SPIN (using partial order reduction) itself fails with 14 philosophers

27. Experimental Results

28. One Last Heuristic The choose-free heuristic:
Works only for abstracted Java programs
Rewards transitions that do not involve nondeterminism introduced by the abstraction
Prefers counterexamples that do not result from loss of precision introduced by the abstraction
Structure of abstraction, not program

29. Previous Work Edelkamp, Lafuente, and Leue Bloem, Ravi, and Somenzi
HSF-SPIN: SPIN + heuristic search framework
Bloem, Ravi, and Somenzi
Symbolic Guided Search: BDDs + heuristics
With BDDs heuristics can aid verification
Cobleigh, Clarke, and Osterweil
FLAVERS verification work

30. Conclusions Structural heuristics: a useful class of heuristics
When model checking is used for debugging, we may not know what kinds of bugs we are hunting
Property-specific heuristics are also useful; approach is complementary, not replacement
Most-blocked can perform as well or better than interleaving in the Remote Agent example, depending on the k limit and search method

31. Future Work Experiment with other, larger examples
Static analysis for property-specific heuristics
Language for properties/search/heuristics
Discover how heuristics work when symbolic execution is introduced into JPF
Counterexample analysis for “bug causality”
What other kinds of structure can be exploited with heuristics?
Counting occurrences of data values, perhaps