Model Checking Java Programs using Structural Heuristics

1
Model Checking Java Programs using Structural
Heuristics

• Alex Groce
• Carnegie Mellon University
• Willem Visser
• NASA Ames Research Center

2
Model Checking

• Explores graph of reachable system states
• Checking for local assertions, invariants and
  general temporal (logic) properties
• Symbolic model checking
• Explicit-state model checking

3
Java PathFinder
Special JVM
4
Depth-first Search

• push initial state on Stack
• while (Stack not empty)
• s top(Stack)
• if s has no more successors
• pop the Stack
• else
• s next successor of s
• if s not already visited
• mark s visited
• if s is a goal state
• then terminate
• push s on Stack

5
Problems with DFS

• Produces lengthy counterexamples
• If state-space is too large to fully explore
• May expend all resources on a single path when
  shallow counterexamples exist
• Failed runs give little information because
  states explored may be very similar

6
Directed Model Checking

• Model checking as a search in a state space
• Why not use heuristics to guide the search?
• Need to know what were looking for
• Can we find good heuristics for model checking?
• Bug-finding rather than verification

7
Best-first Search

• priority queue Q initial state
• while (Q not empty)
• s state in Q with lowest f
• remove s from Q
• for each successor state s of s
• if s not already visited
• mark s visited
• if s is a goal state
• then terminate
• f h(s)
• store (s, f) in Q

8
Two Kinds of Heuristics

• Property-specific heuristics
• Directed at a specific error
• Number of unblocked threads as a measure of
  distance to deadlock
• Static analysis for distance to an assertion
  check
• Focus of most previous work in field

9
Two Kinds of Heuristics

• Structural heuristics
• Designed to explore the structure of a program in
  a systematic fashion
• But what do we mean by structure?

10
Structural Heuristics

• One obvious kind of structure in a program
• Control flow
• Reachable control flow rather than just CFG
• Motivation for branch coverage metrics used in
  software testing

11
Branch Coverage

• Instrument model checker to calculate branch
  coverage
• Using a simple coverage measure as a heuristic
  doesnt work well
• Easily falls into local minima (once any branches
  are taken, every state on that path has better
  coverage)
• Doesnt distinguish between branches explored
  once and branches explored many times

12
The Branch Counting Heuristic

• Count the number of times each branch has been
  taken
• Heuristic value is then
• Branches never before taken get lowest value
• Non-branching transitions are next lowest
• Otherwise, score is equal to the count
• (lower values are explored first)

13
Three Searches
DFS
Branch Counting
CFG
Each CFG state is a basic block that
increments some variable x.
ERROR
BFS
14
Three Searches
DFS
Branch Counting
CFG
BFS
15
Three Searches
DFS
Branch Counting
CFG
BFS
16
Three Searches
DFS
Branch Counting
CFG
BFS
17
Three Searches
DFS
Branch Counting
CFG
Heuristic avoids taking
BFS
18
Three Searches
DFS
Branch Counting
CFG
BFS
19
Three Searches
DFS
Branch Counting
CFG
Expands 15 states
BFS
Terminates only with depth limit
Expands 25 states
20
Experimental Results

• DEOS real-time operating system example
• This version uses an integer valued counter,
  without abstraction

21
Results for DEOS

• All experiments performed on a 1.4GHz Athlon,
  limiting Java heap size to 512MB, all times are
  in seconds

22
The Interleaving Heuristic

• An important (and very hard to find) class of
  errors in Java is concurrency errors
• What kind of structure could we explore to catch
  these?
• Thread-interdependency

23
The Interleaving Heuristic

• Not clear how to heuristically define actual
  thread-interdependence
• So we use an approximation
• Executions in which context is switched more
  often are given better heuristic values
• Explores executions unlikely to appear in testing
  (JVM/JITs schedule quite differently)

24
The Interleaving Heuristic

• Keep track on each path of which threads are
  executed at each transition
• Give lower (better) heuristic score to paths in
  which the most recently executed thread has been
  run less frequently
• Slightly more complicated in practice, counting
  live threads

25
Limiting the Queue

• With heuristics we are more interested in finding
  bugs than in verification
• So, we apply a technique from heuristic search
  literature
• Limit the size of the priority queue!
• When queue has more than k states in it, remove
  all but k states with best heuristic values

26
Experimental Results

• Dining Philosophers
• Comparison to other results
• Godefroid and Khurshid in TACAS 02 paper apply
  genetic algorithms to dining philosophers
• Best result reported is 17 philosophers, 177
  seconds, 50 success rate (on a slower machine)
• HSF-SPIN
• Not clear how to compare (times not given)
• Best result they show is 16 philosophers, and
  SPIN (using partial order reduction) itself fails
  with 14 philosophers

27
Experimental Results
28
One Last Heuristic

• The choose-free heuristic
• Works only for abstracted Java programs
• Rewards transitions that do not involve
  nondeterminism introduced by the abstraction
• Prefers counterexamples that do not result from
  loss of precision introduced by the abstraction
• Structure of abstraction, not program

29
Previous Work

• Edelkamp, Lafuente, and Leue
• HSF-SPIN SPIN heuristic search framework
• Bloem, Ravi, and Somenzi
• Symbolic Guided Search BDDs heuristics
• With BDDs heuristics can aid verification
• Cobleigh, Clarke, and Osterweil
• FLAVERS verification work

30
Conclusions

• Structural heuristics a useful class of
  heuristics
• When model checking is used for debugging, we may
  not know what kinds of bugs we are hunting
• Property-specific heuristics are also useful
  approach is complementary, not replacement
• Most-blocked can perform as well or better than
  interleaving in the Remote Agent example,
  depending on the k limit and search method

31
Future Work

• Experiment with other, larger examples
• Static analysis for property-specific heuristics
• Language for properties/search/heuristics
• Discover how heuristics work when symbolic
  execution is introduced into JPF
• Counterexample analysis for bug causality
• What other kinds of structure can be exploited
  with heuristics?
• Counting occurrences of data values, perhaps