EFFICIENT EXPLICIT-STATE MODEL CHECKING FOR PROGRAMS WITH DYNAMICALLY ALLOCATED DATA

1
EFFICIENT EXPLICIT-STATE MODEL CHECKING FOR
PROGRAMS WITH DYNAMICALLY ALLOCATED DATA

• Marcelo dAmorim
• Ph.D. thesis committee
  Gul Agha
• Jennifer Hou
• Darko Marinov (advisor, chair)
• Mahesh Viswanathan
• August 06, 2007

2
Testing is important

• Programmers develop software with lots of errors
• Testing is the dominant approach in industry to
  assure software quality
• But...testing is expensive NIS02

test automation can assist the programmer in
finding errors
3
model checkers are tools that can assist in the
automation of software testing
goal improve model checking for software testing
4
Reminder about Prelim exam

• Work presented at Prelim
• Symclat (ASE06 paper)
• Improved software testing without model checking
• Mixed Execution (ICFEM06 paper)
• More efficient model checking
• Delta Execution Idea
• More efficient model checking
• Work done since Prelim
• Delta Execution (ISSTA07 paper)

focus of my thesis
5
Model Checking and Testing
model
interesting behaviors
model checker
property

• Model describes behaviors of a system
• description includes states and transition
• The model checker performs state-space
  exploration on the input model
• performs systematic testing

6
Design and implementation models

• Traditionally, model checkers operate on design
  models
• FDR Ros94, SPIN Hol97, NuSMV CCGR99, Alloy
  Analyzer Jac00
• Recently, more common for model checkers to
  operate on programs
• JPF JV01, CMC MPC02, Bogor RDH03,
  SpecExplorer VCST05

7
Design and implementation models

• Traditionally, model checkers operate on design
  models
• FDR Ros94, SPIN Hol97, NuSMV CCGR99, Alloy
  Analyzer Jac00
• Recently, more common for model checkers to
  operate on programs
• JPF JV01, CMC MPC02, Bogor RDH03,
  SpecExplorer VCST05

focus
8
Goal

speed up state-space exploration for programs
with dynamically allocated data (/OO programs)
9
Example test subject

• BinarySearchTree (BST) and Node classes

state
public class BinarySearchTree private Node
root private int size public void add(int
info) public boolean remove(int info)
class Node Node left, right int
info Node(int info) this.info info
1
2
2
operations
10
Example test sequence
sequences of method calls from the initial state

• Examples of sequences

BinarySearchTree bst new BinarySearchTree()
bst.add(1) bst.add(2)
BinarySearchTree bst new BinarySearchTree()
bst.add(2) bst.remove(1)
11
Example scenario Bounded-Exhaustive Exploration

• explore all sequences up to given bounds
• length of sequence
• range of values

• Examples of sequences

BinarySearchTree bst new BinarySearchTree()
bst.add(1) bst.add(2)
BinarySearchTree bst new BinarySearchTree()
bst.add(2) bst.remove(1)
12
Example scenario Bounded-Exhaustive Exploration
bounds
// L length of sequence // N number of input
values // in most experiments L N public
static void main(int L, int N) BST bst new
BST() // empty tree for (int i 0 i lt L
i) int methNum Verify.getInt(0, 1)
int value Verify.getInt(1, N) switch
(methNum) case 0 bst.add(value) break
case 1 bst.remove(value) break
stopIfVisited(bst)
13
Example Bounded-Exhaustive Exploration
0
L N 2
?
remove(2)
add(1)
add(2)
remove(1)

• chooses method and value

14
Example Bounded-Exhaustive Exploration
0
L N 2
?
remove(2)
add(1)
add(2)
remove(1)
1
1

• chooses method and value
• executes transition

15
Example Bounded-Exhaustive Exploration
0
L N 2
?
add(2)
remove(2)
add(1)
remove(1)
1
1

• chooses method and value
• executes transition
• explores paths (backtracking)

16
Example Bounded-Exhaustive Exploration
0
L N 2
?
add(2)
remove(2)
add(1)
remove(1)
2
1
1
1

• chooses method and value
• executes transition
• explores paths (backtracking)

17
Example Bounded-Exhaustive Exploration
0
L N 2
?
add(2)
remove(2)
add(1)
remove(1)
0
2
1
1
1

• chooses method and value
• executes transition
• explores paths (backtracking)
• prunes paths (compares state)

18
Example Bounded-Exhaustive Exploration
0
L N 2
?
add(2)
remove(2)
add(1)
remove(1)
0
0
2
1
1
1

• chooses method and value
• executes transition
• explores paths (backtracking)
• prunes paths (compares state)

19
Example Bounded-Exhaustive Exploration
0
L N 2
?
add(2)
remove(2)
add(1)
remove(1)
0
0
2
1
1
1
?

add(1)

• chooses method and value
• executes transition
• explores paths (backtracking)
• prunes paths (compares state)

1
1
20
Example Bounded-Exhaustive Exploration
0
L N 2
?
add(2)
remove(2)
add(1)
remove(1)
0
0
2
1
1
1
?

add(1)

• chooses method and value
• executes transition
• explores paths (backtracking)
• prunes paths (compares state)

add(2)
1
1
2
1
2
21
Operations

• Costly operations for an important class of model
  checkers
• execution of transitions
• exploration of paths (backtracking)
• pruning of paths (state comparison)

specific goal is to speed up these operations
22
Our contributions

• Two techniques
• Mixed execution
• Delta execution (?Execution)
• Overall time reduction
• execution (mixed and ?)
• backtracking (?)
• state comparison (?)

23
Outline

• Introduction Goal
• ?Execution
• Mixed execution (brief)
• Related Work and Conclusions

24
Our Technique ?Execution

• Observation
• Many of the execution paths in state-space
  exploration partially overlap
• Different states can hold same values

?Execution exploits these overlaps across
executions and states to speed up model checking
25
Example

• Consider separately invoking the method call
  add(4) on the following two Binary Search Trees

BST 2
BST 1
3
3
2
1
3
1
3
2
26
Example
1 public void add(int elem) 2 if (root
null) 3 root new Node(elem) 4 else 5
for (Node temp root true ) 6 if
(temp.info lt elem) 7 if (temp.right
null) 8 temp.right new Node(elem)
9 break 10 else temp
temp.right 11 else if (temp.info gt elem)
12 if (temp.left null) 13
temp.left new Node(elem) 14
break 15 else temp temp.left 16
else return // no duplicates 17
size 18
27
Example
1 public void add(int elem) 2 if (root
null) 3 root new Node(elem) 4 else 5
for (Node temp root true ) 6 if
(temp.info lt elem) 7 if (temp.right
null) 8 temp.right new Node(elem)
9 break 10 else temp
temp.right 11 else if (temp.info gt elem)
12 if (temp.left null) 13
temp.left new Node(elem) 14
break 15 else temp temp.left 16
else return // no duplicates 17
size 18
28
Example
3
1 public void add(int elem) 2 if (root
null) 3 root new Node(elem) 4 else 5
for (Node temp root true ) 6 if
(temp.info lt elem) 7 if (temp.right
null) 8 temp.right new Node(elem)
9 break 10 else temp
temp.right 11 else if (temp.info gt elem)
12 if (temp.left null) 13
temp.left new Node(elem) 14
break 15 else temp temp.left 16
else return // no duplicates 17
size 18
1
3
2
4
1
3
2
4
29
Example Execution
overlap across states
4
4
1
2
overlap across executions
3
3
1
2
4
4
Execution Trace
, 2
1
, 4
, 5
, 6
, 7
, 10
5
, 6
, 7
, 8
, 9
, 17
30
?Execution

• Technique uses set of states to perform
  state-space exploration
• Exploits the overlapping on states and executions
• Reduce time
• execution of transitions
• exploration of paths (backtracking)
• pruning of paths (state comparison)

31
0
standard exploration
?
0
?
?
0
32
0
standard exploration
?
0
?
?
0
delta exploration
33
Back to our example
3
2
3
1
only one execution
add(4)
only one update on field size
4
4
2
1
3
1
3
2
4
4
34
?Execution Set Operations

• Merge merges sets of states into one ?State
• Split splits a set of states into two subsets

35
Combined Execution Splitting
add(4)
36
Combined Execution Splitting
add(4)
splits on root.right null
split
37
Combined Execution Splitting
add(4)
splits on root.right null
split
38
Combined Execution Splitting
add(4)
splits on root.right null
split
splits on root.right.right null
split
39
Combined Execution Splitting
add(4)
splits on root.right null
split
splits on root.right.right null
split
40
Combined Execution Splitting
add(4)
splits on root.right null
split
splits on root.right.right null
split
reduction in number of executions standard has
5, delta has 3
41
Merging
merges
merge
42
Split Merging

• Splits during execution
• Merges sets of post-states from execution

43
Efficient ?Execution

• Representation of set of concrete states
• Program instrumentation
• Optimized state comparison

44
?Execution State representation

• Represents a set of concrete states

conceptual representation
45
Representing state the ?State
3
3
3
3
3
3
3
2
1
1
?Objects
3
3
3
1
1
2
46
Representing state the ?State
3
3
3
3
3
3
3
2
1
1
2
1
1
3
3
3
1
1
2
?Objects for references
2
1
?
?
1
47
Representing state the ?State
3
3
3
3
3
3
3
2
1
1
2
1
1
3
3
2
3
3
3
1
1
2
2
1
?
?
1
?
?
2
3
3
48
Representing state the ?State
3
3
3
1
1
2
2
1
?
?
1
?
?
2
3
3
?
?
?
?
?
?
?
?
1
2
2
3
49
Representing state the ?State
3
3
3
1
1
2
2
1
?
?
1
?
?
?
?
1
2
2
3
50
Constants
becomes constant after split due to root.right
null
3
constants created during merging
1
2
2
3
51
Optimized State Comparison

• State comparison in model checkers
• linearization hashing
• use the hash for comparison in a hashtable

52
Optimized State Comparison
100 1 101 2 101 3 101 4 -1 -1 - 1 -1 1 2 3 3
53
Optimized State Comparison
100 1 101 2 101 3 101 4 -1 -1 - 1 -1 1 2 3 3
100 1 101 2 101 3 -1 101 4 -1 - 1 -1 2 1 3 3
54
Optimized State Comparison
100 1 101 2 101 3 101 4 -1 -1 - 1 -1 1 2 3 3
100 1 101 2 101 3 -1 101 4 -1 - 1 -1 2 1 3 3
100 1 101 2 101 3 -1 -1 101 4 -1 -1 2 1 3 3
55
Optimized State Comparison
100 1 101 2 101 3 101 4 -1 -1 - 1 -1 1 2 3 3
100 1 101 2 101 3 -1 101 4 -1 - 1 -1 2 1 3 3
100 1 101 2 101 3 -1 -1 101 4 -1 -1 2 1 3 3
100 1 101 2 -1 101 3 101 4 -1 -1 -1 2 1 3 3
56
Optimized State Comparison
Observation significant sharing of data!
100 1 101 2 101 3 101 4 -1 -1 - 1 -1 1 2 3 3
100 1 101 2 101 3 -1 101 4 -1 - 1 -1 2 1 3 3
100 1 101 2 101 3 -1 -1 101 4 -1 -1 2 1 3 3
100 1 101 2 -1 101 3 101 4 -1 -1 -1 2 1 3 3
100 1 101 2 -1 101 3 -1 101 4 -1 -1 2 1 3 3
57
Program modification

• Primitives are replaced with ?Objects that
  represent sets of primitive values
• Object references are replaced with generated
  ?Objects that represent sets of references for
  a particular type
• Operations (, -, lt, , etc.) perform on
  ?Objects (i.e., sets of values)

58
Example Modifications
public class BST private DeltaNode root
DeltaNode.NULL private DeltaInt size
DeltaInt.ZERO public void add(DeltaInt
info) // original if (root null) if
(root.eq(DeltaNode.NULL)) else
public DeltaBoolean remove(DeltaInt info) ...
class Node DeltaNode left, right
DeltaInt info Node(DeltaInt info) this.info
info
59
Example Modifications
public class BST private DeltaNode root
DeltaNode.NULL private DeltaInt size
DeltaInt.ZERO public void add(DeltaInt
info) // original if (root null) if
(root.eq(DeltaNode.NULL)) else
public DeltaBoolean remove(DeltaInt info) ...
class Node DeltaNode left, right
DeltaInt info Node(DeltaInt info) this.info
info
reference equality on a ?Object
60
Example Equality of Reference

• eq on the following objects

OBJ-X
OBJ-X
returns true
61
Example Equality of Reference

• eq on the following objects

OBJ-X
OBJ-X
returns false disable all but state 3
split
returns true disable the state 3
62
Summary of time reduction

• Execution of transitions
• Reduction of redundant operations
• Path exploration (backtracking)
• Reduction in number of executions
• Path pruning (state comparison)
• Simultaneous comparison of several states

63
Evaluation

• Conducted experiment on 10 data structures and
  one larger case study, AODV
• Implemented in two model checkers
• Java PathFinder
• BOX (Bounded Object eXplorer)
• Performed bounded-exhaustive exploration using
  breadth-first search

64
Subjects

• Ten basic subject implementations
• BinHeap Visser et al. 2006
• BinarySearchTree Boyapati et al. 2002, Xie et
  al. 2005
• Deque
• FibHeap Visser et al. 2006
• FileSystem Daisy File System, Qadeer 2004
• HeapArray Boyapati et al. 2002, Xie et. al.
  2005
• Queue Darga Boyapati 2006
• Stack Darga Boyapati 2006
• TreeMap Java Collections 1.4
• UBStack Csallner Smaragdakis 2004, Pacheco
  Ernst 2005
• One case-study, a network routing protocol
• Ad-hoc On-Demand Distance Vector (AODV)
  Perkins Royer 1999, Musuvathi et al. 2002,
  Sobeih et al. 2005

65
Results for JPF Implementation
Executions
Exploration Time (sec)
States
Subject-Bound
??Exec
Std
Standard
Std / ?
??Exec
863
4001328
458.81
250083
38.50x
11.91
binheap-8
22688
4127900
214.06
206395
7.11x
30.13
bst-10
810
11223540
623530
19.14x
28.84
552.11
deque-9
209
4901931
544659
18.57x
21.59
400.84x
fibheap-8
1568
194832
1353
5.59x
3.08
17.18
filesystem-4
359
8048090
804809
126.80x
21.49
2724.63
heaparray-9
60
1183960
147995
16.63x
5.08
84.42
queue-7
56
1098056
137257
14.43x
4.14
59.70
stack-7
5269
778910
35405
9.63x
9.43
90.80
treemap-11
931
9911890
991189
46.17x
32.54
1502.24
ubstack-9
10.79x
GMEAN

• 11 additional cases using other bounds (21 cases
  total)

66
Results for JPF Implementation

• 11 additional cases using other bounds (21 cases
  total)

67
Results for JPF Implementation

• 11 additional cases using other bounds (21 cases
  total)

68
Results for JPF Implementation

• 11 additional cases using other bounds (21 cases
  total)

69
Time breakdown for JPF

• fibheap-8

due to constants
due to reduction in number of executions
due to optimized linearization
70
Results for data structures

• JPF exploration time improvement 10.79x
• JPF peak memory usage reduction 33
• BOX exploration time improvement 2.07x
• BOX peak memory usage increase -3

71
AODV Case Study

• AODV routing protocol for wireless networks
  written in the J-Sim network simulator
• 43 classes over 3500 lines of code
• Three variations examined each containing an
  error leading to a property violation
• routes should be cycle free
• Requires at least 8 transitions to show error
• Exploration time improved up to1.53x

72
Outline

• Introduction Goal
• ?Execution
• Mixed execution (brief)
• Related Work and Conclusions

73
Mixed Execution

• Goal Speed up execution operation in model
  checkers with special state representation
• Representation of program state the model checker
  defines so to directly read and write data
• Impact model checking operations
• execution of transitions
• exploration of paths (backtracking)
• pruning of paths (state comparison)
• Examples JPF JV01, BogorVM RDH03,
  SpecExplorer VCST05

74
Mixed Execution

• Technique Executes code on native state
  representation where possible
• Translates between special and native state
  representations
• Works on deterministic blocks only
• model checker does not track state changes
• butconcurrent code contains atomic blocks
• Implementation Java PathFinder (JPF)

75
Java PathFinder (JPF)

• Implements a backtrackable JVM for explicit-state
  model checking of Java bytecodes
• Implemented in Java
• Executes on top of a regular, host JVM

Java program (classfiles)
JPF
JVM
76
JPF Special State Representation

• Represents the heap as array of int. arrays
• Each object encoded with integer id
• Primitive and reference fields encoded as
  integers (distinguished by types)

JPF state
native state
size
root

• 96 2, 97
• 97 1, -1, 98
• 98 2, -1, -1

1
2
tree
node (1)
node (2)
2
left
info
right ( null)
77
Example Tree
native state
JPF state
96 2, 97 97 1, -1, 98 98 2, -1, -1
1
2
2
start from a tree object with integers 1 and 2
invoke add(0)
78
Example Tree
native state
JPF state
96 2, 97 97 1, -1, 98 98 2, -1, -1
1
2
2
JPF execution mutates two integer arrays and
creates another
add(0)
96 3, 97 97 1, 99, 98 98 2, -1,
-1 99 0, -1, -1
1
3
2
0
79
Example Tree
JPF state
96 2, 97 97 1, -1, 98 98 2, -1, -1

• execution on JPF state representation is slow
• special-state
• interpretation

add(0)
96 3, 97 97 1, 99, 98 98 2, -1,
-1 99 0, -1, -1
80
Example Tree
JPF state
native state
96 2, 97 97 1, -1, 98 98 2, -1, -1
1
2
2
add(0)
mixed execution
1
96 3, 97 97 1, 99, 98 98 2, -1,
-1 99 0, -1, -1
3
2
0
81
Evaluation

• Conducted experiments in 7 data structures and
  one larger case study, AODV
• Bounded-Exhaustive Exploration
• Results
• Improved execution time up to 67.2
• Improved overall exploration time up to 42.2
• Speedup depends on bound, scales wellthe larger
  the bound, the higher the speedup

82
Outline

• Introduction Goal
• ?Execution
• Mixed execution
• Related Work and Conclusions

83
Related Work

• Reduction in state space size
• abstraction
• Predicate Abstraction, Program Slicing
• path pruning
• Heap Symmetry Reduction, Thread Symmetry
  Reduction, Partial-Order Reduction
• Recent for OO Abstract Matching VPP06,
  Glass-Box Model Checking DB06

84
Related Work

• Time reduction
• execution
• symbolic execution
• path pruning
• Incremental Hashing MD05

85
Next Steps

• Alternative ways to
• execute, backtrack, compare state

86
Conclusions

• Techniques improve overall exploration time
• ?Execution, on average 10x
• Mixed Execution, up to 42
• Techniques address costly operations
• execution (mixed and ?)
• backtracking (?)
• state comparison (?)

87
Publications on thesis work

• Testing
• Delta Execution for Efficient State-Space
  Exploration of Object-Oriented Programs.Marcelo
  d'Amorim, Steven Lauterburg and Darko Marinov.
  (ISSTA'07) London, UK, July, 2007. (Nominated to
  the list of best papers)
• Invited to the IEEE TSE journal, in preparation
• Delta Execution for Software Reliability.Yuanyuan
  Zhou, Darko Marinov, William Sanders, Craig
  Zilles,Marcelo d'Amorim, Steven Lauterburg, Ryan
  M. Lefever and Joe Tucek (HotDep'07) Edinburgh,
  UK, June, 2007.
• Optimized Execution of Deterministic Blocks in
  Java PathFinder.Marcelo d'Amorim, Ahmed Sobeih
  and Darko Marinov. (ICFEM'06) Macau, SAR,
  November, 2006.
• An Empirical Comparison of Automated Generation
  and Classification Techniques for Object-Oriented
  Unit Testing.Marcelo d'Amorim, Carlos Pacheco,
  Darko Marinov, Tao Xie, and Michael D. Ernst.
  (ASE'06) Tokyo, Japan, September, 2006.
  (Nominated to the list of best papers)
• Invited to the ASE journal, submitted

88
Publications on related work

• Runtime Verification
• Efficient Monitoring of Omega-Languages.Marcelo
  d'Amorim and Grigore Rosu. (CAV'05) Edinburgh,
  Scotland, July 2005
• Checking and Correcting Behaviors of Java
  Programs at Runtime with Java-MOP.Feng Chen,
  Marcelo d'Amorim and Grigore Rosu.In Proc. of
  the 5th Workshop on Runtime Verification (RV'05)
  Edinburgh, Scotland, July, 2005.
• Event-Based Runtime Verification of Java
  Programs.Marcelo d'Amorim and Klaus Havelund.
  (WODA'05) St. Louis, U.S., May, 2005.
• A Formal Monitoring-based Framework for Software
  Development and Analysis.Feng Chen, Marcelo
  d'Amorim, and Grigore Rosu. (ICFEM'04) Seattle,
  U.S., November 2004.
• Other in SE and PL
• An Equational Specification for the Scheme
  Language.Marcelo d'Amorim and Grigore Rosu.In
  Proc. of the 9th Brazilian Symposium on
  Programming Languages (SBLP'05) Recife, Brazil,
  June, 2005.
• Integrating Code Generation and Refactoring.M.
  d'Amorim, C. Nogueira, G. Santos, A. Souza, and
  P. Borba.In Proc. of the Workshop on Generative
  Programming(ECOOP'02) Malaga, Spain, June 2002.

89
Areas of interest

• Productivity in SE
• Testing Runtime Verification (Ph.D. studies)
• Effectiveness
• Efficiency (Ph.D. dissertation)
• Refactoring and Program Generation (pre Ph.D.)