A Tutorial on Automated Verification Tevfik Bultan

1
A Tutorial on Automated VerificationTevfik
Bultan
2
Who are these people and what do they have in
common?

• 2007 Clarke, Edmund M
• 2007 Emerson, E Allen
• 2007 Sifakis, Joseph
• 1996 Pnueli, Amir
• 1991 Milner, Robin
• 1980 Hoare, C. Antony R.
• 1978 Floyd, Robert W
• 1972 Dijkstra, E. W.

3
Outline

• Softwares Chronic Crisis
• Temporal Logics and Model Checking Problem
• Symbolic Model Checking
• Automata Theoretic Model Checking
• Software Verification Using Explicit State Model
  Checking with Java Path Finder
• Bounded Model Checking
• Symbolic Software Model Checking with Predicate
  Abstraction and Counter-Example Guided
  Abstraction Refinement

4
Softwares Chronic Crisis

• Large software systems often
• Do not provide the desired functionality
• Take too long to build
• Cost too much to build
• Require too much resources (time, space) to run
• Cannot evolve to meet changing needs
• For every 6 large software projects that become
  operational, 2 of them are canceled
• On the average software development projects
  overshoot their schedule by half
• 3 quarters of the large systems do not provide
  required functionality

5
Software Failures

• There is a long list of failed software projects
  and software failures
• You can find a list of famous software bugs at
• http//www5.in.tum.de/huckle/bugse.html
• I will talk about two famous and interesting
  software bugs

6
Ariane 5 Failure

• A software bug caused European Space Agencys
  Ariane 5 rocket to crash 40 seconds into its
  first flight in 1996 (cost half billion dollars)

• The bug was caused because of a software
  component that was being reused from Ariane 4
• A software exception occurred during execution of
  a data conversion from 64-bit floating point to
  16-bit signed integer value
• The value was larger than 32,767, the largest
  integer storable in a 16 bit signed integer, and
  thus the conversion failed and an exception was
  raised by the program
• When the primary computer system failed due to
  this problem, the secondary system started
  running.
• The secondary system was running the same
  software, so it failed too!

7
Ariane 5 Failure

• The programmers for Ariane 4 had decided that
  this particular velocity figure would never be
  large enough to raise this exception.
• Ariane 5 was a faster rocket than Ariane 4!
• The calculation containing the bug actually
  served no purpose once the rocket was in the air.
  
• Engineers chose long ago, in an earlier version
  of the Ariane rocket, to leave this function
  running for the first 40 seconds of flight to
  make it easy to restart the system in the event
  of a brief hold in the countdown.
• You can read the report of Ariane 5 failure at
• http//www.ima.umn.edu/arnold/disasters/aria
  ne5rep.html

8
Mars Pathfinder

• A few days into its mission, NASAs Mars
  Pathfinder computer system started rebooting
  itself
• Cause Priority inversion during preemptive
  priority scheduling of threads

• Priority inversion occurs when
• a thread that has higher priority is waiting for
  a resource held by thread with a lower priority
• Pathfinder contained a data bus shared among
  multiple threads and protected by a mutex lock
• Two threads that accessed the data bus were a
  high-priority bus management thread and a
  low-priority meteorological data gathering thread
  
• Yet another thread with medium-priority was a
  long running communications thread (which did not
  access the data bus)

9
Mars Pathfinder

• The scenario that caused the reboot was
• The meteorological data gathering thread accesses
  the bus and obtains the mutex lock
• While the meteorological data gathering thread is
  accessing the bus, an interrupt causes the
  high-priority bus management thread to be
  scheduled
• Bus management thread tries to access the bus and
  blocks on the mutex lock
• Scheduler starts running the meteorological
  thread again
• Before the meteorological thread finishes its
  task yet another interrupt occurs and the
  medium-priority (and long running) communications
  thread gets scheduled
• At this point high-priority bus management thread
  is waiting for the low-priority meteorological
  data gathering thread, and the low-priority
  meteorological data gathering thread is waiting
  for the medium-priority communications thread
• Since communications thread had long-running
  tasks, after a while a watchdog timer would go
  off and notice that the high-priority bus
  management thread has not been executed for some
  time and conclude that something was wrong and
  reboot the system

10
Softwares Chronic Crisis

• Software product size is increasing exponentially
• faster, smaller, cheaper hardware
• Software is everywhere from TV sets to
  cell-phones
• Software is in safety-critical systems
• cars, airplanes, nuclear-power plants
• We are seeing more of
• distributed systems
• embedded systems
• real-time systems
• These kinds of systems are harder to build
• Software requirements change
• software evolves rather than being built

11
Summary

• Softwares chronic crisis Development of large
  software systems is a challenging task
• Large software systems often Do not provide the
  desired functionality Take too long to build
  Cost too much to build Require too much resources
  (time, space) to run Cannot evolve to meet
  changing needs

12
What is this?
13
First Computer Bug

• In 1947, Grace Murray Hopper was working on the
  Harvard University Mark II Aiken Relay Calculator
  (a primitive computer).
• On the 9th of September, 1947, when the machine
  was experiencing problems, an investigation
  showed that there was a moth trapped between the
  points of Relay 70, in Panel F.
• The operators removed the moth and affixed it to
  the log. The entry reads "First actual case of
  bug being found."
• The word went out that they had "debugged" the
  machine and the term "debugging a computer
  program" was born.

14
Can Model Checking Help

• The question is Can the automated verification
  techniques we have been discussing be used in
  finding bugs in software systems?
• Today I will discuss some automated verification
  techniques that have been successful in
  identifying bugs

15
State of the art in automated verification
Model Checking

• What is model checking?
• Automated verification technique
• Focuses on bug finding rather than proving
  correctness
• The basic idea is to exhaustively search for bugs
  in software
• Has many flavors
• Explicit-state model checking
• Symbolic model checking
• Bounded model checking

16
Model Checking Evolution

• Earlier model checkers had their own input
  specification languages
• For example Spin, SMV
• This requires translation of the system to be
  verified to the input langauge of the model
  checker
• Most of the time these translations are not
  automated and use ad-hoc simplifications and
  abstractions
• More recently several researchers developed tools
  for model checking programs
• These model checkers work directly on programs,
  i.e., their input language is a programming
  language
• These model checkers use well-defined techniques
  for restricting the state space or use automated
  abstraction techniques

17
Explicit-State Model Checking Programs

• Verisoft from Bell Labs
• C programs, handles concurrency, bounded search,
  bounded recursion.
• Uses stateless search and partial order
  reduction.
• Java Path Finder (JPF) at NASA Ames
• Explicit state model checking for Java programs,
  bounded search, bounded recursion, handles
  concurrency.
• Uses techniques similar to the techniques used in
  Spin.
• CMC from Stanford for checking systems code
  written in C

18
Symbolic Model Checking of Programs

• CBMC
• This is the bounded model checker we discussed
  earlier, bounds the loop iterations and recursion
  depth.
• Uses a SAT solver.
• SLAM project at Microsoft Research
• Symbolic model checking for C programs. Can
  handle unbounded recursion but does not handle
  concurrency.
• Uses predicate abstraction and BDDs.

19
Beyond Model Checking

• Promising results obtained in the model checking
  area created a new interest in automated
  verification
• Nowadays, there is a wide spectrum of
  verification/analysis/testing techniques with
  varying levels of power and scalability
• Bounded verification using SAT solvers
• Symbolic execution using combinations of decision
  procedures
• Dynamic symbolic execution (aka concolic
  execution)
• Various types of symbolic analysis shape
  analysis, string analysis, size analysis, etc.

20
What to Verify?

• Before we start talking about automated
  verification techniques, we need to identify what
  we want to verify
• It turns out that this is not a very simple
  question
• First we will discuss issues related to this
  question

21
Temporal Logics and Model Checking Problem
22
A Mutual Exclusion Protocol

• Two concurrently executing processes are trying
  to enter a critical section without violating
  mutual exclusion

Process 1 while (true) out a true
turn true wait await (!b or !turn)
cs a false Process 2 while (true)
out b true turn false wait await
(!a or turn) cs b false
23
Reactive Systems A Very Simple Model

• We will use a very simple model for reactive
  systems
• A reactive system generates a set of execution
  paths
• An execution path is a concatenation of the
  states (configurations) of the system, starting
  from some initial state
• There is a transition relation which specifies
  the next-state relation, i.e., given a state what
  are the states that can follow that state

24
State Space

• The state space of a program can be captured by
  the valuations of the variables and the program
  counters
• For our example, we have
• two program counters pc1, pc2
• domains of the program counters out, wait, cs
• three boolean variables turn, a, b
• boolean domain True, False
• Each state of the program is a valuation of all
  the variables

25
State Space

• Each state can be written as a tuple
  (pc1,pc2,turn,a,b)
• Initial states (o,o,F,F,F), (o,o,F,F,T),
  (o,o,F,T,F), (o,o,F,T,T), (o,o,T,F,F),
  (o,o,T,F,T), (o,o,T,T,F), (o,o,T,T,T)
• initially pc1o and pc2o
• How many states total?
• 3 3 2 2 2 72
• exponential in the number of variables and the
  number of concurrent components

26
Transition Relation

• Transition Relation specifies the next-state
  relation, i.e., given a state what are the states
  that can come immediately after that state
• For example, given the initial state (o,o,F,F,F)
• Process 1 can execute
• out a true turn true
• or Process 2 can execute
• out b true turn false
• If process 1 executes, the next state is
  (w,o,T,T,F)
• If process 2 executes, the next state is
  (o,w,F,F,T)
• So the state pairs ((o,o,F,F,F),(w,o,T,T,F)) and
  ((o,o,F,F,F),(o,w,F,F,T)) are included in the
  transition relation

27
Transition Relation

• The transition relation is like a graph, edges
  represent the next-state relation

(o,o,F,F,F)
(w,o,T,T,F)
(o,w,F,F,T)
(w,w,T,T,T)
(o,c,F,F,T)
28
Transition System

• A transition system T (S, I, R) consists of
• a set of states S
• a set of initial states I ? S
• and a transition relation R ? S ? S
• A common assumption in model checking
• R is total, i.e., for all s ? S, there exists s
  such
• that (s,s) ? R

29
Execution Paths

• An execution path is an infinite sequence of
  states
• x s0, s1, s2, ...
• such that
• s0 ? I and for all i ? 0, (si,si1) ? R
• Notation For any path x
• xi denotes the ith state on the path (i.e., si)
• xi denotes the ith suffix of the path (i.e., si,
  si1, si2, ... )

30
Execution Paths

• A possible execution path
• ((o,o,F,F,F), (o,w,F,F,T), (o,c,F,F,T))?
• (? means repeat the above three states infinitely
  many times)

(o,o,F,F,F)
(w,o,T,T,F)
(o,w,F,F,T)
(w,w,T,T,T)
(o,c,F,F,T)
31
Temporal Logics

• Pnueli proposed using temporal logics for
  reasoning about the properties of reactive
  systems
• Temporal logics are a type of modal logics
• Modal logics were developed to express modalities
  such as necessity or possibility
• Temporal logics focus on the modality of temporal
  progression
• Temporal logics can be used to express, for
  example, that
• an assertion is an invariant (i.e., it is true
  all the time)
• an assertion eventually becomes true (i.e., it
  will become true sometime in the future)

32
Temporal Logics

• We will assume that there is a set of basic
  (atomic) properties called AP
• These are used to write the basic (non-temporal)
  assertions about the program
• Examples atrue, pc0c, xy1
• We will use the usual boolean connectives ? , ?
  , ?
• We will also use four temporal operators
• Invariant p G p (aka p)
  (Globally)
• Eventually p F p (aka p) (Future)
• Next p X p (aka p) (neXt)
• p Until q p U q

33
LTL Properties
. . .
X p
p
. . .
G p
p
p
p
p
p
p
. . .
F p
p
. . .
p U q
p
p
p
p
q
34
Example Properties

• mutual exclusion G ( ? (pc1c ? pc2c))
• starvation freedom
• G(pc1w ? F(pc1c)) ? G(pc2w ? F(pc2c))
• Given the execution path
• x ((o,o,F,F,F), (o,w,F,F,T), (o,c,F,F,T))?
• x pc1o
• x X (pc2w)
• x F (pc2c)
• x (?turn) U (pc2c ? b)
• x G ( ? (pc1c ? pc2c))
• x G(pc1w ? F(pc1c)) ? G(pc2w ? F(pc2c))

35
LTL Model Checking

• Given a transition system T and an LTL property p
  
• T p iff for all execution paths x in T, x p
• For example
• T ? G ( ? (pc1c ? pc2c))
• T ? G(pc1w ? F(pc1c)) ? G(pc2w ? F(pc2c))
  
• Model checking problem Given a transition system
  T and an LTL property p, determine if T is a
  model for p (i.e., if T p)
• Complexity (SR) ? 2O(f)

36
Linear Time vs. Branching Time

• In linear time logics we look at the execution
  paths individually
• In branching time logics we view the computation
  as a tree
• computation tree unroll the transition relation

Transition System
Execution Paths
Computation Tree
s3
s3
s3
s2
s1
s4
s3
s1
s4
s4
s1
s2
s3
s3
s2
. . .
s3
s4
s1
s3
. . .
. . .
. . .
. . .
s4
s1
. . .
. . .
. . .
37
Computation Tree Logic (CTL)

• In CTL we quantify over the paths in the
  computation tree
• We use the same four temporal operators X, G, F,
  U
• However we attach path quantifiers to these
  temporal operators
• A for all paths
• E there exists a path
• We end up with eight temporal operators
• AX, EX, AG, EG, AF, EF, AU, EU

38
CTL Properties
Transition System
Computation Tree
p
s3
p
p
s2
s1
s4
s3
s1
p
s4
s2
p
s3
s3 p s4 p s1 ? p s2 ? p
s3 EX p s3 EX ? p s3 ? AX p s3 ? AX ?
p s3 EG p s3 ? EG ? p s3 AF p s3 EF ?
p s3 ? AF ? p
p
s3
p
s4
s1
. . .
. . .
. . .
p
s4
s1
. . .
. . .
. . .
39
CTL Model Checking

• Given a transition system T (S, I, R) and a CTL
  property p
• T p iff for all initial state s ? I, s p
• Model checking problem Given a transition system
  T and a CTL property p, determine if T is a model
  for p (i.e., if T p)
• Complexity O(f ? (SR))
• For example
• T ? AG ( ? (pc1c ? pc2c))
• T ? AG(pc1w ? AF(pc1c)) ? AG(pc2w ?
  AF(pc2c))
• Question Are CTL and LTL equivalent?

40
CTL vs. LTL

• CTL and LTL are not equivalent
• There are properties that can be expressed in LTL
  but cannot be expressed in CTL
• For example FG p
• There are properties that can be expressed in CTL
  but cannot be expressed in LTL
• For example AG(EF p)
• Hence, expressive power of CTL and LTL are not
  comparable

41
Symbolic Model Checking
42
Temporal Properties ? Fixpoints Emerson and
Clarke 80

• Here are some interesting CTL equivalences
• AG p p ? AX AG p
• EG p p ? EX EG p
• AF p p ? AX AF p
• EF p p ? EX EF p
• p AU q q ? (p ? AX (p AU q))
• p EU q q ? (p ? EX (p EU q))
• Note that we wrote the CTL temporal operators in
  terms of themselves and EX and AX operators

43
Fixpoint Characterizations

• Fixpoint Characterization Equivalences
• AG p ? y . p ? AX y AG p p ? AX AG p
• EG p ? y . p ? EX y EG p p ? EX EG p
• AF p ? y . p ? AX y AF p p ? AX AF p
• EF p ? y . p ? EX y EF p p ? EX EF p
• p AU q ? y . q ? (p ? AX (y)) p AU qq ? (p
  ? AX (p AU q))
• p EU q ? y . q ? (p ? EX (y)) p EU q q ?
  (p ? EX (p EU q))

44
Least Fixpoint

• Given a monotonic function F, the least fixpoint
  ? y . F y is the limit of the following sequence
  (assuming F is ?-continuous)
• ?, F ?, F2 ?, F3 ?, ...
• If S is finite, then we can compute the least
  fixpoint using the above sequence

45
EF Fixpoint Computation

• EF p ? y . p ? EX y is the limit of the
  sequence
• ?, p?EX ?, p?EX(p?EX ?) , p?EX(p?EX(p? EX ?)) ,
  ...
• which is equivalent to
• ?, p, p ? EX p , p ? EX (p ? EX (p) ) , ...

46
EF Fixpoint Computation
p
s2
s1
s4
s3
p
Start ? 1st iteration p?EX ? s1,s4 ? EX(?)
s1,s4 ? ? s1,s4 2nd iteration p?EX(p?EX ?)
s1,s4 ? EX(s1,s4) s1,s4
?s3s1,s3,s4 3rd iteration p?EX(p?EX(p? EX
?)) s1,s4 ? EX(s1,s3,s4) s1,s4
?s2,s3,s4s1,s2,s3,s4 4th iteration p?EX(p?EX
(p?EX(p? EX ?))) s1,s4 ? EX(s1,s2,s3,s4)
s1,s4 ? s1,s2,s3,s4 s1,s2,s3,s4
47
EF Fixpoint Computation
EF(p) ? states that can reach p ? p ? EX(p) ?
EX(EX(p)) ? ...


p
EF(p)

48
Greatest Fixpoint

• Given a monotonic function F, the greatest
  fixpoint ? y . F y is the limit of the following
  sequence (assuming F is ?-continuous)
• S, F S, F2 S, F3 S, ...
• If S is finite, then we can compute the greatest
  fixpoint using the above sequence

49
EG Fixpoint Computation

• Similarly, EG p ? y . p ? EX y is the limit of
  the sequence
• S, p?EX S, p?EX(p ? EX S) , p?EX(p ? EX (p ? EX
  S)) , ...
• which is equivalent to
• S, p, p ? EX p , p ? EX (p ? EX (p) ) , ...

50
EG Fixpoint Computation
p
p
s2
s1
s4
s3
p
Start S s1,s2,s3,s4 1st iteration p?EX S
s1,s3,s4?EX(s1,s2,s3,s4) s1,s3,s4?s1,s2,s3
,s4s1,s3,s4 2nd iteration p?EX(p?EX S)
s1,s3,s4?EX(s1,s3,s4) s1,s3,s4?s2,s3,s4
s3,s4 3rd iteration p?EX(p?EX(p?EX S))
s1,s3,s4?EX(s3,s4) s1,s3,s4?s2,s3,s4s3,
s4
51
EG Fixpoint Computation
EG(p) ? states that can avoid reaching ?p ? p ?
EX(p) ? EX(EX(p)) ? ...


EG(p)
52
Symbolic Model CheckingMcMillan et al. LICS 90

• Basic idea Represent sets of states and the
  transition relation as Boolean logic formulas
• Fixpoint computation becomes formula manipulation
• pre-condition (EX) computation Existential
  variable elimination
• conjunction (intersection), disjunction (union)
  and negation (set difference), and equivalence
  check
• Use an efficient data structure for boolean logic
  formulas
• Binary Decision Diagrams (BDDs)

53
Symbolic Pre-condition Computation

• Remember the function
• EX 2S ? 2S
• which is defined as
• EX(p) s (s,s) ? R and s ? p
• We can symbolically compute pre as follows
• EX(p) ? ??V R ? pV / V
• V current-state boolean variables
• V next-state boolean variables
• pV / V rename variables in p by replacing
  current-state variables with the corresponding
  next-state variables
• ??V f existentially quantify out all the
  variables in V from f

54
An Extremely Simple Example

• Variables x, y boolean
• Set of states
• S (F,F), (F,T), (T,F), (T,T)
• S ? True
• Initial condition
• I ? ? x ? ? y
• Transition relation (negates one variable at a
  time)
• R ? x?x ? yy ? xx ? y?y ( means ?)

55
An Extremely Simple Example

• Given p ? x ? y, compute EX(p)
• EX(p) ? ??V R ? pV / V
• ? ??V R ? x ? y
• ? ??V (x?x ? yy ? xx ? y?y ) ? x ? y
• ? ??V (x?x ? yy) ? x ? y ? (xx ? y?y)
  ? x ? y
• ?V ?x ? y ? x ? y ? x ? ?y ? x ? y
• ?x ? y ? x ? ?y
• EX(x ? y) ? ?x ? y ? x ? ?y
• In other words EX((T,T)) ? (F,T), (T,F)

56
An Extremely Simple Example
3
2

• Lets compute compute EF(x ? y)
• The fixpoint sequence is
• False, x?y , x?y ? EX(x?y) , x?y ? EX (x?y
  ? EX(x?y)) , ...
• If we do the EX computations, we get
• False, x ? y , x ? y ? ?x ? y ? x ? ?y,
  True
• EF(x ? y) ? True
• In other words EF((T,T)) ? (F,F),(F,T),
  (T,F),(T,T)

1
0
1
2
3
57
An Extremely Simple Example

• Based on our results, for our extremely simple
  transition system T(S,I,R) we have
• I ? EF(x ? y) hence
• T EF(x ? y)
• (i.e., there exists a path from each initial
  state where eventually x and y both become true
  at the same time)
• I ? EX(x ? y) hence
• T EX(x ? y)
• (i.e., there does not exist a path from each
  initial state where in the next state x and y
  both become true)

58
An Extremely Simple Example

• Lets try one more property AF(x ? y)
• To check this property we first convert it to a
  formula which uses only the temporal operators in
  our basis
• AF(x ? y) ? ? EG(?(x ? y))
• If we can find an initial state which satisfies
  EG(?(x ? y)), then we know that the transition
  system T, does not satisfy the property AF(x ? y)

59
An Extremely Simple Example

• Lets compute compute EG(?(x ? y))
• The fixpoint sequence is
• True, ?x ? ?y, (?x ? ?y) ? EX(?x ? ?y) ,
  
• If we do the EX computations, we get
• True, ?x ? ?y, ?x ? ?y,
• EG(?(x ? y)) ? ?x ? ?y
• Since I ? EG(?(x ? y)) ? ? we conclude that T
  AF(x ? y)

1
0
0
1
2
60
Symbolic CTL Model Checking Algorithm

• Translate the formula to a formula which uses the
  basis
• EX p, EG p, p EU q
• Atomic formulas can be interpreted directly on
  the state representation
• For EX p compute the precondition using
  existential variable elimination as we discussed
• For EG and EU compute the fixpoints iteratively

61
SMV McMillan 93

• BDD-based symbolic model checker
• Finite state
• Temporal logic CTL
• Focus hardware verification
• Later applied to software specifications,
  protocols, etc.
• SMV has its own input specification language
• concurrency synchronous, asynchronous
• shared variables
• boolean and enumerated variables
• bounded integer variables (binary encoding)
• SMV is not efficient for integers, but that can
  be fixed
• fixed size arrays

62
SMV Language

• An SMV specification consists of a set of modules
  (one of them must be called main)
• Modules can have access to shared variables
• Modules can be composed asynchronously using the
  process keyword
• Module behaviors can be specified using the
  ASSIGN statement which assigns values to next
  values of variables in parallel
• Module behaviors can also be specified using the
  TRANS statements which allow specification of the
  transition relation as a logic formula where
  next state values are identified using the next
  keyword

63
Example Mutual Exclusion Protocol

• Two concurrently executing processes are trying
  to enter a critical section without violating
  mutual exclusion

Process 1 while (true) out a true
turn true wait await (b false or turn
false) cs a false Process
2 while (true) out b true turn
false wait await (a false or turn) cs
b false
64
Example Mutual Exclusion Protocol in SMV
MODULE process1(a,b,turn) VAR pc out, wait,
cs ASSIGN init(pc) out next(pc)
case pcout wait pcwait (!b
!turn) cs pccs out 1 pc
esac next(turn) case pcout 1
1 turn esac next(a) case
pcout 1 pccs 0 1 a
esac next(b) b FAIRNESS running
MODULE process2(a,b,turn) VAR pc out, wait,
cs ASSIGN init(pc) out next(pc)
case pcout wait pcwait (!a
turn) cs pccs out 1 pc
esac next(turn) case pcout 0
1 turn esac next(b) case
pcout 1 pccs 0 1 b
esac next(a) a FAIRNESS running
65
Example Mutual Exclusion Protocol in SMV
MODULE main VAR a boolean b boolean
turn boolean p1 process process1(a,b,turn)
p2 process process2(a,b,turn) SPEC
AG(!(p1.pccs p2.pccs)) -- AG(p1.pcwait -gt
AF(p1.pccs)) AG(p2.pcwait -gt AF(p2.pccs))

• Here is the output when I run SMV on this example
  to
• check the mutual exclusion property

smv mutex.smv -- specification AG (!(p1.pc cs
p2.pc cs)) is true resources used user
time 0.01 s, system time 0 s BDD nodes
allocated 692 Bytes allocated 1245184 BDD nodes
representing transition relation 143 6
66
Example Mutual Exclusion Protocol in SMV

• The output for the starvation freedom property

smv mutex.smv -- specification AG (p1.pc wait
-gt AF p1.pc cs) AG ... is true resources
used user time 0 s, system time 0 s BDD nodes
allocated 1251 Bytes allocated 1245184 BDD
nodes representing transition relation 143 6
67
Example Mutual Exclusion Protocol in SMV

• Lets insert an error
• change pcwait (!b !turn) cs
• to pcwait (!b turn) cs

68
smv mutex.smv -- specification AG (!(p1.pc cs
p2.pc cs)) is false -- as demonstrated by the
following execution sequence state 1.1 a 0 b
0 turn 0 p1.pc out p2.pc out stuttering s
tate 1.2 executing process p2 state 1.3 b
1 p2.pc wait executing process p2 state
1.4 p2.pc cs executing process p1 state
1.5 a 1 turn 1 p1.pc wait executing
process p1 state 1.6 p1.pc cs stuttering
resources used user time 0.01 s, system time 0
s BDD nodes allocated 1878 Bytes allocated
1245184 BDD nodes representing transition
relation 143 6
69
Symbolic Model Checking with BDDs

• BDDs are used as a data structure for encoding
  trust sets of Boolean logic formulas in symbolic
  model checking
• One can use BDD-based symbolic model checking for
  any finite state system using a Boolean encoding
  of the state space and the transition relation
• Why are we using symbolic model checking?
• We hope that the symbolic representations will be
  more compact than the explicit state
  representation on the average
• In the worst case we may not gain anything

70
Problems with BDDs

• The BDD for the transition relation could be huge
• Remember that the BDD could be exponential in the
  number of disjuncts and conjuncts
• Since we are using a Boolean encoding there could
  be a large number of conjuncts and disjuncts
• The EX computation could result in exponential
  blow-up
• Exponential in the number of existentially
  quantified variables

71
Heuristics

• Instead of computing a monolithic BDD for the
  whole transition system partition the transition
  relation in order to keep the BDD size small
• Use good variable ordering in order to keep the
  BDD sizes small
• Use heuristics to find good variable orderings,
• Use dynamic variable ordering heuristics that
  change the variable ordering dynamically if the
  BDD size grows too much
• Use other data structures (such as multi-terminal
  decision diagrams)

72
Counter-Example Generation

• Remember Given a transition system T (S, I, R)
  and a CTL property p T p iff for all initial
  state s ? I, s p
• Verification vs. Falsification
• Verification
• Show initial states ? truth set of p
• Falsification
• Find a state ? initial states ? truth set of ?p
• Generate a counter-example starting from that
  state
• The ability to find counter-examples is one of
  the biggest strengths of the model checkers

73
An Example

• We want to check the property AG(p)
• We compute the fixpoint for EF(?p)
• We check if the intersection of the set of
  initial states I and the truth set of EF(?p) is
  empty
• If it is not empty we generate a counter-example
  path starting from the intersection

EF(?p) ? states that can reach ?p ? ?p ?
EX(?p) ? EX(EX(?p)) ? ...

I

• In order to generate the
• counter-example path, save
• the fixpoint iterations.
• After the fixpoint computation
• converges, do a second pass
• to generate the counter-example path.

?p
EF(?p)

Generate a counter-example path starting from a
state here
74
Automata Theoretic Model Checking
75
LTL Properties ? Büchi automata Vardi and
Wolper LICS 86

• Büchi automata Finite state automata that accept
  infinite strings
• The better known variant of finite state automata
  accept finite strings (used in lexical analysis
  for example)
• A Büchi automaton accepts a string when the
  corresponding run visits an accepting state
  infinitely often
• Note that an infinite run never ends, so we
  cannot say that an accepting run ends at an
  accepting state
• LTL properties can be translated to Büchi
  automata
• The automaton accepts a path if and only if the
  path satisfies the corresponding LTL property

76
LTL Properties ? Büchi automata
true
p
G p
?p
true
F p
p
?p
p
p
?p
G (F p)
?p

• The size of the property automaton can be
  exponential in the size of the LTL formula
  (recall the complexity of LTL model checking)

77
Büchi Automata Language Emptiness Check

• Given a Buchi automaton, one interesting question
  is
• Is the language accepted by the automaton empty?
• i.e., does it accept any string?
• A Büchi automaton accepts a string when the
  corresponding run visits an accepting state
  infinitely often
• To check emptiness
• Look for a cycle which contains an accepting
  state and is reachable from the initial state
• Find a strongly connected component that contains
  an accepting state, and is reachable from the
  initial state
• If no such cycle can be found the language
  accepted by the automaton is empty

78
LTL Model Checking

• Generate the property automaton from the negated
  LTL property
• Generate the product of the property automaton
  and the transition system
• Show that there is no accepting cycle in the
  product automaton (check language emptiness)
• i.e., show that the intersection of the paths
  generated by the transition system and the paths
  accepted by the (negated) property automaton is
  empty
• If there is a cycle, it corresponds to a
  counterexample behavior that demonstrates the bug

79
LTL Model Checking Example
Property to be verified
Example transition system
G q
Negation of the property
1
p,q
? G q ? F ?q
Property automaton for the negated property
2
3
q
p
true
?q
q
Each state is labeled with the propositions that
hold in that state
Equivalently
q,p,q
?,p,q, p,q
?, p
1
2
80
Transition System to Buchi Automaton Translation
Example transition system
Corresponding Buchi automaton
i
1
p,q
p,q
p,q
1
q
2
3
q
p
q
2
3
Each state is labeled with the propositions that
hold in that state
p
81
Buchi automaton for the transition system (every
state is accepting)
Product automaton
1,1
p,q
1
p,q
p,q
2,1
p,q
q
2
q
3,1
q
3
4
p
q
p
3,2
4,2
p
Property Automaton
Accepting cycle (1,1), (2,1), (3,1), ((4,2),
(3,2))? Corresponds to a counter-example path for
the property G q
q,p,q
?,p,q, p,q
?, p
1
2
82
SPIN Holzmann 91, TSE 97

• Explicit state model checker
• Finite state
• Temporal logic LTL
• Input language PROMELA
• Asynchronous processes
• Shared variables
• Message passing through (bounded) communication
  channels
• Variables boolean, char, integer (bounded),
  arrays (fixed size)
• Structured data types

83
SPIN

• Verification in SPIN
• Uses the LTL model checking approach
• Constructs the product automaton on-the-fly
• It is possible to find an accepting cycle (i.e. a
  counter-example) without constructing the whole
  state space
• Uses a nested depth-first search algorithm to
  look for an accepting cycle
• Uses various heuristics to improve the efficiency
  of the nested depth first search
• partial order reduction
• state compression

84
Example Mutual Exclusion Protocol

• Two concurrently executing processes are trying
  to enter a critical section without violating
  mutual exclusion

Process 1 while (true) out a true
turn true wait await (b false or turn
false) cs a false Process
2 while (true) out b true turn
false wait await (a false or turn) cs
b false
85
Example Mutual Exclusion Protocol in Promela
define cs1 process1_at_cs define cs2
process2_at_cs define wait1 process1_at_wait define
wait2 process2_at_wait define true 1 define
false 0 bool a bool b bool turn proctype
process1() out a true turn true wait
(b false turn false) cs a
false goto out proctype process2() out
b true turn false wait (a false
turn true) cs b false goto
out init run process1() run process2()
86
Property automaton generation

• Input formula
• means G
• ltgt means F
• spin f option
• generates a Buchi automaton for the input LTL
  formula

spin -f "! (! (cs1 cs2)) never /
! (! (cs1 cs2)) / T0_init if
((cs1) (cs2)) -gt goto accept_all
(1) -gt goto T0_init fi accept_all
skip spin -f "!((wait1 -gt
ltgt(cs1))) never / !((wait1 -gt ltgt(cs1)))
/ T0_init if ( !((cs1))
(wait1) ) -gt goto accept_S4 (1) -gt
goto T0_init fi accept_S4 if
(! ((cs1))) -gt goto accept_S4
fi

• Concatanate the generated never claims to the end
  of the specification file

87
SPIN

• spin a mutex.spin generates a C program
  pan.c from the specification file
• This C program implements the on-the-fly
  nested-depth first search algorithm
• You compile pan.c and run it to the model
  checking
• Spin generates a counter-example trace if it
  finds out that a property is violated

88
mutex -a warning for p.o. reduction to be valid
the never claim must be stutter-invariant (never
claims generated from LTL formulae are
stutter-invariant) (Spin Version 4.2.6 -- 27
October 2005) Partial Order
Reduction Full statespace search for
never claim assertion
violations (if within scope of claim)
acceptance cycles (fairness disabled)
invalid end states - (disabled by never
claim) State-vector 28 byte, depth reached 33,
errors 0 22 states, stored 15
states, matched 37 transitions (
storedmatched) 0 atomic steps hash
conflicts 0 (resolved) 2.622 memory usage
(Mbyte) unreached in proctype process1
line 18, state 6, "-end-" (1 of 6
states) unreached in proctype process2
line 27, state 6, "-end-" (1 of 6
states) unreached in proctype init (0
of 3 states)
89
Problems/Heuristics for explicit state model
checking

• State space explosion Number of states can be
  exponential in the number of variables and
  concurrent components
• Heuristics used by Spin
• On the fly checking use a depth first search
  that computes the product of the property
  automaton and the transition relation while
  looking for a violation
• Bit-state hashing
• do not store the full state information
• might skip some unvisited states so it is not
  sound
• Partial order reduction
• only explore a representative subset of
  interleavings among the concurrent processes
• this can be done in a sound manner

90
Software Verification Using Explicit State Model
Checking with Java Path Finder
91
Java Path Finder

• Program checker for Java
• Properties to be verified
• Properties can be specified as assertions
• static checking of assertions
• It can also verify LTL properties
• Implements both depth-first and breadth-first
  search and looks for assertion violations
  statically
• Uses static analysis techniques to improve the
  efficiency of the search
• Requires a complete Java program
• It can only handle pure Java, it cannot handle
  native code

92
Java Path Finder, First Version

• First version
• A translator from Java to PROMELA
• Use SPIN for model checking
• Since SPIN cannot handle unbounded data
• Restrict the program to finite domains
• A fixed number of objects from each class
• Fixed bounds for array sizes
• Does not scale well if these fixed bounds are
  increased
• Java source code is required for translation

93
Java Path Finder, Current Version

• Current version of the JPF has its own virtual
  machine JPF-JVM
• Executes Java bytecode
• can handle pure Java but can not handle native
  code
• Has its own garbage collection
• Stores the visited states and stores current path
  
• Offers some methods to the user to optimize
  verification
• Traversal algorithm
• Traverses the state-graph of the program
• Tells JPF-JVM to move forward, backward in the
• state space, and evaluate the assertion
• The rest of the slides are on the current version
  of JPF
• W. Visser, K. Havelund, G. Brat, S. Park and F.
  Lerda. "Model Checking Programs." Automated
  Software Engineering Journal Volume 10, Number 2,
  April 2003.

94
Storing the States

• JPF implements a depth-first search on the state
  space of the given Java program
• To do depth first search we need to store the
  visited states
• There are also verification tools which use
  stateless search such as Verisoft
• The state of the program consists of
• information for each thread in the Java program
• a stack of frames, one for each method called
• the static variables in classes
• locks and fields for the classes
• the dynamic variables (fields) in objects
• locks and fields for the objects

95
Storing States Efficiently

• Since different states can have common parts each
  state is divided to a set of components which are
  stored separately
• locks, frames, fields
• Keep a pool for each component
• A table of field values, lock values, frame
  values
• Instead of storing the value of a component in a
  state store an index at which the component is
  stored in the table in the state
• The whole state becomes an integer vector
• JPF collapses states to integer vectors using
  this idea

96
State Space Explosion

• State space explosion if one of the major
  challenges in model checking
• The idea is to reduce the number of states that
  have to be visited during state space exploration
• Here are some approaches used to attack state
  space explosion
• Symmetry reduction
• search equivalent states only once
• Partial order reduction
• do not search thread interleavings that generate
  equivalent behavior
• Abstraction
• Abstract parts of the state to reduce the size of
  the state space

97
Symmetry Reduction

• Some states of the program may be equivalent
• Equivalent states should be searched only once
• Some states may differ only in their memory
  layout, the order objects are created, etc.
• these may not have any effect on the behavior of
  the program
• JPF makes sure that the order which the classes
  are loaded does not effect the state
• There is a canonical ordering of the classes in
  the memory

98
Symmetry Reduction

• A similar problem occurs for location of
  dynamically allocated objects in the heap
• If we store the memory location as the state,
  then we can miss equivalent states which have
  different memory layouts
• JPF tries to remedy this problem by storing some
  information about the new statements that create
  an object and the number of times they are
  executed

99
Partial Order Reduction

• Statements of concurrently executing threads can
  generate many different interleavings
• all these different interleavings are allowable
  behavior of the program
• A model checker has to check all possible
  interleavings that the behavior of the program is
  correct in all cases
• However different interleavings may generate
  equivalent behaviors
• In such cases it is sufficient to check just one
  interleaving without exhausting all the
  possibilities
• This is called partial order reduction

100
state space search generates 258 states with
symmetry reduction 105 states with partial order
reduction 68 states with symmetry reduction
partial order reduction 38 states
class S1 int x class FirstTask extends
Thread public void run() S1 s1 int x
1 s1 new S!() x 3 class Main
public static void main(String args)
FirstTask task1 new FirstTask() SecondTask
task2 new SecondTask() task1.statr()
task2.start()
class S2 int y class SecondTask extends
Thread public void run() S2 s2 int x
1 s2 new S2() x 3
101
Static Analysis

• JPF uses following static analysis techniques for
  reducing the state space
• slicing
• partial evaluation
• Given a slicing criterion slicing reduces the
  size of a program by removing the parts of the
  program that have no effect on the slicing
  criterion
• A slicing criterion could be a program point
• Program slices are computed using dependency
  analysis
• Partial evaluation propagates constant values and
  simplifies expressions

102
Abstraction vs. Restriction

• JPF also uses abstraction techniques such as
  predicate abstraction to reduce the state space
• Still, in order to check a program with JPF,
  typically, you need to restrict the domains of
  the variables, the sizes of the arrays, etc.
• Abstraction over approximates the program
  behavior
• causes spurious counter-examples
• Restriction under approximates the program
  behavior
• may result in missed errors
• If both under and over approximation techniques
  are used then the resulting verification
  technique is neither sound nor complete
• However, it is still useful as a debugging tool
  and it is helpful in finding bugs

103
JPF Java Modeling Primitives

• Atomicity (used to reduce the state space)
• beginAtomic(), endAtomic()
• Nondeterminism (used to model non-determinism
  caused by abstraction)
• int random(int) boolean randomBool() Object
  randomObject(String cname)
• Properties (used to specify properties to be
  verified)
• AssertTrue(boolean cond)

104
Annotated Java Code for a Reader-Writer Lock

• import gov.nasa.arc.ase.jpf.jvm.Verify
• class ReaderWriter
• private int nr
• private boolean busy
• private Object Condr_enter
• private Object Condw_enter
• public ReaderWriter()
• Verify.beginAtomic()
• nr 0 busyfalse
• Condr_enter new Object()
• Condw_enter new Object()
• Verify.endAtomic()
• public boolean read_exit()
• boolean resultfalse
• synchronized(this)
• nr (nr - 1)
• resulttrue

• private boolean Guarded_r_enter()
• boolean resultfalse
• synchronized(this)
• if(!busy)nr (nr 1)resulttrue
• return result
• public void read_enter()
• synchronized(Condr_enter)
• while (! Guarded_r_enter())
• tryCondr_enter.wait()
• catch(InterruptedException e)
• Verify.assertTrue(!busy nr0 )
• private boolean Guarded_w_enter()
• public void write_enter()
• public boolean write_exit()

105
JPF Output

• gtjava gov.nasa.arc.ase.jpf.jvm.Main rwmain
• JPF 2.1 - (C) 1999,2002 RIACS/NASA Ames Research
  Center
• JVM 2.1 - (C) 1999,2002 RIACS/NASA Ames Research
  Center
• Loading class gov.nasa.arc.ase.jpf.jvm.reflection.
  JavaLangObjectReflection
• Loading class gov.nasa.arc.ase.jpf.jvm.reflection.
  JavaLangThreadReflection
• No Errors Found
• -----------------------------------
• States visited 36,999
• Transitions executed 68,759
• Instructions executed 213,462
• Maximum stack depth 9,010
• Intermediate steps 2,774
• Memory used 22.1MB
• Memory used after gc 14.49MB

106
Example Error Trace

• 1 error found Deadlock
• Path to error
• Steps to error 2521
• Step 0 Thread 0
• Step 1 Thread 0
• rwmain.java4 ReaderWriter monitornew
  ReaderWriter()
• Step 2 Thread 0
• ReaderWriter.java10 public
  ReaderWriter( )
• Step 2519 Thread 2
• ReaderWriter.java71 while (!
  Guarded_w_enter())
• Step 2520 Thread 2
• ReaderWriter.java73
  Condw_enter.wait()

107
Bounded Model Checking
108
Bounded Model Checking

• Represent sets of states and the transition
  relation as Boolean logic formulas
• Instead of computing the fixpoints, unroll the
  transition relation up to certain fixed bound and
  search for violations of the property within that
  bound
• Transform this search to a Boolean satisfiability
  problem and solve it using a SAT solver

109
What Can We Guarantee?

• Note that in bounded model checking we are
  checking only for bounded paths (paths which have
  at most k1 distinct states)
• So if the property is violated by only paths with
  more than k1 distinct states, we would not find
  a counter-example using bounded model checking
• Hence if we do not find a counter-example using
  bounded model checking we are not sure that the
  property holds
• However, if we find a counter-example, then we
  are sure that the property is violated since the
  generated counter-example is never spurious
  (i.e., it is always a concrete counter-example)

110
Bounded Model Checking Proving Correctness

• One can also show that given an LTL property f,
  if E f holds for a finite state transition
  system, then E f also holds for that transition
  system using bounded semantics for some bound k
• So if we keep increasing the bound, then we are
  guaranteed to find a path that satisfies the
  formula
• And, if we do not find a path that satisfies the
  formula, then we decide that the formula is not
  satisfied by the transition system
• Is there a problem here?

111
Proving Correctness

• We can modify the bounded model checking
  algorithm as follows
• Start from an initial bound.
• If no counter-examples are found using the
  current bound, increment the bound and try again.
• The problem is We do not know when to stop

112
Proving Correctness

• If we can find a way to figure out when we should
  stop then we would be able to provide guarantee
  of correctness.
• There is a way to define a diameter of a
  transition system so that a property holds for
  the transition system if and only if it is not
  violated on a path bounded by the diameter.
• So if we do bounded model checking using the
  diameter of the system as our bound, then we can
  guarantee correctness if no counter-example is
  found.

113
Bounded Model Checking

• What are the differences between bounded model
  checking and BDD-based symbolic model checking?
• In bounded model checking we are using a SAT
  solver instead of a BDD library
• In symbolic model checking we do not unroll the
  transition relation as in bounded model checking
• In bounded model checking we do not execute the
  iterative fixpoint computations as in symbolic
  model checking
• In symbolic model checking for finite state
  systems both verification and falsification
  results are guaranteed
• In bounded model checking we can only guarantee
  the falsification results, in order to guarantee
  the verification results we need to know the
  diameter of the system

114
Bounded Model Checking

• A bounded model checker needs an efficient SAT
  solver
• zChaff SAT solver is one of the most commonly
  used ones
• Most SAT solvers require their input to be in
  Conjunctive Normal Form (CNF)
• So the final formula has to be converted to CNF
• Similar to BDD-based symbolic model checking,
  bounded model checking was also first used for
  hardware verification
• More recently, it has been applied to
  verification of software

115
Bounded Model Checking for Software

• CBMC is a bounded model checker for ANSI-C
  programs
• Handles function calls using inlining
• Unwinds the loops a fixed number of times
• Allows user input to be modeled using
  non-determinism
• So that a program can be checked for a set of
  inputs rather than a single input
• Allows specification of assertions which are
  checked using the bounded model checking

116
Loops

• Unwind the loop n times by duplicating the loop
  body n times
• Each copy is guarded using an if statement that
  checks the loop condition
• At the end of the n repetitions an unwinding
  assertion is added which is the negation of the
  loop condition
• Hence if the loop iterates more than n times in
  some execution, the unwinding assertion will be
  violated and we know that we need to increase the
  bound in order to guarantee correctness
• A similar strategy is used for recursive function
  calls
• The recursion is unwound up to a certain bound
  and then an assertion is generated stating that
  the recursion does not go any deeper

117
A Simple Loop Example
Original code
Unwinding the loop 3 times
x0 while (x lt 2) yyx x
x0 if (x lt 2) yyx x if (x lt 2)
yyx x if (x lt 2) yyx
x assert (! (x lt 2))
Unwinding assertion
118
From Code to SAT

• After eliminating loops and recursion, CBMC
  converts the input program to the static single
  assignment (SSA) form
• In SSA each variable appears at the left hand
  side of an assignment only once
• This is a standard program transformation that is
  performed by creating new variables
• In the resulting program each variable is
  assigned a value only once and all the branches
  are forward branches (there is no backward edge
  in the control flow graph)
• CBMC generates a Boolean logic formula from the
  program using bit vectors to represent variables

119
Another Simple Example
Original code
Convert to static single assignment
x1x0y0 if (x1!1) x22 else
x3x11 x4(x1!1)?x2x3 assert(x4lt3)
xxy if (x!1) x2 else
x assert(xlt3)
Generate constraints
C ? x1x0y0 ? x22 ? x3x11 ?(x1!1 ? x4x2 ?
x11 ? x4x3) P ? x4 lt 3
Check if C ? ? P is satisfiable, if it is then
the assertion is violated C ? ? P is converted
to boolean logic using a bit vector
representation for the integer variables
y0,x0,x1,x2,x3,x4
120
Bounded Verification Approaches

• What we have discussed above is bounded
  verification by bounding the number of steps of
  the execution.
• For this approach to work the variable domains
  also need to be bounded, otherwise we cannot
  convert the problems to boolean SAT
• Bounding the execution steps and bounding the
  data domain are two orthogonal approaches.
• When pe