Lecture 10: Verifying first order properties in PVS

1

• Automatic Generation of Path Conditions
• for Concurrent Timed Systems

Doron Peled, University of Warwick Joint work
with Saddek Bensalem, Stavros Tripakis, Hongyang
Qu
2
Unit testing Selection of test cases (for
white-box testing)

• The main problem is to select a good coverage
• criterion. Some standard options are
• Cover all paths of the program.
• Execute every statement at least once.
• Each decision (diamond node on flow chart) has a
  true or false value at least once.
• Each condition predicate is taking each truth
  value at least once.
• Check all possible combinations of conditions in
  each decision.

3
Cover all the paths of the program (skip)
Infeasible. Consider the flow diagram on the
left. It corresponds to a loop. The loop body has
5 paths. If the loops executes 20 times there are
520 different paths! May also be unbounded
number of iterations depend on data. So also
forget about covering all executions!
4
How to cover the executions?

• if (Agt1)(B0) then XX/A
• if (A2)(Xgt1) then XX1
• Choose values for A,B,X at the beginning that
  would force the right path/conditions/predicates.
• Value of X may change, depending on A,B.
• What do we want to cover? Paths? Statements?
  Conditions?

5
Statement coverageExecute every statement at
least once

• By choosing
• A2,B0,X3
• each statement will be chosen.
• The case where the tests fail is not checked!

• if (Agt1)(B0) then XX/A
• if (A2)(Xgt1) then XX1

Now x1.5
6
Decision coverageEach decision (diamond node in
flow graph) tested with true and false outcome at
least once.

• Can be achieved using
• A3,B0,X3
• A2,B1,X1
• Problem Does not test individual predicates.
  E.g., when Xgt1 is erroneous in second decision.

• if (Agt1)(B0) then XX/A
• if (A2)(Xgt1) then XX1

7
Condition coverage (skip)Each predicate has a
true and false value at least once.

• if (Agt1)(B0) then XX/A
• if (A2)(Xgt1) then XX1

• For example
• A1,B0,X3
• A2,B1,X0
• lets each condition be true and false once.
• Problemcovers only the path where the first test
  fails and the second succeeds.

8
PreliminaryRelativizing assertions

• ?(B) x1 y1 x2 y2 /\ y2 gt 0
• Relativize ??B) w.r.t. the assignment becomes
  ??B) Y\g(X,Y)
• (I.e., ?( B) expressed w.r.t. variables at A.)
• ? ?(B)A ?x10 x2 x1 /\ x1gt0
• Think about two sets of variables,beforex, y,
  z, afterx,y,z.
• Rewrite ?(B) using after, and the assignment as a
  relation between the set of variables. Then
  eliminate after.
• Here x1y1 x2 y2 /\ y2gt0 /\x1x1 /\
  x2x2 /\ y10 /\ y2x1now eliminate x1, x2,
  y1, y2.

A
Yg(X,Y)
(y1,y2)(0,x1)
B
A
(y1,y2)(0,x1)
B
9
Verification conditions tests
B
T
F

• ??C) is transformed to ??B) t(X,Y) /\ ??C)
• ??D) is transformed to ??B)?t(X,Y) /\ ??D)
• ??B) ??D) /\ ?y2?x2

t(X,Y)
C
D
B
F
T
y2gtx2
D
C
10
How to find values for coverage?

• Put true at end of path.
• Propagate path backwards.
• On assignment, relativize expression.
• On yes edge of decision node, add decision as
  conjunction.
• On no edge, add negation of decision as
  conjunction.
• Can be more specific when calculating condition
  with multiple condition coverage.

Agt1/\B0
no
yes
XX/A
A2\/Xgt1
true
no
yes
XX1
true
11
How to find values for coverage?
(A?2 /\ X/Agt1) /\ (Agt1 B0)
Agt1/\B0
A?2 /\X/Agt1
no
yes
Need to find a satisfying assignment A3, X6,
B0 Can also calculate path condition forwards.
XX/A
A ? 2/\Xgt1
A2\/Xgt1
true
no
yes
XX1
true
12
How to cover a flow chart?(skip)

• Cover all nodes, e.g., using search strategies
  DFS, BFS.
• Cover all paths (usually impractical).
• Cover each adjacent sequence of N nodes.
• Probabilistic testing. Using random number
  generator simulation. Based on typical use.
• Chinese Postman minimize edge traversalFind
  minimal number of times time to travel each edge
  using linear programming or dataflow
  algorithms.Duplicate edges and find an Euler
  path.

13
Test cases based on data-flow analysis (skip)

• Partition the program into pieces of code with a
  single entry/exit point.
• For each piece find which variables are
  set/used/tested.
• Various covering criteria
• from each set to each use/test
• From each set to some use/test.

X3
tgty
xgty
zzx
14
Some real life story

• An expert programmer inspects the code of NASA
  MER.
• He observe using his experience and intuition
  that some execution path is suspicious.
• He decides how to force this path to execute,
  e.g., by figuring some inputs and initial values.
• He executes the path, showing his supervisor the
  presence of an error.
• We want to build some tools to help him with this
  process.
• Well use LTL to help with formalizing the
  intuition on where the error is.

15
Learning from another technique Model Checking

• Automaton description of a system B.
• LTL formula ?. Translate ?? into an automaton P.
• Check whether L(B) ? L(P)?.
• If so, S satisfies ?. Otherwise, the intersection
  includes a counterexample.
• Repeat for different properties.

?
?
16
Unit Testing
Model Checking
17
New Test case generation based on LTL
specification
18
Path conditions

• Path in flow chart ? multiple executions
  following path.
• First order formula.
• All executions of a path must start with initial
  values satisfying the path condition.
• In deterministic code, there can be only one
  execution starting with particular values, hence
  all executions starting with initial values
  satisfying the path condition will follow that
  path.
• In nondeterministic code, each such initial value
  has an execution following a path. May need to
  insert synchronizing code.
• Generalizations include inputs,
  being more specific about decisions
  made in path.

19
Goals

• Verification of software.
• Compositional verification. Use only a unit of
  code instead of the whole code.
• Parameterized verification. Verifies a procedure
  with any value of parameters in one shot
• Generating test cases via path conditions A
  truth assignment satisfying the path condition.
  Helps derive the demonstration of errors.
• Generating appropriate values to missing
  parameters.

20
Spec at l2U (at l2/\ x?y /\ ?(at l2/\(at
l2U at l2 /\ x?2?y )))
Observationeach node hasconjunctions of
predicates onprogram variables and
programcounters

• Automatic translation of LTL formula into an
  automaton GPVW95
• LTL is interpreted over finite sequences.
• Can use other (linear) specification.
• Property specifies the path we want to find
  (SPIN never claim),not the property that must
  hold for all paths (for this, take the negation).

21
Divide and Conquer

• Intersect property automaton with theflow chart,
  regardless of the statements and program
  variables expressions.
• Add assertions from the property automaton to
  further restrict the path condition.
• Calculate path conditions for sequences found in
  the intersection.
• Calculate path conditions on-the-fly. Backtrack
  when condition is false.Thus, advantage to
  forward calculation of path conditions
  (incrementally).

22
Spec (only program counters here)at l2U (at
l2/\? at l2/\(at l2U at l2))
at l2
l2xxz
at l2
X

at l2
at l2
l3xltt
at l2
at l2
l2xxz
Either all executions of a path satisfy the
formula or none.
at l2
Sifts away path not satisfying formula. Then
calculate path condition.
23
Spec at l2U (at l2/\ x?y /\ ?(at l2/\(at
l2U at l2 /\ x?2?y )))
x?y
l2xxz
at l2
X

at l2/\ x?y
l3xltt
x?2?y
at l2
l2xxz
at l2/\ x?2?y
Only some executions of path may satisfy formula
Modify calculation of path condition to
incorporate property
24
Calculating the intersection of the property
automaton and flow graph (abstract variables
away).
a
a
a
?ltgt?a
s1
s2
q1
s3
q2
a
?a
Acceptance isdetermined bypropertyautomaton.
s1,q1
s2,q1
s1,q2
s3,q2
25
How to generate test cases

• Take the intersection of an LTL automaton (for a
  never claim) with the flow graph. Some paths
  would be eliminated for not satisfying the
  assertions on the program counters.
• Seeing same flow chart node does not mean a loop
  program variables may value. Use iterative
  deepening.
• For each initial path calculate the path
  condition. Backtrack if condition simplifies to
  false.
• Report path condition based on flow graph
  pathLTL assertions.
• Always simplify conditions!

26
How the LTL formula directs the search
Consider (x4)U (x5/\o)
x4
xlt5
true
false
x5
xx1
y7
27
How the LTL formula directs the search
Consider x4U (x5/\o)
x4
xlt5
true
false
x5
xx1
y7
28
How the LTL formula directs the search
Consider x4U (x5/\o)
x4
X4
xlt5
true
false
x5
xx1
y7
29
How the LTL formula directs the search
Consider x4U (x5/\o)
x4
X4
xlt5
true
false
x5
X4
xx1
y7
30
How the LTL formula directs the search
Consider x4U (x5/\o)
x4
X4
xlt5
true
false
x5
X4
This is in acontradiction
xx1
y7
X4
xlt5
true
31
How the LTL formula directs the search
Consider x4U (x5/\o)
x4
X5
xlt5
true
false
x5
X4
xx1
y7
32
How the LTL formula directs the search
Consider x4U (x5/\o)
x4
X5
xlt5
true
false
x5
X4
xx1
y7
33
Example GCD
l0
l1xa
l2yb
l3zx rem y
l4xy
l5yz
l6z0?
yes
no
l7
34
Example GCD
l0
l1xa
l2yb
Oopswith an error (l4 and l5 were switched).
l3zx rem y
l4yz
l5xy
l6z0?
yes
no
l7
35
Why use Temporal specification

• Temporal specification for sequential software?
• Deadlock? Liveness? No!
• Captures the testers intuition about the
  location of an errorI think a problem may
  occur when the program runs through the main
  while loop twice, then the if condition holds,
  while tgt17.

36
Example GCD
l0
l1xa
l2yb
agt0/\bgt0/\at l0 /\?at l7
l3zx rem y
at l0/\agt0/\bgt0
l4yz
l5xy
l6z0?
yes
no
at l7
l7
37
Example GCD
l0
l1xa
l2yb
agt0/\bgt0/\at l0/\?at l7
l3zx rem y
Path 1 l0l1l2l3l4l5l6l7agt0/\bgt0/\a rem
b0 Path 2 l0l1l2l3l4l5l6l3l4l5l6l7
agt0/\bgt0/\a rem b?0
l4yz
l5xy
l6z0?
yes
no
l7
38
Potential explosion
Bad point potential explosion Good point may be
chopped on-the-fly
39
Now we add time

• Detailed model, for each transition we have 4
  parameters l, u, L, U
• l Needs to be enabled at least that much.
• u Cannot be enabled without taken longer
  than that.
• L Least time for transformation to occur
  (after been chosen).
• U Transformation cannot take more than that.

40
Translation to timed automata
s1at l
?c1?c2x20
?c1?c2x20
c1??c2x10
c1??c2x10
c1?c2 x1,x20
?c1??c2
?c1??c2
?c1??c2
s3,at lx2ltu2x1ltu1
c1??c2
?c1?c2
s4,at lx2ltu2
s2,at lx1ltu1
c1?c2x20
c1?c2x10
Timing out the enablednessZero counters,Cannot
wait enabled too much.
41
Translation to timed automata
s1at l
?c1?c2x20
?c1?c2x20
c1??c2x10
c1??c2x10
c1?c2 x1,x20
?c1??c2
?c1??c2
?c1??c2
s3,at lx2ltu2x1ltu1
c1??c2
?c1?c2
s4,at lx2ltu2
s2,at lx1ltu1
c1?c2x20
c1?c2x10
ac
x1?l1x10
bc
x1?l1x10
x2?l2x20
x2?l2x20
bc
s5x1ltU1
ac
s6x2ltU2
Can fire only if waited enough,Zero counters
again.
42
Translation to timed automata
s1at l
?c1?c2x20
?c1?c2x20
c1??c2x10
c1??c2x10
c1?c2 x1,x20
?c1??c2
?c1??c2
?c1??c2
s3,at lx2ltu2x1ltu1
c1??c2
?c1?c2
s4,at lx2ltu2
s2,at lx1ltu1
c1?c2x20
c1?c2x10
ac
x1?l1x10
bc
x1?l1x10
x2?l2x20
x2?l2x20
bc
s5x1ltU1
ac
s6x2ltU2
x1?L1
x2?L2
af
bf
s8
s7
43
Should we really look at paths?

• Its easy to select an interleaved sequence.
• But due to time limitations, it may execute in a
  different order.
• Just the order on events from the same process
  and using same variables is to be considered.

a
a
b
b
c
d
c
Samevariable
d
Sameprocess
44
Generate an automaton for all consistent
interleavings
a
b
a
b
c
d
a
b
c
Intersect this automaton with automaton for
system.Calculate partial order condition
start from leaves.When there is a choice,
usedisjunct.
c
b
d
d
c
45
Generate an automaton for all consistent
interleavings
a
b
a
b
c
c
b
d
d
c
46
Generate an automaton for all consistent
interleavings
a
b
a
b
c
c
b
d
d
c
47
Generate an automaton for all consistent
interleavings
a
b
a
b
c
c
b
d
d
c
48
An example a simple network protocol
49
The flow charts
50
Path no timeout
51
Precondition

• The simplified precondition l gt 110

52
The diagrams
53
The PET tool

• Basic mode interactive choice of a path,
  calculating of path conditions.
• Model checking mode.
• Iterative model checking mode apply model
  checking recursively to find successive segments,
  control backtracking.
• Unit checking mode.
• Calculating path condition simplify, simplify,
  simplify.Use SML and HOL for rewriting and
  deciding on Pressburger arithmetic. Plan using
  other tools!
• Problem US patent 6,408,430 belongs to Lucent!

54
(No Transcript)
55
(No Transcript)
56
(No Transcript)
57
(No Transcript)
58
(No Transcript)
59
(No Transcript)
60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
(No Transcript)
65
(No Transcript)
66
(No Transcript)
67
(No Transcript)
68
Drivers and Stubs(skip)
l0
l1xa

• Driver represents the program or procedure that
  called our checked unit.
• Stub represents a procedure called by our
  checked unit.
• In our approach replace both of them with a
  formula representing the effect the missing code
  has on the program variables.
• Integrate the driver and stub specification into
  the calculation of the path condition.

l2yb
l3zx rem y /\xx/\yx
l4yz
l5xy
l6z0?
yes
no
l7
69
Some references

• Translating LTL into automataGerth, Peled,
  Vardi, Wolper, Simple on-the fly automatic
  verification of temporal logic, PSTV 1995.
• The PET toolGunter, Peled, Path Exploration
  Tool, Tacas 1999, LNCS 1579
• Unit CheckingGunter, Peled, Unit Checking
  symbolic model checking for unit of code, LNCS
  2772 (Z.M. birthday volume)
• Forcing an execution under nondeterminismQu,
  Peled, Enforcing Concurrent Temporal Behavior, RV
  2004

70
Conclusions

• Model checking and testing have a lot in common.
  Can use ideas from model checking for generating
  test cases.
• Unit Testing Model checking of infinite state
  spaces.But semidecidable
• Dont know when to stop search (undecideable),
• Dont know when condition equivalent false
  (undecideable).
• Tools, visual user interface.
• Generalization to real time systems.
• Wanted collaborations integrate stronger
  simplifiers, more complicated data structures,
  implementing abstractions.