On Bridging Simulation and Formal Verification

1
On Bridging Simulation and Formal Verification
Eugene Goldberg Cadence Research Labs (USA)
VMCAI-2008, San Francisco, USA
2
Summary

• Introduction
• motivation
• main idea
• Stable sets of points (SSPs) and proofs
• checking if a set of points contains an SSP
• simulation based proof system
• extraction of sufficient test sets from proofs
• Generation of practical tests
• test generation for design change/manufacturing
  faults
• theoretical and experimental justification of our
  approach
• Conclusion

3
Motivation
Given a design, formal verification is to check
if property ? holds at all points of the search
space.
Simulation is to informally check if property ?
holds by testing it for a (relatively small)
set of points.
Simulation is still the main workhorse of
verification. It works surprisingly well taking
into account that only negligible part of the
search space is sampled!!
In this paper, we study this phenomenon by the
example of the satisfiability problem. As an
application, we consider testing combinational
circuits.
4
Simulation in Terms of SAT
Let F be a CNF formula (i.e. conjunction of
disjunctions also called clauses).
Satisfiability problem (SAT) Given F, find an
assignment p such that F(p)1 or prove that
F(all_space)0.
Simulation Given a CNF formula F, find a set of
points T s.t. if F(T) 0, then F is (most
likely) unsatisfiable.
In other words, find T such that F(T)0 implies
F(all_space) 0
5
Interpretation of Points as Tests
Let N be a circuit. Let X,Y describe the input
and internal variables of N. Let z describe the
output variable of N.
Let CNF formula FN(X,Y,z) specify N. Checking
satisfiability of N reduces to checking the
satisfiability of FN ? z.
z
N
A point p is a complete assignment to the
variables X ? Y ? z. The corresponding test t
is the projection of p to X.
Y
.
So p consists of a test t and a computation
vector (y,z). Since p falsifies clauses of FN it
is computation with a fault.
X
6
Main Idea
We formulate simulation as a proof system.
Instead of proving one big theorem (F implies
an empty clause), we deduce a sequence of simpler
lemmas (F implies a clause L). Each derived
clause is added to F.
In contrast to regular simulation, we use
points not only for sampling search space but for
derivation. In this derivation we employ the
machinery of Stable Sets of Points.
7
Summary

• Introduction
• motivation
• main idea
• Stable sets of points (SSPs) and proofs
• checking if a set of points contains an SSP
• simulation based proof system
• extraction of sufficient test sets from proofs
• Generation of practical tests
• test generation for design change/manufacturing
  faults
• theoretical and experimental justification of our
  approach
• Conclusion

8
A Stable Set of Points (SSP)
A complete assignment ? a point.
Example of 1-neighborhood of point p w.r.t a
clause C (written Nbhd(p,C)) such that C(p)0. C
x ? z, p(x0,y0,z0). Nbhd(p,C) p1,p2,
where p1(x1,y0,z0) and p2(x0,y0,z1).
Let F be a CNF formula and T be a set of
points. T is called an SSP if ? p ? T, ? clause
C of F such that Nbhd(p,C) ? T.
A CNF formula F is unsatisfiable iff there is a
set of points that is an SSP.
9
Testing if a Set of Points Contains SSP
Let F be a CNF formula and T be a set of
points. To check if T is an SSP For every
point p of T, check if there is a clause C of F
s.t. Nbhd(p,C) ? T.
This procedure is linear both in T and F.
Even if T is not an SSP, a subset T of T may be
an SSP. To check if T contains an SSP For every
point p of T, check if there is a clause C of F
s.t. Nbhd(p,C) ? T. If C does not exist, remove
p from T and restart this procedure.
This procedure is quadratic in T and linear in F.
10
Simulation as a Proof System
We formulate simulation as a proof system with
two derivation rules. Rule 1 Generate a point
pi and derive the value of F(pi). Rule 2 Let
Tp1,,pk be the current set of points. Derive
a clause C such that F implies C and add it to F.
Derivation of C is valid only if T contains an
SSP of the CNF formula F? C (and so F ? C
holds).
11
A Sufficient Test Set
Given a CNF formula F, a set of points T is
called sufficient if there is a set of lemma
clauses L1,,Lk (Lk?) such that all k
derivations F ? L1, F ? L1 ? L2 , . F ? L1
? ? Lk-1 ? Lk are valid (i.e. they can be
proved by simulation using T)
In other words, T is a sufficient test set if the
set of valid derivations is sufficient to prove
that F is unsatisfiable by applying only the
second rule (no new points are needed).
12
Extracting a Sufficient Test Set from a
Resolution Proof
Let LL1,..,Lk be the set of resolvents of a
resolution proof R that F is unsatisfiable. We
assume that resolvents Li are numbered in their
derivation order (so Lk ?).
Then there is always a sufficient test set whose
size is ? 2?k (proving that F?L1? ? Li-1 ? Li
needs only two points of T).
Sufficient test sets can be very small even for
very large formulas!!!!
13
Proving Resolvent by Building a Stable Set of
Two Points
Let C be the resolvent of C? and C?. One can
prove that C?? C? ? C by building an SSP of
only two points.
Example Let C? x ? y ? z, C? x ? w. The
resolvent C is y ? z ? w. To prove C? ? C? ? C
it suffices to show the CNF formula C? ? C? ? C
is unsatisfiable. (Here C y ? z ? w).
After making assignments y0,z0,w0, the formula
C? ? C? ? C turns into x ? x. Any two points
p? and p? with y0,z0,w0, that are different
only in the value of variable x is an SSP for
C? ? C? ? C .
14
Summary

• Introduction
• motivation
• main idea
• Stable sets of points (SSPs) and proofs
• checking if a set of points contains an SSP
• simulation based proof system
• extraction of sufficient test sets from proofs
• Generation of practical tests
• test generation for design change/manufacturing
  faults
• theoretical and experimental justification of our
  approach
• Conclusion

15
Applying Our Theory to (design
change/manufacturing testing)
We cannot apply our theory directly to checking
if F is unsatisfiable. (To prove that a set of
points is sufficient we need a set of lemma
clauses i.e. another proof)
However, there are numerous indirect ways to
apply our theory. One application is to detect
incorrect design changes/ manufacturing faults.
Let F be a CNF formula. Let R be a proof that F
is unsatisfiable. Find a set of points such that
most likely detect satisfiable variations of F.
(These variations describe tech. faults/design
changes.) Such a test set can be extracted from
a sufficient set of points specified by R.
16
Tight Sufficient Tests
Let T be a sufficient test set for F.
Informally, T is tight if every point p ? T
falsifies as few clauses as possible.
Given a resolution proof L1,..,Lk, a sufficient
test set T is built as T1? ?Tk. Here Ti is a
two point SSP that proves that C? ? C? ? Li
where C? ? C? are the parents of resolvent Li.
When building Ti one can arbitrarily assign
variables that are not in C? or C?.
To build a tight sufficient test, free variables
are assigned so as to minimize the number of
falsified clauses.
17
Big Picture
Let F be an unsatisfiable formula. To detect
satisfiable variations (faults in F) we
generate a tight sufficient test set T. Such a
set can be extracted, for example, from a
resolution proof that F is unsatisfiable.
The sufficiency of T guarantees its
completeness. The tightness of T increases
the probability of detec-ting small satisfiable
variations.
Suppose, for example, that a point p of T
falsifies only one clause C of F. Then if F is a
variation of F consisting of disappearance of C
(along with some other changes) it is likely that
F(p) 1. Hence p detects that F is satisfiable.
18
Theoretical Justification of Our Approach
Let N1 and N2 be identical copies of circuit N.
Let Rnat be a natural resolution proof that
their miter is unsatisfiable.
Let Tp1,..,pm be a tight sufficient test set
specified by Rnat. We show that the set inp(T)
inp(p1),,inp(pm) detects all (testable)
stuck-at faults of N.
So, in a sense, our theory predicts the high
quality of the stuck-at fault model for circuit
testing.
19
Experimental Justification
We experimentally compared quality of random
test sets and test sets extracted from
resolution proofs that two copies of a circuit
were identical.
Resolution proofs were generated by a
state-of-the-art SAT-solver. We used MCNC
benchmark circuits.
We applied these two kinds of test sets (of the
same size) to detecting literal appearance
faults. Such faults are more subtle than
stuck-at faults and so are harder to detect.
Experiments show that tests extracted from
resolution proofs significantly outperform random
tests.
20
Conclusion
We use SAT to show that there is a close
relation between high quality test sets and
formal proofs.
We show how high quality tests can be extracted
from proofs of unsatisfiability represented as a
sequence of lemma clauses.

• Our approach answers (at least in principle) two
  important questions.
• When does one stop simulation? (When the set of
  generated points encrypts a formal proof).
• What is an ideal metric in functional
  verification? (A formal proof.)