Formal Software Verification

1
Formal Software Verification
DPS, Pace University
2
References

• Use of the slides of Dr. Clark Barrett (with
  permission)
• ITR NSF Little Engines of Proof with Stanford
  Research Institute and Clarkson University, 2003
• Automated Deduction with Constraints and
  Simplification in Equational Theories, Christelle
  Scharff, PhD, 1999
• ICS (I can Solve or Integrated Canonizer and
  Solver), Harald Ruess, 2002
• Abstract Congruence Closure, Leo Bachmair, Ashish
  Tiwari, and Laurent Vigneron, 2003
• Unsound Theorem Proving, Christopher Lynch, 2004
• Direct Combination of Completion and Congruence
  Closure, Christelle Scharff and Leo Bachmair,
  2002 and 2004
• On prototyping Deduction with Constraints and
  Simplification in a Rewriting Language,
  Christelle Scharff, 2004

3
Therac-25
4
Ariane 5 Rocket

• On June 4, 1996, an unmanned Ariane 5 rocket
  launched by the European Space Agency exploded
  just 40 seconds after its lift-off.
• Value of rocket and cargo 500 million

5
Blackout

• In August, 2003, the largest blackout in our
  countrys history occurred.
• Estimated cost to New York City alone 1.1
  billion.

6
What do these events have in common?

• Caused by Software Bugs!
• Each of the overdoses from the Therac-25 was the
  result of a bug in the controlling software.
• The Ariane 5 explosion was the result of an
  unsafe floating point to integer conversion in
  the rockets software system.
• A software bug caused an alarm system failure at
  FirstEnergy in Akron, Ohio. An early response to
  those alarms would likely have prevented the
  blackout.

7
More horror stories

• Software Reliability Principles and Practice, p
  25, by G. J. Myers
• Appolo 8 spacecraft erased part of the computer's
  memory.
• Eighteen errors were detected during the 10-day
  flight of Apollo 14.
• An error in a single FORTRAN statement resulted
  in the loss of the first American probe to Venus.

8
Why must we suffer so?

• Engineers are supposed to be good at building
  things.
• When we build a building, we dont expect it to
  crumble and have to be rebuilt twice a week.
• Why dont we have software engineers who build
  better software?
• Its not all Microsofts fault.
• Software may be the most complex thing ever
  created.
• Getting software right is a very, very difficult
  task.
• Existing engineering techniques are inadequate.

9
Outline

• What is Formal Verification?
• Model Checking
• Theorem Proving
• Systems and Tools
• Demonstrations (Eclipse, ICS, GACC)
• Conclusions and Future Work

10
What is Formal Verification?

• Formal software verification has been the
  Holy Grail of computer science for many decades
  Bill Gates
• Formal Verification (Proving) versus Validation
  (Testing)
• Create a mathematical model of the system
• An inaccurate model can introduce or mask bugs.
• Fortunately, this can often be done
  automatically.
• Specify formally what the properties of the
  system should be
• Prove that the model has the desired properties
• Much better than any testing method
• Covers all possible cases
• This is the hard part
• There are a variety of tools and techniques

11
Proof techniques

• Model Checking
• Typically relies on low-level Boolean logic
• Proof is automatic
• Does not always scale to large systems
• Theorem Proving
• Typically uses more expressive logic (higher
  order logic)
• Proof is automatic in first order logic
• Proof is manually directed (in higher order
  logic)
• Unlimited scalability
• Advanced techniques combine elements of both

12
Outline

• What is Formal Verification?
• Model Checking
• Theorem Proving
• Systems and Tools
• Demonstrations (ICS, GACC, Eclipse)
• Conclusions and Future Work

13
Formal Models

• Typically, a formal model is a graph in which
  each vertex represents a state of the program,
  and each edge represents a transition from one
  state to another.

14
Reachable States and Checking Properties

• We can check a property by verifying that it is
  true in every reachable state. If the property
  is false, then there is a bug.

Initial State
Final State
15
Outline

• What is Formal Verification?
• Model Checking
• Theorem Proving
• Systems and Tools
• Demonstrations (Eclipse, ICS, GACC)
• Conclusions and Future Work

16
Theorem Proving

• Theorem proving relies on human ingenuity and
  symbolic manipulations to prove that a program
  satisfies some properties.
• Typically, proving a single property about a
  program will require proving many other
  properties as well.
• Theorem Prover Software based on Inference
  Rules Strategy
• Inference Rules How to deduce new data? How to
  remove data?
• Strategy How to apply the inference rules?

17
Theorem Proving

• What are the properties of theorem provers?
• Soundness (Mandatory Property)
• Do not prove that True False
• Prove only true formulae
• Completeness (Optional Property)
• If a formula is true, it can be proved.
• Termination (Undecidable Property)
• Does the application of the rules with the
  strategy terminate?

18
Theorem Proving

• One approach is to annotate the program with
  theorems to be proved (e.g. pre-conditions,
  post-conditions, invariants, assertions), and
  then prove that each theorem really does hold.
• Hoare Logic
• Consider a simple program
• Suppose we wish to prove that, if x y before
  the execution of this program, then at the end of
  its execution, x y. We can annotate the
  beginning and the end of the program with these
  properties.

19
Theorem Proving

• What is the condition that will guarantee that x
  y after executing y y 1 ?
• To find out, propagate the assertions upward
  through the program. We imagine trying to prove x
  y using primed variables to represent the
  values after the execution of y y 1, and
  unprimed variables for the values before it
• (?) ? y y 1 ? x x ? x y
• (?) ? x y 1

20
Theorem Proving

• What is the condition that will guarantee that x
  y 1 after executing x x 1 ?
• To find out, we imagine trying to prove
• x y 1 using primed variables to represent
  the values after the execution of x x 1, and
  unprimed variables for the values before it
• (?) ? x x 1 ? y y ? x y 1
• (?) ? x 1 y 1

21
Theorem Proving

• We prove that the program is correct by proving
  that
• x y implies x 1 y 1
• using a theorem prover.

22
Outline

• What is Formal Verification?
• Model Checking
• Theorem Proving
• Systems and Tools
• Demonstrations (Eclipse, ICS, GACC)
• Conclusions and Future Work

23
Systems and Tools

• Model checkers
• SMV
• SPIN
• Theorem provers
• Little Engines of proof
• Automated domain-specific theorem provers
  (uninterpreted functions, linear arithmetic,
  arrays, bitvectors...)
• ICS, CVC lite
• GACC (Graph-based Abstract Congruence Closure)
• Other Engines of proof
• PVS, Isabelle, HOL
• BaCCS (Basic Completion with Constraint and
  Simplification)

24
Microsoft SLAM

• Clever combination of model checking and
  automated theorem proving
• An abstract program is created in which all
  conditions are replaced with Boolean variables
• Resulting Boolean program is model checked
• If model checking fails, the potential error path
  is checked in the original program using an
  automated theorem prover
• Successfully used to find bugs in Windows
  drivers.
• reducing the frequency of Blue Screens of
  Death!

25
Outline

• What is Formal Verification?
• Model Checking
• Theorem Proving
• Systems and Tools
• Demonstrations (Eclipse, ICS, GACC)
• Conclusions and Future Work

26
Eclipse

• Eclipse is an open source IDE developed by IBM
• Eclipse is a kind of universal tool platform
  an open extensible IDE for anything and nothing
  in particular.
• Eclipse can be used to create applications as
  diverse as web sites, java programs, C
  programs...
• An extension to Eclipse is called a plugin.
• http//www.eclipse.org

27
Demo of ICS (I can solve or Integrated Canonizer
and Solver)

• Developed at SRI. http//www.ics.com
• Decides formulae in a useful combination of
  theories (non interpreted formulae, bitvectors,
  reals, arrays)
• means AND, means OR, and means NOT.
• Example 1
• Icsgt sat x 2 x 3
• Answer unsat
• Example 2
• Icsgt sat x 2 x 3 x lt 3
• Answer sat, model x 2 3 - 1 x gt 0

28
Demo of ICS (I can solve or Integrated Canonizer
and Solver)

• If we want to prove p ? q, then we use
  refutational theorem proving.
• In fact, we assume that the hypotheses are true,
  and that the conclusion is false. Then, if p ? q
  is true, we will derive a contradiction (unsat).
  Otherwise, the resulting model provides us with a
  counter-example.
• Example 3
• If x 2 and y 4, then x y 6 ????
• icsgt sat x 2 y 4 x y 6
• Answer unsat
• If x 2 or x 5, then x lt 4 ????
• icsgt sat x 2 x 5 x lt 4
• Answer sat, model -4 x gt 0 x 5

29
Demo of GACC (Graph-based Abstract Congruence
Closure)

• Congruence Closure is a method for deciding
  ground equalities.
• Example From f(a) b and a c, we can deduce
  f(c) b.
• The system GACC implements a new algorithm of
  congruence closure based on the use of graph with
  labeled vertices and edges.
• GACC was implemented by Eugene Kipnis from Pace
  University under my supervision, and is available
  for online experimentation at http//www.csis.pac
  e.edu/scharff/SOFTWARE

30
Outline

• What is Formal Verification?
• Model Checking
• Theorem Proving
• Systems and Tools
• Demonstrations (Eclipse, ICS, GACC)
• Conclusions and Future Work

31
Conclusions

• Formal Software Verification is starting to
  become practical
• Still lots of work to be done
• How can it make you a better programmer?
• Document your code with the properties and
  invariants that you think should be true
• When you modify code, convince yourself that you
  are not breaking any invariants
• Hopefully, someday software will be as safe and
  reliable as the other objects built by engineers!

32
Environment

• CAFME Center create by Dr. Skevoulis
• CS 851 Software Validation and Verification
• New course offered in the Fall 2004
• Part of the degree in Software Design and
  Development
• Validation Testing
• Verification Proving
• ITR NSF Grant Little Engines of Proof with
  Stanford Research Institute, and Clarkson
  University

33
Future Work 1 Eclipse Plugin

• A java program is annotated with assertions to be
  proved
• Assertions are written in the ICS syntax
• Some assertions are computed. Assertions are
  propagated up through the program line by line
• ICS will be called from the java program, and the
  assertions will be checked
• Parts of the program will be proved correct, or
  bugs will be discovered

34
Future Work 2 Strategies

• Rewriting languages (e.g. ELAN, MAUDE, OBJ,
  ASFSDF) are very expressive, and very convenient
  to prototype theorem provers
• Rewriting languages share many features with
  functional programming languages (no side-effect,
  use of recursion, pattern matching), but they
  are based on rewriting instead of lambda calculus
• Can we develop a Java API that would be used to
  prototype theorem provers? Can we define the
  strategy on the rules in an XML document?

35
Future Work 3 Correctness of Abstract Congruence
Closure

• Abstract congruence closure is a framework that
  defines a set of inference rules and different
  strategies for congruence closure. These
  strategies include strategies that simulate
  Nelson-Oppen, Shostak and Downey-Sethi-Tarjan
  algorithms
• The rules and the strategies can be implemented
  easily using a rewriting language such as ELAN
• This implementation would be a formal framework
  to prove the correctness of the abstract
  congruence closure algorithm, and, besides, would
  permit intensive benchmarking on criteria to be
  defined

36
Future Work 4 Unsound Theorem Proving

• Many problems given to a theorem prover will be
  false conjectures. It is important to be able to
  find quickly bugs in programs.
• An inference system for unsound theorem proving
  (or theorem (dis)proving) has been defined by
  Christopher Lynch.
• The theorem (dis)prover can be prototyped in a
  rewriting language such as ELAN.
• This framework could be used to show the
  correctness of the procedure, and benchmark the
  method on criteria to be defined