Assume%20Guarantee%20Testing

1
Assume-Guarantee Testing
Colin Blundell (UPenn) Dimitra Giannakopoulou
(Riacs) Corina Pasareanu (QSS)
Robust Software Engineering NASA Ames Research
Center Moffett Field, CA, USA
2
Component-Based Systems

• Systems built in a modular fashion (from
  components)
• Also verify systems in a modular fashion!
• infer properties form local properties of
  components
• construct appropriate environments to check
  components in isolation
• Efficient verification of assembled system based
  on knowledge about its structure

3
Compositional Verification
Does system made up of M1 and M2 satisfy property
P?
M1

• Check P on entire system too many states!
• Use the natural decomposition of the system into
  its components to break-up the verification task
• Check components in isolation M1 satisfies P?
• Typically a component is designed to satisfy its
  requirements in specific contexts
• Assume-guarantee reasoning introduces assumption
  A representing M1s context
• Reasons about triples ?A? M1 ?P?

A
M2
4
Assume-Guarantee Rules

• Simplest assume-guarantee rule
• Symmetric rules - example

M1
A1
A2

1. ?A1? M1 ?P?
2. ?A2? M2 ?P?
3. C (A1, A2, P)
4. ?true? M1 M2 ?P?

M2
Coming up with assumptions is a non-trivial
process
5
Approaches

• Infer assumptions automatically
• Two solutions developed
• Algorithmic generation of assumption
  (controller) knowledge of environment is not
  required
• ASE02, JASE05
• Incremental assumption computation based on
  counterexamples, learning and knowledge of
  environment
• TACAS03, SAVCBS03 (symmetric rules)

Challenge what about actual implementations?
6
Context

• Components modeled as labeled transition systems
  (LTSs)
• assembled with parallel composition operator
  that synchronizes shared actions and interleaves
  remaining actions
• a trace t is a sequence of actions that can be
  performed from the initial state t ?? is t where
  all actions not in ? are removed
• A property P is an LTS
• describes legal behaviors in terms of its
  alphabet ?P
• An assumption A is an LTS
• restricts the environment to the set of its legal
  behaviors

7
Design/Code Level Analysis
M1
M2
P
A
Design
C1
C2
A
P
Code

• Does M1 M2 satisfy P? Model check.
• Does C1 C2 satisfy P?
• Model check (ICSE2004) good results but may
  not scale
• TEST!

8
Assume Guarantee Testing
Assume-Guarantee traces C1 in ? send ?
i1 C2 out ? send ? i2
Monolithic - C1 C2 traces in ? out ? send ?
i1 ? i2 in ? out ? send ? i2 ? i1 out ? in ?
send ? i1 ? i2 out ? in ? send ? i2 ? i1
9
Discovering Bugs with Fewer Tests
Monolithic
AG
send
send
10
AG Testing in Practice

• Create testing environments for C1 and C2
• for C1, universal environment restricted by
  assumption A
• for C2, universal environment
• Components can be checked separately as soon as
  they are code complete
• More refined testing environments fewer false
  positives
• More control over component interface easier to
  reproduce behavior and exercise more traces
• Potential for checking more behaviors of the
  system with the same amount of coverage, as in
  our example
• incompatible traces may not execute shared events
  in the same order
• enough traces must be generated to ensure
  appropriate coverage
• Open problem what is an appropriate measure of
  component coverage?
• potentially use component models as coverage
  metric

11
Predictive Analysis
12
Predictive Analysis

• No problem with incompatible traces
• AG approach is more efficient that existing
  predictive approaches that compose traces
• If models are unavailable, can apply our
  assumption generation techniques to the projected
  traces (needs further experimentation)

13
Case Study K9 Executive

• Executive component executes flexible plans
• Branching on state / temporal conditions and
  support for floating contingencies
• 35K lines of C code
• multiple threads that communicate through an
  event queue
• Models created in collaboration with developers
• assumptions generated at model level several
  synchronization problems detected
• compositional MC achieved x10 improvement in
  space over monolithic MC
• AG testing detected inconsistency between model
  and code

14
Testing Framework
Design-level Assumptions and Properties
instrumentation
K9 Executive
event stream
input plans
reports
15
Related Work

• Component based design and verification
• Mocha model checker (Alur et al.), interface
  automata (de Alfaro Henzinger), thread-modular
  verification (Flanagan et al.)
• Assume guarantee model checking for source code
• with Verisoft using manually provided assumptions
  (Dingel)
• with JPF (Giannakopoulou, Pasareanu, Cobleigh)
  using automatically generated design level
  assumptions
• Specification-based testing
• Jagadeesan et al., Raymond et al., use
  specifications (assumptions) to generate test
  inputs and (guarantees) to generate oracles
• Grieskamp et al. use AsmL to generate FSMs that
  are traversed in different ways to generate test
  inputs
• Predictive analysis, Sen et al.
• Use of AG reasoning to combine runtime monitors,
  Levy et al.

16
Conclusions Future Work

• AG testing improves traditional testing
• detects violations of system requirements when
  testing individual components
• predicts violations of system requirements based
  on correct system runs
• Measure the benefits
• come up with appropriate coverage criteria for
  component-based testing when have individual
  components been tested enough to guarantee
  correctness of assembly?
• potentially use models for definition of
  coverage