Technology of Test Case Generation

1
Technology ofTest Case Generation

• Levi Lúcio
• University of Geneva
• Marko Samer
• Vienna University of Technology

2
Overview
Introduction Model Checking Symbolic
Execution Theorem Proving Conclusions
3
Introduction

• In this presentation three technologies for test
  case generation are introduced
• Symbolic Execution
• Theorem Proving
• Model Checking (presentation not available)
• All the models used by these methods can be
  reduced to state machines
• Test case generation can be seen as state space
  exploration for traces obeying a criteria.

4
Overview
Introduction Model Checking Symbolic
Execution Theorem Proving Conclusions
5
Overview
Introduction Model Checking Symbolic
Execution Theorem Proving Conclusions
6
Symbolic ExecutionTalk Overview

• The technique
• How symbolic execution works
• Issues related to the approach
• Symbolic execution of abstract models

• Test case generation
• A classification of test case generation
  frameworks using symbolic execution
• Abstract model based test case generation
• Code based test case generation.

7
Symbolic Execution

• Technique invented in the 1970s for verifying the
  consistency of code
• The main principle consists of executing the code
  with symbolic inputs

• in which a symbolic input consists of replacing
  a  real input  (number, string, structure,
  object) with a symbol
• For example, if the program asks a number to the
  user,  a  would be passed to the program as a
  representation of all possible numbers.

8
The technique

• Allows exploring the possible control paths in an
  program. Calculates the path condition for each
  control path, composed of symbolic equations.

Int foo (int a, int b) 1 a 2 if (agtb) 3
aa-b 4 else 5 ab-a 6 if (alt-1) 7
a-a 8 return a
Control paths for foo
1,2,3,6,7,8
1,2,3,6,8
1,2,5,6,7,8
1,2,5,6,8
9
The technique (cont)
a a1, ba2 PC True
Int foo (int a, int b) 1 a 2 if (agtb) 3
aa-b 4 else 5 ab-a 6 if (alt-1) 7
a-a 8 return a
10
The technique (cont)

• From the symbolic execution of foo, one can
  retrieve the following information

Control Path Path Condition
1,2,3,6,7,8 a1-a2 gt -1, a1-a2 lt -2 Not feasible
1,2,3,6,8 a1-a2 gt -1, a1-a2 gt -2 ? a1-a2 gt -1
1,2,5,6,7,8 a1-a2 lt -1, a1-a2 gt 0 Not feasible
1,2,5,6,8 a1-a2 lt -1, a1-a2 lt 0 ? a1-a2 lt -1

• Each path condition fully documents the decisions
  taken during the execution of the code.

11
Test case generation

• It is possible to generate test cases by finding
  values that satisfy the path condition equations

Control Path Path Condition Example test case
1,2,5,6,8 a1-a2 lt -1 a1 1, a2 3

• Test case (a1, b3) would force the execution of
  foo to follow the control path 1,2,5,6,8
• Test cases are thus created as by-products of the
  activity of symbolic execution
• Symbolic execution uncovered that in the case of
  foo instruction 7 is never reached...

12
Issues related to the approach

• Dealing with loops
• Loops generate infinite control paths (unless the
  number of iterations is known). Solution?
• Prompt the user at each loop iteration
• Establish an upper bound for the number of
  iterations
• Try to automatically find a fixed point to the
  loop (costly).
• Solving path condition equations
• Necessary for
• understanding when a path is infeasible
• generating test cases.
• Algorithms heavy on computing resources.

13
Symbolic execution of abstract models

• Up until now we have considered symbolic
  execution only as a white box verification
  technique
• However, nothing prevents from applying the same
  technique to an abstract model (state machine)

• State space of a model can be symbolically
  searched for interesting control paths
• Symbolic execution reduces state space explosion
  by associating classes of inputs.

14
Symbolic execution for test case generation

• In the text several frameworks for test case
  generation from different models of the
  application are studied
• We classify the frameworks according to two axis

Synergies with other verification techniques
Model Checking
Theorem proving
Abstract (B, AUTOFOCUS, CO-OPN), Code (Java)
Application model
15
Abstract model based test case generation

• In the text three frameworks are described that
  have their starting point on different abstract
  models
• B (framework from Legeard, Peureux et al)
• AUTOFOCUS (framework from Pretschner et al)
• CO-OPN (framework from Buchs et al)
• The models can be considered as state machines
  and the technique for test case generation is
  similar

16
Abstract model based test case generation (cont)

• The step of translation into a logic programming
  language (theorem prover) is necessary to
• Animate the (static) abstract specification
• Search the state space
• Perform symbolic execution.

• Two kinds of logic programming languages may be
  used
• Pure Prolog
• Constraint Logic Programming (CLP) Languages
  enabled to deal with numeric constraints at the
  unification level.

17
Test generation from an Abstract ModelExample

• The specification language chosen is B - supports
  development of C code from an abstract model
• In B an application can be seen as a state
  machine
• States defined by state variables
• Transitions parameterized operations.

• Framework from Legeard et al divides the activity
  of test case generation into three steps
• Translating B into CLPS-B (custom CLP language)
• Calculate boundary states for the application
• Generate the test case traces through the state
  space.

18
Test generation from an Abstract ModelExample
(cont)

• In a boundary state one or more state variables
  assume a boundary value
• Calculation of boundary values is done by
  partitioning the domain of state variables (by
  symbolic execution)

19
Test generation from an Abstract ModelExample
(cont)

• Traces that represent interesting test cases are
  then generated as follows
• Calculate the preamble trace to the boundary
  state
• Calculate the body invocation step
• Calculate the identification traces.

• An instantiation phase of the concatenated traces
  is necessary to create the real tests
• Until then only the path condition exists.

20
Test generation from an Abstract ModelExample 2
AUTOFOCUS

• AUTOFOCUS is a CASE tool inspired from UML-RT
  (real time) and directed towards embedded
  systems
• In this paradigm a system is defined as a
  hierarchic network of components, represented by
  state machines

21
Test generation from an Abstract ModelExample 2
AUTOFOCUS (cont)

• Framework from Pretschner et al considers three
  possible test coverage classes
• Functional covers the specifications execution
  scenarios
• Structural e.g. test critical components of the
  system
• Stochastic generate random state space traces.
• The coverage heuristics are annotated in the
  specification

22
Test generation from an Abstract ModelExample 2
AUTOFOCUS (cont)

• AUTOFOCUS specification is translated into a CLP
  language for animation.
• Each transition of a bottom level component K is
  modeled into a formula of type

• Interesting traces (according to coverage
  heuristics) are generated by symbolic execution
  over system inputs.
• Instantiation of symbolic traces is the last step

23
Code based test case generation

• Why take a detour from model based test case
  generation (title of the section)?
• In the text we present a framework from Khurshid
  et al that generates test cases from Java code.
  It uses
• A model checker (Java PathFinder) to overcome
  some difficulties in symbolic execution
• The symbolic execution technique to overcome some
  difficulties in model checking.
• We think an insight on the framework is enriching
  for the discussion

24
Test generation from codeExample

• Symbolic execution helps dealing with state space
  explosion by dealing with inputs as classes of
  values
• Model checking deals automatically with loops,
  recursion or method invocation

25
Test generation from codeExample (cont)

• The code is instrumented to deal with symbolic
  values and accumulate path conditions (via Java
  libraries)

• The algorithm that performs symbolic execution
  knows how to deal with complex dynamic structures
  (objects)!
• A phase of test case instantiation is required

26
Overview
Introduction Model Checking Symbolic
Execution Theorem Proving Conclusions
27
Overview
Introduction Model Checking Symbolic
Execution Theorem Proving Conclusions