An Automated Framework for TestData Generation

1
An Automated Framework for Test-Data Generation

• Nigel Tracey
• 9th October 1998

2
Overview

• Introduction
• Approaches to automation
• Testing as an optimisation problem
• Optimisation techniques
• Simulated annealing
• Genetic algorithms
• Specification failure testing
• Structural testing
• Assertion/exception testing
• Reuse testing
• Results, future work and conclusions

3
Introduction

• Software testing is (or can be)
• Difficult
• Tedious and labour intensive
• Poorly integrated into the development process
• Abused and/or misunderstood
• Software testing is expensive and time-consuming
• Typically at least 50 of initial development
  costs
• However, remains the primary method for gaining
  confidence in the correct functioning of software
• Automation, automation, automation
• Reduce development costs and increase confidence
  and possibly quality

4
Approaches to Automation

• Static (symbolic execution)
• Symbolically executes software to obtains
  constraints to be solved
• Solutions to the constraints represents the
  desired test-data
• Problems
• Computationally expensive, arrays, loops, module
  calls and undecidable nature of
  constraint-solving
• Dynamic (search)
• Perform a dynamic search for test-data actually
  executing the software
• Aim to find optimal values of a cost-function
• Problems
• Complexity of search space (size, non-linearity)

5
Testing as Optimisation
Inputs
Outputs
Program
X 3
Cost 48
X gt 50
X 50
Cost 1
X 55
Cost 0
Test Condition - could be internal to the program
in which case it is evaluated during program
execution or it could be external to the program
and evaluated after execution
6
Cost Surfaces

• Simple cost surface - if X were an input even
  simple hill climbing would succeed. However X may
  be form using an unknown mapping of input values
  to values of X.
• Examples of more complex cost surfaces later

7
Optimisation

• Search techniques which optimise a cost function
• Simulated annealing
• Based on local hill climbing search
• Makes few assumptions, require only
• Cost function
• Depends on the problem being addressed
• Generation mechanism
• Neighbourhood structure
• Genetic algorithms
• based on simulation of evolution process
• Tabu-Search

8
Testing Specification Failure

• Formalised Specification (Pre/Post Condition)
• Gain confidence in implementation using
• Proofs
• Finding bugs using proof is extremely expensive
• Demonstration (i.e. testing)
• Bugs are highlighted by test-data which
  satisfies the pre-condition but after execution
  does not satisfy the post-condition
• Automatic generation of such test-data could
• Make proofs cheaper
• Highlight problems quicker than traditional
  testing

9
Specification Testing
Valid Input Domain
Input Domain
Program
Valid Output Domain
Output Domain
10
Specification Failure - Bug
Valid Input Domain
Input Domain
Program
Valid Output Domain
Output Domain
11
Dynamic Approach

• Need cost function to guide the search
• Pre-condition and negated post-condition
  converted to Disjunctive Normal Form
• Example
• Pre (N ³ 0 Ù N 10) Ù
• Post (N lt 10 N N 1) Ù (N 10 N 0)
• Becomes
• (N ³ 0 Ù N 10) Ù (N lt 10 Ù N ¹ N 1)
  Ú(N ³ 0 Ù N 10) Ù (N 10 Ù N ¹ 0)
• Can process each disjunct separately
• Each disjunct encodes a possible error condition
  of the implementation

12
Dynamic Approach

• Add values for each term

• Example
• X gt 10 Y lt 20
• X 5, Y 25 then cost (10 - 5) K (25 -
  20) 10 2K
• X 10, Y 20 then cost (10 - 10) K (20 -
  20) K 2K
• X 11, Y 19 then cost 0

13
Dynamic Approach

• The cost function guides the search
• The cost surface is likely to be complex
• It is likely to be large
• It may also not be linear or continuous
• Implies hill-climbing or neighbourhood searches
  will fail
• For this reason heuristic global optimisation
  techniques are required

14
Dynamic Approach - Example
15
Structural Testing as Optimisation

• Search for test-data which causes execution of
  the desired part of the software
• Need to provide guidance for the search
• Branch predicates determine the path executed
• Cost function is based on the relational
  expression in the branch predicate, e.g.
• a b, if (abs(a-b) 0) then 0 else K
• a lt b, if (a - b lt 0) then 0 else (a-b) K
• a ? b, min (cost (a), cost (b))
• Example, X lt 10
• X 20, cost (20-10)K 10 K
• X 10, cost (10-10)K K
• X 9, cost 0
• Cost is evaluated during execution, SUT is
  instrumented

16
Instrumentation

• Execution monitor calls
• Record that node X has been executed
• If X is current target node then update target
  node to next node on path
• Branch evaluation calls (return boolean value)
• If required to be true, add cost (branch pred),
  if required to be false add cost (? branch pred)
• For loops the desired number of iterations
  determines whether cost (loop pred) or cost (?
  loop pred) is added
• Within loops, the cost of predicates is deferred
  until loop exit. The minimal cost value is added
  at that point. This prevents punishment of taking
  an undesirable branch immediately as the
  desirable branch may be taken later

17
Example

• Path 1, 2, 8, 12
• N-10, D2
• Cost_1 (0) (2K) 2K
• N-10, D-2
• Cost_1 (0) (0)
• Cost_4 (10-2)K 8K
• N-10, D-20
• Cost_1 (0) (0)
• Cost_4 0
• Success!

18
Boundary Value Analysis

• Test-data near boundaries is considered better
• Can detect badly placed boundaries, which is a
  common programming bug
• Need to guide search to boundary
• Example,
• X lt 42, if the desired path requires this to be
  false the cost-function will search for data
  satisfying X gt 42
• Adding the cost of X 42, gives bias to boundary
  value
• The cost of bias is scaled so that the branch
  cost dominates the guidance
• Removes zero cost stopping criteria for search
• Search will still terminate with optimal values,
  but may take longer

19
Assertion Testing

• Again find negative tests to highlight bugs
• Test for false assertions or exception conditions
• Only Constraint_Error in SPARK Ada
• Dynamic Approach
• Objective function as before but calculated at
  point of assertion or exception check
• Search also requires guidance to test-data which
  exercises the assertion or exception check
• Achieved using instrumentation
• Each conditional branch adds a cost related to
  how close it is to having the right outcome

20
Assertion Testing - Example
21
Assertion/Exception Testing

• Goal is to find data to break assertions or cause
  exceptions
• Twofold guidance to search
• Guidance to statement associated with the
  assertion or possible exception
• Cost augmented
• cost (? assertion pred)
• cost (exception condition)
• Improving guidance
• Converting the assertion or exception condition
  to disjunctive normal form
• Each disjunct represents one way the software may
  fail
• Perform separate searches for each disjunct in
  turn

22
Component Reuse Testing

• When reusing a component it is vital the
  environment meets the assumptions of the
  component
• i.e. for all data values at point of call the
  pre-condition of the reusable component is true.
• In reuse testing we are looking for test-data
  that shows a component is being reused unsafely
• Search guidance, again twofold
• Guidance to point of call to reusable component
• Cost augmented with cost (? pre-condition)
• Again DNF conversion can improve guidance

23
Reuse - Bug
Program
Valid Input Domain
Reusable Component
24
Evaluation

• Collection of Ada 95 programs (20 - 2000 LOC)
• 100 branch coverage achieved
• Search time 2 to 35 seconds
• Required some tunings of optimisation technique
  parameters
• Specification failure

25
Evaluation Contd

• Assertion/Exception testing

26
Further Work

• Investigation of strengths of each optimisation
  technique
• More Intelligence
• Integration with static techniques/decision
  procedures
• Improved search guidance
• Data flow analysis, alternative cost functions
• Metrics for selecting search technique and
  parameters
• Large scale study
• Working on aircraft EEC code - 100,000 LOC
• WCET and structural coverage

27
Conclusions

• Work so far has demonstrated feasibility
• Automatic structural testing may allow better
  target of testing resources
• 50 of test effort can be spent on achieving full
  coverage however metrics show it rarely finds
  problems
• Ability to automatically find counter-example
  tests may significantly reduce costs of formal
  development methods
• Optimisation gives a general test-data generation
  framework for many testing problems
• New problems can be automated by devising a
  suitable cost function