Testing against a deterministic extended finite state machine

1
Testing against a deterministic extended finite
state machine

• Rob M. Hierons
• Goldsmiths College, University of London

2
Overview

• Finite State Machines (FSMs) and Extended Finite
  State Machines (EFSMs)
• Test criteria and Fault Models.
• Applying Genetic algorithms.

3
Finite State Machines

• A finite state machine M is defined by the
  following
• A finite set of states S
• An initial state s0
• An input alphabet X
• An output alphabet Y
• Next state and output functions and

4
Example
5
How finite state machines operate

• Suppose the FSM M is currently in state s and is
  sent input x
• Then
• M moves to state s (s,x)
• M outputs y (s,x)
• This defines a transition (s,s,x/y).

6
Relevance of FSMs

• Often an important aspect of a system may be
  described in terms of state transitions.
• FSMs are often used to describe the control
  structure of a system.

7
Testing from an FSM

• When testing an implementation against an FSM it
  is normal to use one of the following
• A test criterion that says when we can stop.
• A fault model that described the possible
  behaviours of the implementation under test (IUT).

8
Extended Finite State Machines

• An EFSM is an FSM in which
• Each logical state has some associated
  memory/store
• Each transition involves the application of an
  operation that takes the store and input and
  generates output and a new value for the store.

9
Why EFSMs

• Many specification languages effectively use
  EFSMs.
• Examples
• SDL, Estelle, LOTOS.
• Statecharts.
• Note these languages allow concurrency - we
  shall not consider this aspect.

10
Example

• The following describes part of a simple bank
  account.

11
The store

• The store is composed of two values
• the current balance b
• the current overdraft limit Od
• There are two states
• sc where the current balance is non-negative
• so where the current balance is negative.

12
The operations

• The example contains the following operations
• m - returns the current balance
• c - changes the overdraft limit
• w - withdraw money
• d - deposit money
• Operators d and w have two variants
• suffix 1 - after the operation b 0
• suffix 2 - after the operation blt0

13
Specifying the operations

• They are defined by following (x 0)

14
Using FSM test methods when testing against an
EFSM

• The following approach might be used
• Abstract the EFSM to form an FSM.
• Generate tests from the FSM.
• Translate these tests into tests for the EFSM.
• This approach might test the control structure of
  a system.
• Tests, for the data aspects, are added to this.

15
Weaknesses

• Testing the control and data aspects separately
  suffers from a number of problems including
• Test sequences generated from the FSM need not be
  feasible in the EFSM.
• Faults in the control and data aspects may mask
  one another.

16
Alternatives

• We might apply one of the following
• Test a number of transitions from M.
• Use a fault model for M and generate tests from
  this.
• Here we shall largely focus on the first of these
  BUT in effect we shall use a fault model for each
  transition.

17
Checking a transition

• Suppose we wish to check transition t of M.
• We need to do the following
• Find some input sequence that should lead to t
  being executed.
• Find further input sequences that check the state
  and store of M after t.
• For simplicity we shall assume that
• the states simply partition the store - we dont
  have to separately check the state.

18
Executing a transition

• Suppose we wish to execute some transition t
• We shall consider two approaches
• Derive some path p that ends in t and try to find
  input that executes p.
• Without deriving a path prior to this, try to
  find an input sequence that drives M into the
  initial state of t and then find input to execute
  t.

19
Executing a path

• Suppose we wish to execute sequence t1tk of
  transitions.
• The search space is the set Xk of input
  sequences of length k.
• We thus get a chromosome comprising of these k
  values or bit representations of them.

20
Fitness 1

• Suppose some input sequence x from Xk has been
  chosen.
• Suppose also that x leads to the execution of
  sequence t1tjtj1tk (tj1 tj1).
• Then we might set the fitness to be j the
  distance along p followed by x.

21
Defining this fitness function

• Given sequence y of length i let pre(j,y) denote
  the prefix to y that is of length j (j i).
• Suppose also that, given input sequence x, p(x)
  denotes the sequence of transitions from M
  triggered by x
• Then the fitness of x is
• maxjpre(j,p)pre(j,p(x))

22
Refining this fitness

• The fitness above does not include an element
  that states how close the input is to taking the
  next transition tj1 from p.
• In order to include this we might
• Partition the domain D of tj1 into D (input
  that takes tj1) and D-.
• Devise a metric for D.
• determine the distance d from the nearest element
  of D.

23
The new fitness function

• Suppose input sequence x follows the first j
  elements of p and the store/input for the next
  transition is distance d from D.
• Then the fitness is
• Note this fitness cannot exceed that of an input
  sequence that follows p further.

24
Deriving fitness functions

• A local fitness function for each operation may
  be defined in advance.
• When a path is chosen a fitness function based on
  the local fitness functions may be produced.

25
Example

• Suppose we wish to test the transition that takes
  M from sc to so using operation w2.
• We might initially choose this path of length 1
  but
• assuming the initial store is (0,0) this is not
  feasible.
• Instead we will look at the path cw2.

26
Fitness 1

• Suppose the input sequence xy is chosen.
• Any value for x will lead to the execution of c.
• The precondition for w2 after c is
• y x and ygt0
• If xy leads to the execution of cw2 it has
  fitness 2.

27
Fitness 2

• Suppose xy does not lead to the execution of cw2.
• Then the fitness might be
• This gives a fitness between 1 and 2, increasing
  as we get closer to satisfying the precondition
  for w2.

28
Boundary tests

• We might want to test near boundaries.
• Suppose x follows p, in x the input/store for t
  is z, D denotes the input/store space for t, and
  D- denotes the values from D that dont follow t
• Suppose z is distance d from D-.
• The fitness of x is.

29
Reaching a state without choosing a path

• We might choose input sequences and determine how
  close they come to the initial state of t.
• Then we need some notion of distance between
  states.
• This might be
• d(s,s) the length of the shortest path in M
  from s to s.

30
The fitness function.

• Suppose input sequence x reaches state s of M.
• Suppose also that we wish to reach state st.
• Then the fitness of x is

31
Properties of this fitness function

• If x reaches state st then the fitness is simply
  1/(1x) and thus the optimal value is given by
  the shortest sequence that reaches st
• The factor of 2 guarantees that an increase in
  sequence length of 1 that gets 1 closer increases
  the fitness.

32
Relative Merits.

• The main disadvantage of the path based approach
  is
• The path chosen may not be feasible or it may
  only be followed by a small number of sequences
• The main advantage of the path based approach is
• The fitness function is a little more refined.
• The input sequence has fixed length.

33
Checking the Store

• Suppose some input sequence x that executes t has
  been chosen.
• We wish to follow x by sequences that check the
  final store of t (we assume the states simply
  partition the store).
• We assume other ops are correct.
• We shall suppose that
• z is the expected store after t
• Z is the set of alternatives.

34
A greedy algorithm

• Let D(Alt,z,y) z Alt y distinguishes z and
  z and E(Alt,z) denote the elements of Alt that
  are equivalent to z.
• Let Alt Z
• While Alt E(Alt,z) do
• Find optimal test y
• Alt Alt \ D(Alt,z,y)
• Od
• Tests are used in the order generated.

35
The fitness of a sequence.

• If we can find a fitness function for the tests
  we might use GAs to find the optimal test to
  distinguish states.
• Given Alt, y, the fitness of y is

36
Example

• Consider the application of test sequence xy to
  test w2
• The expected final store is (-y,x).
• Let B denote the set of balances and OD the set
  of overdraft values.
• Then we might set ZB OD.
• This might be very large.

37
Reducing the set of alternatives

• We might assume that
• tests that detect small differences in the store
  will detect large differences in the store.
• In this case it seems likely that this is
  correct.
• Then we can set
• Alt -y-1,-y,-y1 x-1,x,x1

38
Generating tests

• Here the problem is sufficiently simple for us to
  note that
• m determines whether the first component of the
  store is correct and so reduces Alt to Alt2-y
  x-1,x,x1
• Using w2 with x-y will now produce an error if
  the second component is x-1, reducing the space
  to Alt3-y x,x1
• Finally, w2 with x-y1 will erroneously work if
  the overdraft limit is x1.

39
Fault Models

• Given EFSM M a fault model F is a set of
  behaviours that describe the alternatives for the
  implementation under test.
• Usually the behaviours in F will be described by
  EFSMs.

40
Testing from a Fault Model

• Given fault model F we might generate tests that
  distinguish between M and elements of F.
• Again, we might use a greedy algorithm and GAs to
  find tests within this.

41
Conclusions

• It is not always sufficient to separately test
  control and data.
• The complexity of EFSMs often makes analytic
  approaches impractical.
• Instead we might consider using meta-heuristics
  and either
• test individual transitions
• use general fault models.

42
Future Work

• We need to apply some of these approaches.
• Often a fault model or the set of alternative
  values for a store will be very large we need
  techniques to reduce these.