Systematic Testing Scratching the surface

1
Systematic Testing(Scratching the surface)

• The art of economic testing...

2
Rehearsal for the exam ?

• How is reliability defined?
• How is testing defined?
• What is a failure? What is a defect?
• What is a test case?

3
Run-time cycle

• Failures occur when defective code is executed
  with inputs that expose the defect.
• I_e input that will lead the system into a
  failure state

Input space
I_e
Program execution state
failure states
4
Reliability and its cost

• Reliability is a probability of not causing
  failures.
• Therefore
• Reliability is correlated to the probability of
  the an input belonging to I_e occurs !
• What does this mean in practice ?
• What does this effect the way we spend time
  (money) on fault removal and tolerance ?

5
Example

• IE has an enormousamount of possibleconfiguratio
  ns!
• Imagine testing allpossible combinations!
• Which one would youtest most thoroughly?

6
Testing techniques
7
One viewpoint

• The probability of defects is a function of the
  code complexity. Thus we may identify three
  different testing approaches
• No testing. Complexity is so low that the test
  code will become more complex. Example set/get
  methods
• Explorative testing gut feeling, experience.
  TDD relies heavily on making the smart test
  case but does not dictate any method.
• Systematic testing Rigid method that can be
  argued to have high defect detection probability.

8
Our focus

• There are a lot of different testing techniques.
• We will only look at a few
• Equivalence class testing
• Boundary value analysis

9
Classes of testing
10
Classes of tests

• Black-box (BB) testing (functional/spec testing)
• No knowledge of realization of specification
  only specification is available...

input
output
Unit Under Test
Specification
11
Classes of tests

• White-box testing (glass-box / structural)
• Complete knowledge of realization.

12
Unit under Test

• Note that units in test (UUT) can be any software
  entity
• function
• method
• binary component
• subsystem
• Full system
• The techniques are more or less the same...

13
The problem of testing

• How do we know that our program is free of
  defects?
• Exhaustive input testing.

14
... and what does that mean?

• Lets revisit a simple contract
• How many test cases must we make to ensure that
  there are no errors in the implementation?

public interface Account / indsæt det
angivne beløb på denne konto. _at_param amount
beløb i hele kroner der ønskes indsat. Dersom
amountlt0 opfattes det som en fejl (der ikke
rapporteres) og kontoens indestående ændres
ikke. / public void deposit( int amount
) ...
15
Coverage (Dækning)

• We would have to test all integer values I
  Integer.MIN_VALUEInteger.MAX_VALUE !!!
• If we did so, we would have full coverage.
• In practice this is of course not feasible.
• BUT?
• We would also have to test with all possible
  account balances for each parameter value!

16
Principle

• When exhaustive testing is out of question the
  question is how to get good testing at the
  smallest possible cost?
• Principle A good test case has a high
  probability of detecting a defect.
• Systematic testing is about finding good test
  cases. Black box testing is when you analyze the
  specification in order to identify this set.

17
Equivalence Testing
18
Equivalence Class Partitioning

• Insight
• Many input are processed by the same algorithm
  part
• thus if input A is processed by the same
  algorithm as input B then adding a test case for
  B does not increase our belief in the algorithms
  reliability much.
• Example System.abs(x)
• System.abs(37) 37 pass
• Will x 42 have a high probability of finding a
  defect?
• Will x 342331 ?
• Will x -37

19
Equivalence Class Partitioning

• Thus we divide the input space into partitions
  named Equivalence Classes (ECs) with the
  representation property
• Representation
• if one element x in a EC expose a defect then all
  other elements in EC will expose the same defect
• If System.abs(37) 37 fails I am highly
  confident that so will System.abs(39) 39...

20
Equivalence Classes

• Definition
• Equivalence class (EC)
• A subset of all possible inputs to the UIT that
  has the property that if one element in the
  subset demonstrates a defect during testing then
  we assume that all other elements in the subset
  will demonstrate the same defect.

21
ECs

• ECs allow us to reduce the number of test cases
  by a very high factor
• System.abs from 232 to 2.
• Below From N to 4.
• ECs are (sub)sets (Da (del)mængder)

EC1
EC3
EC2
Unit Under Test
EC4
22
The process

• The process is
• 1. Find the ECs
• Document it in an EC table
• 2. Define test cases based upon ECs
• Document it in an Test case table
• Loop. Usually you missed something so iterations
  doesnt harm anyone ?

23
The well worn example

• 1. Find the ECs, document them
• Give each EC a reference number, 1 etc. We will
  use them in the test table.

24
Test table

• Next define the test cases, and document them

25
Easy?

• Looks easy, doesnt it?
• It is not!
• Often finding the ECs is a challenge. The devil
  is in the detail.

26
Result

• Reduction in number of test cases
• Exhaustive test
• 4.294.967.296
• Equivalence class test
• 2 test cases
• Factor 2 billion reduction ?

27
EC criteria

• Three criteria are required for an EC to be
  sound
• 1 Coverage. Every possible input belongs to one
  of the equivalence classes.
• 2 Representation. If a failure is demonstrated
  on a particular member of an equivalence class,
  the same failure is demonstrated by any other
  member of that class.
• 3 Disjointness. No input belongs to more than
  one equivalence class.

28
EC criteria

• Are these ECs sound?

-42
Coverage ? Representation ? Disjointness ?
39
37
-92431523
EC-1
EC-2
29
Finding the ECs

• More difficult than it appears

30
Reality

• It may appear that finding ECs is simple. In
  practice it is often rather hairy?
• Example Backgammon move validation
• validity v ( move, player, board-state,
  die-value )
• is Red move (B1-B2) valid on this board given
  this die?
• Problem
• multiple parameters player, move, board, die
• complex parameters board state
• coupled parameters die couples to move!
• EC boundary is not a constant!

31
A process

• To find the ECs you will need to look carefully
  at the specification and especially all the
  conditions associated.
• Conditions express choices in our algorithms and
  therefore typically defines disjoint algorithm
  parts!
• And as the test cases should at least run through
  all parts of the algorithms, it defines the
  boundaries for the ECs.
• Burnstein names a set of heuristics for doing
  this. She simply copied those stated by Myers in
  1969, but she couldnt remember well, so one is
  wrong ?

32
Partitioning Heuristics

• 1) If you have a range of values specification
• make 3 ECs
• 1 in range valid
• 2 above range invalid
• 3 below range invalid
• Ex. Adjust time of day, legal values between
  0..23
• Three ECs
• 1 0..23 2 gt 23 3 lt 0

33
Partitioning Heuristics

• 2) If you have a set, S, of values specification
• make S1 ECs
• 1..S one for each member in S valid
• S1 for a value outside S invalid
• A vending machine accepts 1,2,5,10,20 kr coins
• 6 ECs 1, 2, 5, 10, 20, 17

34
Partitioning Heuristics

• 3) If you have a boolean condition specification
• make 2 ECs
• the first character must be non-digit
• 2 ECs a1, 1a
• the object reference must not be null
• you get it, right??

35
Partitioning Heuristics

• 4) If you question the representation property of
  any EC, then repartition!
• Split EC into smaller ECs

36
Valid and Invalid ECs

• Authors like Meyers and Burnstein classify ECs as
  either valid or invalid.
• The words can be misinterpreted because it is not
  always a matter of the input being valid or not.
  I suggest that you consider it to define two
  classes of method processing
• end-to-end processing ECs valid
• bail-out processing ECs invalid
• This is arguably an algorithmic viewpoint
  (whitebox)

37
ECs classes

• Consider an archetypical algorithm/method
• function algorithm(p1,p2,p3)
• if ( ! valid(p1) ) return
• if ( ! valid(p2) ) return
• if ( ! valid(p3) ) return
• A calculateA(p1,p2,p3)
• if ( abnormal-condition(A) ) return
• B calculateB(p1,p2,p3)
• if ( abnormal-condition(B) ) return
• C calculateC(p1,p2,p3)
• return C
• A normal operation/normal use case will make the
  function process end-to-end (valid EC)
• p1,p2,p3 are all used in processing the result
• An error condition/abnormal processing will make
  the function bail-out (invalid EC)
• only pN is used to process some partial result

end-to-end
bail-out
38
Example
39
Find the ECs

• Define the EC table
• Legend
• Condition
• denote the condition in the specification we are
  looking at
• x
• the name of a particular EC
• 1 year year lt 1900
• Invalid partition
• set containing elements that are treated as
  abnormal by the UIT
• Valid partition
• set containing elements that will result in
  normal processing by the UIT

40
Invalid input checking

• However! Take care in your design to be explicit
  where invalid input handling is made.
• You do not want to check invalid data in every
  method !
• If, for instance, it was a precondition that year
  is within valid range then we would not define
  partition 1 and 2 because they would not
  occur and we of course should not define test
  cases that a method is explicitly defined not to
  handle!
• War story
• Large Danish bank system. Performance was
  terrible. They found that the check does account
  exist? was executed gt 1000 times pr. transaction
  !

41
Morale

• Be explicit in your design where the invalid
  input checks are made
• typically near to external interfaces humans,
  hardware equipment, storage, ...
• ... And encapsulate core logic (business logic)
  from the outside and assume the client will
  adhere to the contract...
• There is clearly a trade-off here !

42
Heuristics on output

• Sometimes code structure is determined by
  specified output rather than input space.
• Example
• String myFormat(double x) ?
• 12.3 iff 0 lt x lt 99.9
• otherwise
• Use the heuristics here as well.
• Exercise Which is applicable here?
• range? set? boolean condition?

43
From ECs to Testcases
44
Defining the Test cases

• The input space is 2-dimensional (year,month).
  Thus to define test cases we must of course
  combine elements from the year ECs with
  elements from the month ECs
• Example
• Test case (1830,4) combines two ECs which?
• Test case (1964,3) ?

45
How many test cases

• So
• 3 ECs for month
• 3 ECs for year
• How many test cases all in all?

46
Graphically

• How many sets have both parameters invalid?

month partitioning
EC-1 5 ? 1
EC-2 5 ?3
EC-3 5 ?2
gt12
EC-5 6 ?3
EC-4 6 ?1
EC-6 6 ? 2
EC-9 4 ?2
EC-8 4 ?3
EC-7 4 ?1
lt1
lt1900
gt3000
year partitioning
47
Is this a good way?

• The question is Are all these test cases good
• Consider (1830, 0)
• Invalid year, invalid month
• And the following code
• Does the test case help us?

48
Masking of invalid partitions

• The effect is called masking and appears
  typically when you have multiple invalid
  parameters at the same time because error
  checking code typically tests one condition at a
  time thus the early correct checks simply masks
  the later defective checks.
• Morale Only one condition invalid at a time!

if ( year lt 1900 ) throw new IllegalArgumentExcept
ion() if ( month lt -1 ) throw new
IllegalArgumentException()
49
Myers rule for invalid ECs

• 1 Until all invalid partitions have been
  covered, define a new EC (and test case) that
  covers one, and only one, of the uncovered
  invalid partitions

50
Interesting subsets after Myers rule
month partitioning
EC-1 5 ? 1
EC-2 5 ?3
EC-3 5 ?2
gt12
EC-5 6 ?3
EC-4 6 ?1
EC-6 6 ? 2
EC-9 4 ?2
EC-8 4 ?3
EC-7 4 ?1
lt1
lt1900
gt3000
51
Finding ECs is an iterative process
52
But

• Isnt there a thing with leapyears?
• Is year 1977 representative for 1964 and 2000?
• No, earths rotation around the sun is not a
  multipla of earths own rotation speed, there is
  about ¼ day more in a solar year than in our
  calendars year. If this accumulates then it will
  snow in august within a few hundred years. So we
  make leap years that are ¾ day longer than the
  solar year to catch up on the error

53
...but

• Or said in another way Does partition 3
  (1900-3000) qualify to have the representation
  property?
• I fear not - so a further subdivision will
  probably lead to better test coverage and
  therefore higher reliability.
• Morale Finding ECs is also an iterative process

54
2nd try...

• Now we can question the representation property
  of the 6 partition because does the first two
  month behave differently than the rest in leap
  years? We are worried so lets repartition

55
Defining valid test cases

• So
• 4 ECs for year, and 2 ECs for month
• What is the total of test cases?
• Combining every combination will lead to a
  combinatorial explosion of test cases ?

56
Myers rule for valid ECs

• 1 Until all valid partitions have been covered,
  define a new EC (and test case) covering as many
  uncovered valid partitions as possible
• What is the good argument this time???
• Why am I allowed to combine as many valid as
  possible???

57
The final test cases

• I combine valid ones, take only one invalid at a
  time. All in all 8 test cases (contrast 16)
• (Output needs to be looked up in a calendar?)

58
Graphically

• The resulting test case subsets.

month
Note no test cases for several ECs!
gt12
6b) 3-12
6a) 1-2
lt1
gt3000
lt1900
3a
year
3c
3b
3d
59
Conditions versus Parameters

• Specification conditions are often coupled to
  several (method) parameters. You must partition
  on the condition then, not the parameters!

a
a
E3
E5
E6
E4
E1
E2
E1
E2
E3
b
b
Independent parameters
Coupled parameters
60
Example 3

• A simple example of coupled parameters
• What are the partitions of the three dimensional
  input space (x,y,T) ?

61
Example 3

• Here we must express the partitions in terms of
  the combination of parameters...

62
Graphically
y
2
1
x
63
Conditions by functions

• Sometimes conditions are specified in terms of
  functions of method parameters. Example Moving
  a checker in Backgammon
• boolean move(Position from, Position to)
• Depends (among other things) on the die thrown.
• Condition distance match die
• Valid conditions
• 1 d(f,t) lt die 2 d(f,t) die 3 d(f,t) gt
  die

64
A note on the function

• The function may be purely a way for us to
  specify the EC table, it will not occur in the
  testing code, and sometimes not even in the
  production code.

65
Summary EC testing

• Process
• Find the ECs
• Look at the conditions specified in the
  specification
• Use the heuristics for range, set, boolean
• If in doubt, repartition!
• Define the test case
• Method 1 Make all combinations
• Method 2 Myers rules
• Valid combine as many as possible
• Invalid Only one invalid EC at at time
• ECs must be sound
• Coverage (all input space covered)
• Representation (Representative will show same
  defect as all others in same EC)

66
Myers rule for valid ECs

• 1 Until all valid partitions have been covered,
  define a new EC (and test case) covering as many
  uncovered valid partitions as possible
• 2 Until all invalid partitions have been
  covered, define a new EC (and test case) that
  covers one, and only one, of the uncovered
  invalid partitions

67
Boundary value analysis
68
Boundary value analysis

• Experience shows that test cases focusing on
  boundary conditions have high payoff.
• Some that spring into my mind is
• iteration over a C array at its max size
• off by one errors in comparisons
• if ( x lt MAX_SIZE ) and not if ( x lt MAX_SIZE )
• null as value for a reference/pointer

69
Complements EP analysis

• Boundary analysis requires elements to be
  selected such that each edge of the equivalence
  class is subject of a test

70
Boundaries

• Boundaries are between ECs!

month
gt12
6b) 3-12
6a) 1-2
lt1
gt3000
lt1900
3a
year
3c
3b
3d
71
Some rules

• range of values test cases for the ends of the
  range and just beyond
• number of values test cases for one above and
  one beneath maximum and minimum values. 1-255
  would be 0, 1, 255, and 256
• ordered sets focus on first and last element
• i.e. are they ordered properly?
• be creative ?

72
Summary

• Black-box / White-box testing
• Exhaustive testing is not feasible
• Equivalence Testing
• EC set of input elements where each one will
  show same defect as all others in the same set
  (representation)
• Boundary analysis
• be sceptical about values at the boundaries
• Especially on the boundary between valid and
  invalid ECs