Interaction Testing with Covering Arrays

1
Interaction Testingwith Covering Arrays

• Alan WilliamsSchool of Information Technology
  and Engineering (SITE)University of Ottawa
• awilliam_at_site.uottawa.cawww.site.uottawa.ca/awi
  lliam

2
Outline

• Background of the problem test configuration
  selection
• What is a covering array, and why can they help?
• Various methods to find covering arrays
• Whats new adding a graph
• Some results

3
Testing of Component-Based Systems
Payment Server
BusinessWeb Server
Client browser
Type MasterCard,Visa,American Express
Business Database
Type Netscape,Explorer,
Type WebSphere,Apache,.NET
Type DB/2,Oracle,Access

• Suppose you have an e-commerce application, that
  is intended to run on the above infrastructure
  no matter which version of a component is chosen.

4
Testing in the Presence ofPolymorphism and
Inheritance
Method invocation
class hierarchy
...

• Sender, receiver, and message parameters could be
  instances of various classes within the class
  hierarchy.

5
Equivalence Class Testing of Methods

• public void aMethod( int x, int y, int z )
• ...
• Suppose that the equivalence classes for each
  individual parameter are
• x (-?, -4 (-4, 3 (3, 8 (8, ?)
• y (-?, 12) 12, 34 (34, ?)
• z (-?, 0) 0 (0, ?)
• What about combinations of equivalence classes
  for x, y, and z?

6
The common problem

• We have several parameters (components, class
  hierarchies, variables).
• For each parameter, there is a finite set of
  discrete values (component types, object
  classes, equivalence classes)
• The value for each parameter can be chosen
  independently of any other value.

7
Issues for Testers

• Assumption Suppose we already have a
  sufficient test suite for a single
  configuration.
• If we do not have the resources to test all
  configurations, which ones should be selected for
  testing?
• If test cases are automated, they will need
  modification for a particular execution
  environment.

8
Objectives

• Develop a measure that shows how well potential
  interactions are covered by a set of test
  configurations.
• Determine how to achieve the highest interaction
  coverage with the fewest number of configurations.

9
Test Configurations

• Created by selecting one value for each parameter
  from the set of permitted values.
• The entire e-commerce application test suite is
  run for each configuration that is selected.

Explorer
Browser
Apache
Web server
Visa
Payment method
DB/2
Database
10
Interaction elements

• Choose a subset of the parameters.
• The size of the subset is the interaction degree
• Choose specific values for those parameters.

Explorer
Browser
Web server
Visa
Payment method
Database
11
Example

• Suppose that we have three parameters P1, P2, P3
• For each parameter, there are two possible
  values.
• Values are
• A, B for parameter P1.
• C, D for parameter P2.
• E, F for parameter P3.
• Degree of interaction coverage is 2.
• We want to cover all potential 2-way interactions
  among parameter values.

12
Set of potential test configurations
P1
P2
P3
Three parameters, P1, P2, P3 each of which has
two values. There are 23 8 potential test
configurations, C1,, C8.
C1
C2
C3
C4
C5
C6
C7
C8
13
Set of potential 2-way interactions
P1
P1
P2
P3
P1
P2
P2
P3
P3

• There are potential interactions
• Coverage measure percentage of these
  interactions included.

14
Interactions included in a configuration
A
E
C
One test configuration...
covers 3 interactions.
15
Interaction Coverage Goal
using a subset of all test configurations.
Goal cover all interactions
16
Selection of Configurations
A
E
C
A
C
A
E
E
C
A
F
C
A
D
A
F
F
C
A
E
D
A
F
D
B
C
B
E
E
D
B
E
C
B
D
B
F
F
D
B
F
C
B
E
D
B
F
D
Degree 2 coverage 3 / 12 25
Degree 3 coverage 1 / 8 12.5
17
Selection of Configurations
A
E
C
A
C
A
E
E
C
A
F
C
A
D
A
F
F
C
A
E
D
A
F
D
B
C
B
E
E
D
B
E
C
B
D
B
F
F
D
B
F
C
B
E
D
B
F
D
Degree 2 coverage 6 / 12 50
Degree 3 coverage 2 / 8 25
18
Selection of Configurations
A
E
C
A
C
A
E
E
C
A
F
C
A
D
A
F
F
C
A
E
D
A
F
D
B
C
B
E
E
D
B
E
C
B
D
B
F
F
D
B
F
C
B
E
D
B
F
D
Degree 2 coverage 9 / 12 75
Degree 3 coverage 3 / 8 37.5
19
Selection of Configurations
A
E
C
A
C
A
E
E
C
A
F
C
A
D
A
F
F
C
A
E
D
A
F
D
B
C
B
E
E
D
B
E
C
B
D
B
F
F
D
B
F
C
B
E
D
B
F
D
Degree 2 coverage 12 / 12 100
Degree 3 coverage 4 / 8 50
20
Choosing the degree of coverage

• In one experiment, covering 2 way interactions
  resulted in the following average code coverage
• 93 block coverage.
• 83 decision coverage.
• 76 c-use coverage.
• 73 p-use coverage.
• Source Cohen, et al, The combinatorial design
  approach to automatic test generation, IEEE
  Software, Sept. 1996.
• Another experience report investigating
  interactions among 2-4 components
• Dunietz, et al, Applying design of experiments
  to software testing, Proc. Of ICSE 97.

21
Section summary

• We have defined how to measure coverage of
  potential system interactions.
• Strategy for choosing test configurations
• Maximize coverage of interaction elements for a
  given degree.
• Choose interaction degree based on
• Degree of interaction risk that can be tolerated.
• Test budget constraints.

22
Objectives

• Develop a measure that shows how well potential
  interactions are covered by a set of test
  configurations.
• Determine how to achieve the highest interaction
  coverage with the fewest number of configurations.

23
But, how did we know what to select?

• Four ways to find the configurations
• Look in a reference book ?
• Constraint-based approach.
• Heuristics
• Combinatorial designs

24
Comparison of Methods

• Look in a reference book
• CRC Handbook of Combinatorial Designs
• The specific number of parameters and values in
  your situation has to be in the book!
• Constraint-based approach
• Requires solution to 0,1 integer program
• Gives optimal solution
• NP-complete problem not feasible for realistic
  situations

25
Method 2 Constraint-based approach
ACE
ACF
ADE
ADF
BCE
BCF
BDE
BDF

• Minimizex1  x2  x3  x4  x5  x6  x7  x8
  xi ? 0,1

26
Solution using freewareinteger/linear program
solver.

• process killed after 6.5 hours
• result at time the process was terminated.

27
Method 3 Heuristics

• The one shown here, In-Parameter Order, is due to
  Lei and Tai.
• Start with the first two parameters, and generate
  all possible combinations
• Then, add a third parameter. For the test
  configurations already generated, choose values
  for the new parameter so that the largest number
  of interactions are covered.
• If there are interactions left uncovered at the
  end of this process, add additional
  configurations
• Repeat until all parameters have been added.

28
Example

• 3 parameters
• First parameter can take values A or B
• Second parameter can take values J or K
• Third parameter can take values X, Y, or Z
• 12 possible configurations

29
Start with 2 parameters
Interaction elements
Test configurations
A
X
J
X
A
J
A
Y
J
Y
A
K
A
Z
J
Z
B
J
B
X
K
X
B
K
B
Y
K
Y
B
Z
K
Z
30
Add spaces for next parameter
Interaction elements
Test configurations
A
J
A
J
A
X
J
X
A
K
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
31
Try values to see which covers the most
interactions
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 2 interactions covered
32
Interaction elements
Test configurations
A
J
Y
A
J
A
X
J
X
A
K
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 2 interactions covered
Y 2 interactions covered
33
Interaction elements
Test configurations
A
J
Z
A
J
A
X
J
X
A
K
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 2 interactions covered
Y 2 interactions covered
Z 2 interactions covered
34
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
Choose X
35
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
X
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 1 interaction covered
36
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Y
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 1 interaction covered
Y 2 interactions covered
37
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Z
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 1 interaction covered
Y 2 interactions covered
Z 2 interactions covered
38
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Y
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
Choose Y
39
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Y
A
K
A
Y
J
Y
B
J
X
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 1 interaction covered
40
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Y
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 1 interaction covered
Y 2 interactions covered
41
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Y
A
K
A
Y
J
Y
B
J
Z
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 1 interaction covered
Y 2 interactions covered
Z 2 interactions covered
42
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Y
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
Choose Y
43
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Y
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
X
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 2 interactions covered
44
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Y
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
Y
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 2 interactions covered
Y 0 interactions covered
45
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Y
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
Z
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 2 interactions covered
Y 0 interactions covered
Z 2 interactions covered
46
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Y
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
X
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
Choose X
47
Additional configurations needed to complete
coverage
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Y
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
X
B
K
B
X
K
X
A
Z
B
Y
K
Y
B
Z
K
Z
48
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Y
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
X
B
K
B
X
K
X
A
Z
B
Y
K
Y
B
Z
B
Z
K
Z
49
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Y
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
X
B
K
B
X
K
X
A
Z
J
B
Y
K
Y
B
Z
B
Z
K
Z
50
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Y
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
X
B
K
B
X
K
X
A
Z
J
B
Y
K
Y
B
Z
K
B
Z
K
Z

• 6 out of 12 configurations are selected for
  testing

51
Method 4

• Use principles of combinatorial designs, used in
  the design of statistical experiments.

52
Statistical Experimental Design

• Used in many fields other than computer science.
• Objective
• Create an experiment to test several factors at
  once.
• Individual effect of each factor level.
• Interactions among factors.
• Minimize the number of experiments needed.
• Facilitate result analysis.
• Application to software system testing
• Can be used in any situation where there are a
  set of parameters, each of which have a set of
  (discrete) values.

53
Orthogonal Arrays

• Orthogonal arrays are a standard construction
  used for statistical experiments.
• Strength 2 select any 2 columns and all ordered
  pairs occur the same number of times.
• Covers all 2-way interactions.
• Orthogonal arrays can be found in statistical
  tables, or can be calculated from algebraic
  finite fields.
• Many existence restrictions.

54
An orthogonal Array
A
E
C
A
F
D
B
F
C
B
E
D

• This is a strength 2 orthogonal array because you
  can choose any 2 columns and every combination of
  values occurs the same number of times (once)

55
Adaptation to Software Testing

• If we are testing strictly for software
  interactions, we can use a different experimental
  design.
• Why?
• If each component has been tested on its own, we
  can eliminate the need for testing for the effect
  of a single parameter.
• Software testing yields a discrete test result
  (pass, fail, ), rather than requiring
  analysis of real valued results.

56
Adaptation to Software Testing

• The result
• Each interaction needs to be covered at least
  once, instead of the same number of times.
• In many cases, fewer configurations are required.
• The construction for this purpose is called a
  covering array.

57
Covering Arrays

• Definition of covering array
• If we select d columns, all possible ordered
  d-tuples occur at least once.
• A covering array of strength d will ensure than
  any consistent interaction problem caused by a
  particular combination of two elements is
  detected. Problems caused by an interaction of d
  1 (or more) elements may not be detected.
• Choosing the degree of coverage defines the
  trade-off in risk we are making
• Fewer test configurations versus potential
  uncovered interactions.

58
A covering array
Four parameters, two values for each.
A
E
C
G
B
F
D
G
B
E
D
H
B
F
C
H
A
F
D
H

• Note that in some cases, a specific interaction
  appears several times, but all interactions
  appear at least once.

59
Recursive Covering Array Construction

• Problem
• If the range of values is 1, , n, then an
  orthogonal array can handle at most n 1
  parameters.
• Existence of suitable orthogonal arrays.
• Goal
• Generate covering arrays for problems of
  arbitrary size.
• Method
• Assemble larger covering array from smaller
  building blocks.
• No heuristics.

60
Constructing larger covering arrays
With 3 values per parameter, an orthogonal array
can handle up to 4 parameters. Generate
orthogonal array from finite field permutations.
61
Duplicate orthogonal array three times for 12
parameters
62
Check coverage so far For the first column...
63
we have pair-wise coverage with the rest of the
orthogonal array (by definition)
64
but we also have pair-wise coverage with the
corresponding columns in the duplicate orthogonal
arrays.
65
We have also covered the (x,x) combinations in
the identical columns, but not the (x,y)
combinations.
66
Use reduced array, which covers only the
(x,y) combinations
This is the original orthogonal array, with the
first 3 rows andfirst column removed.
67
and add new configurations to cover
missing combinations.
68
This covers the remaining combinations for the
first column.
1
2
3
1
2
3
69
The same scheme applies to other columns.
All pair-wise combinations have now been covered
70
O
O
O

• O orthogonal array
• R4 columns are duplicated 4 times consecutively

R4
71
Multistage Covering Arrays
O
O
O
O
O
O
O
O
O
R4
R4
R4
R12
72
Some results

• Results from the recursive construction example
• 13 components, 3 types for each component.
• Number of potential test configurations
  1,594,323.
• Number of degree 2 interaction elements 702.
• Minimum number of configurations for 100
  coverage of degree 2 interaction elements 15.
• Achieving coverage of interaction elements
  results in a number of test configurations that
  is proportional to.
• The logarithm of the number of components.
• The maximum number of types for any component,
  raised to the power of the interaction coverage
  degree.

73
Number of configurations neededfor degree 2
coverage
values
74
Comparison of Methods

• Recursive construction from small covering arrays
• Fast, next best method for number of
  configurations
• Inflexible only degree 2, and prefers to have
  the same number of values for each parameter
• In-parameter order (IPO) heuristic
• Slow, generates large number of configurations
• Extensible to higher degrees of coverage
• Can start from pre-existing set of configurations
• We will show an additional improvement to the IPO
  method in this talk.

75
Our Example Again
Payment Server
BusinessWeb Server
Client browser
Type MasterCard,Visa,American Express
Business Database
Type Netscape,Explorer
Type WebSphere,Apache,.NET
Type DB/2,Oracle,Access
76
Test configurations for degree 2 coverage
Data Base
Configuration
Payment
Web Server
Browser
1
MasterCard
DB/2
Netscape
WebSphere
2
Oracle
Netscape
Visa
Apache
3
Access
Netscape
AmEx
.NET
4
Access
Explorer
Visa
WebSphere
5
DB/2
AmEx
Apache
Explorer
6
Oracle
Explorer
MasterCard
.NET
7
Oracle
(dont care)
AmEx
WebSphere
8
Access
MasterCard
Apache
(dont care)
9
DB/2
Visa
.NET
(dont care)
77
What was not considered?
Payment Server
BusinessWeb Server
Client browser
Type MasterCard,Visa,American Express
Business Database
Type Netscape,Explorer
Type WebSphere,Apache,.NET
Type DB/2,Oracle,Access

• The topology of the network

78
Covering Arrays on Graphs

• Modify our goal
• Choose any 2 columns and every combination of
  values that are connected by a graph edge occurs
  at least once
• Why?
• Even degree 2 coverage of all pairs of parameters
  may still yield a large number of configurations
  beyond the range of a test budget.
• We make the assumption that components directly
  connected to each other are more likely to have
  interaction problems than ones that are
  indirectly connected.

79
Modifying the heuristic

• The original heuristic starts with 2 parameters,
  and then adds one parameter at a time.
• Coverage of interaction elements with all
  previous parameters is taken into account when
  choosing new values.
• Two changes
• When adding a new parameter, cover interaction
  elements only for previous parameters that are
  connected.
• Make sure first two parameters are connected!

80
Same case as before,with connectivity graph

• 3 parameters
• First parameter can take values A or B
• Second parameter can take values J or K
• Third parameter can take values X, Y, or Z
• Connectivity graph

1
2
3
81
Start with 2 connected parameters
Interaction elements
Test configurations
A
X
J
X
A
J
A
Y
J
Y
A
K
A
Z
J
Z
B
J
B
X
K
X
B
K
B
Y
K
Y
B
Z
K
Z
82
Add spaces for next parameter
Interaction elements
Test configurations
A
J
A
J
A
X
J
X
A
K
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
83
Try values to see which covers the most
interactions
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 1 interaction covered
84
Interaction elements
Test configurations
A
J
Y
A
J
A
X
J
X
A
K
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 1 interaction covered
Y 1 interaction covered
85
Interaction elements
Test configurations
A
J
Z
A
J
A
X
J
X
A
K
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 1 interaction covered
Y 1 interaction covered
Z 1 interaction covered
86
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
Choose X
87
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
X
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 1 interaction covered
88
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Y
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 1 interaction covered
Y 1 interaction covered
89
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
Z
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 1 interaction covered
Y 1 interaction covered
Z 1 interaction covered
90
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
X
A
K
A
Y
J
Y
B
J
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
Choose X
91
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
X
A
K
A
Y
J
Y
B
J
X
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 0 interactions covered
92
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
X
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 0 interactions covered
Y 1 interaction covered
93
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
X
A
K
A
Y
J
Y
B
J
Z
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 0 interactions covered
Y 1 interaction covered
Z 1 interaction covered
94
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
X
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
Choose Y
95
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
X
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
X
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 0 interactions covered
96
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
X
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
Y
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 0 interactions covered
Y 1 interaction covered
97
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
X
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
Z
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
X 0 interactions covered
Y 1 interaction covered
Z 1 interaction covered
98
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
X
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
Y
B
K
B
X
K
X
B
Y
K
Y
B
Z
K
Z
Choose Y
99
Additional configurations needed to complete
coverage
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
X
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
Y
B
K
B
X
K
X
Z
J
B
Y
K
Y
B
Z
K
Z
100
Interaction elements
Test configurations
A
J
X
A
J
A
X
J
X
A
K
X
A
K
A
Y
J
Y
B
J
Y
B
J
A
Z
J
Z
B
K
Y
B
K
B
X
K
X
Z
J
B
Y
K
Y
Z
K
B
Z
K
Z
101
Algorithm Implementation

• Read in an adjacency matrix
• Store as edge list
• Use a breadth first search to order the
  parameters
• The order of the parameters would not otherwise
  be significant, so we can choose a useful order.
• When determining list of interaction elements to
  cover, choose only pairs that are connected.

102
Results (1)

• 10 values, each with 10 parameters
• Original heuristic produces 170 configurations in
  3308 ms.

103
Example 2 Connectivity graph
4
number of values for this parameter
5
4
5
3
run 1
9
6
run 2
15
7
2
104
Results (2)

• Change from run 1 to 2
• Parameter with degree 2 in connectivity graph
  increased from 9 to 15 values

105
TConfig Test configuration generator
Try it www.site.uottawa.ca/awilliam/TConfig.jar
106
Comments from testers

• Inclusion of specified or existing tests
• I have some recommended configurations and I
  want to be sure they are included.
• I already have a collection of tests that are
  working fine, and have been developed at great
  expense. How do I determine which additional
  tests need to be added to bring the test suite to
  a certain level of interaction coverage?
• Changes in set of allowed parameters and values
• What additional configurations are required if
• a new component is added to the system?
• a new version of an existing component becomes
  available?

107
The road ahead

• Now that we know which test configurations to
  select, is there a way to automatically modify
  test scripts for each configuration?
• Design model to support deployment to multiple
  environments
• Build deployment into modelling notations?
• Virtual machines for execution?

108
Thank you!

• This presentation is available at
• http//www.site.uottawa.ca/awilliam