A Measure for Component Interaction Test Coverage

1
A Measure for Component Interaction Test Coverage

• Alan W. WilliamsRobert L. Probert
• School of Information Technology and
  EngineeringUniversity of OttawaOttawa ON Canada
• awilliam,bob_at_site.uottawa.ca
• www.site.uottawa.ca/awilliam,bob

2
Outline

• Context testing of heterogeneous systems.
• Test configurations.
• Interaction elements.
• Coverage of interaction elements.
• Test configuration generation.
• Some results.

3
Objectives

• First
• Develop a measure that shows how well potential
  interactions are covered by a set of test
  configurations.
• Second
• Determine how to achieve the highest interaction
  coverage with the fewest number of configurations.

4
Testing of Component-Based Systems
Payment Server
BusinessWeb Server
Client browser
Type MasterCard,Visa,American Express
Business Database
Type Netscape,Explorer
Type iPlanet,Domino, Apache
Type DB/2,Oracle,Access

• The goal verify reliability and
  interoperability by testing as many system
  configurations as possible, given time and budget
  constraints.

5
Testing in the Presence ofPolymorphism and
Inheritance
Method invocation
class hierarchy
...

• Sender, receiver, and message parameters could be
  instances of various classes within the class
  hierarchy.

6
A test configuration is
Explorer
WebSphere
Visa
DB/2

• For each component, one of the legal types of
  that component is selected.

• The entire e-commerce application test suite is
  run for each configuration that is selected.

7
An interaction element is
Explorer
Visa

• Choose a subset of the parameters
• The size of the subset is the interaction degree.

• Choose specific values for the parameters.

8
Example

• Suppose that we have three parameters.
• For each parameter, there are two possible
  values.
• Types are
• A, B for parameter 1.
• J, K for parameter 2.
• Y, Z for parameter 3.
• Degree of interaction coverage is 2.
• We want to cover all potential 2-way interactions
  among parameter values.

9
Set of all possible test configurations
Three parameters, two values for each.
There are 23 8 possible test configurations.
10
Set of all possible degree 2interaction elements
There are ( ) ? 2 2 12 possible interaction
elements.
3
2
Coverage measure percentage of interaction
elements covered.
11
Test configurations assets of interactions
A
J
Y
One test configuration...
covers 3 possible interaction elements.
12
Interaction test coverage goal
using a subset of all test configurations.
Goal cover all interaction elements
13
Selection of test configurations forcoverage of
interaction elements
Interaction elements
Test configurations
Degree 2 coverage 3 / 12 25
Degree 3 coverage 1 / 8 12.5
14
Selection of test configurations forcoverage of
interaction elements
Interaction elements
Test configurations
A
J
A
Y
J
Y
A
J
Z
A
K
A
Z
J
Z
A
K
Y
B
J
B
Y
K
Y
A
K
Z
B
K
B
Z
K
Z
B
Y
J
B
Z
J
K
Y
B
Degree 2 coverage 6 / 12 50
K
Z
B
Degree 3 coverage 2 / 8 25
15
Selection of test configurations forcoverage of
interaction elements
Interaction elements
Test configurations
A
J
A
Y
J
Y
A
K
A
Z
J
Z
B
J
B
Y
K
Y
B
K
B
Z
K
Z
Degree 2 coverage 9 / 12 75
Degree 3 coverage 3 / 8 37.5
16
Selection of test configurations forcoverage of
interaction elements
Interaction elements
Test configurations
A
J
A
Y
J
Y
A
K
A
Z
J
Z
B
J
B
Y
K
Y
B
K
B
Z
K
Z
Degree 2 coverage 12 / 12 100
Degree 3 coverage 4 / 8 50
17
Choosing the degree of coverage

• In one experiment, covering 2 way interactions
  resulted in the following average code coverage
• 93 block coverage.
• 83 decision coverage.
• 76 computational-use (c-use) coverage.
• 73 predicate use (p-use) coverage.
• Source Cohen, et al, The combinatorial design
  approach to automatic test generation, IEEE
  software, Sept. 1996.
• Another experience report investigating
  interactions among 2-4 components
• Dunietz, et al, Applying design of experiments
  to software testing, proc. Of ICSE 97.

18
The Minimum Set Cover Problem
A set X
A collection T of subsets of X
A selection S from the collection T

• Objective cover all elements in set X with the
  smallest number of subsets in the collection
  S.
• This is a well-known NP-complete problem.

19
Comparison with Minimum Set Cover Problem
Test configurations
Set to be covered
20
Comparison with Minimum Set Cover Problem
Collections of subsets
Set to be covered
21
Comparison with theMinimum Set Cover Problem

• Special properties of this collection of subsets
• All subsets have the same cardinality (3).
• Number of subsets in collection T not 2p for p
  parameters.
• Each element of the set X to be covered occurs in
  the same number (2) of sets in the collection T.
• Therefore, this problem is unlikely to be
  NP-complete.

22
Statistical Experimental Design

• Used in many fields other than computer science.
• Objective
• Create an experiment to test several factors at
  once.
• Individual effect of each factor.
• Interactions among factors.
• Minimize the number of experiments needed.
• Facilitate result analysis.
• Application to software system testing
• Can be used in any situation where there are a
  set of parameters, each of which have a set of
  (discrete) values.
• Assumption each parameter can be set
  independently.
• When there are dependencies, enumerate legal
  combinations.

23
Adaptation to Software Testing

• If we are testing strictly for software
  interactions, we can use a smaller experimental
  design.
• Why?
• If each component has been tested on its own, we
  can eliminate the need for testing for the effect
  of a single parameter.
• Software testing yields a discrete test result
  (pass or fail), rather than requiring result
  analysis of real valued results.
• The result
• Each interaction needs to be covered at least
  once, instead of the same number of times.
• Fewer configurations are required.
• The construction for this purpose is called a
  covering array.

24
Covering Arrays

• Definition of covering array
• If we select d parameters, all possible ordered
  d-tuples occur at least once.
• A covering array of strength d will ensure than
  any consistent interaction problem caused by a
  particular combination of two elements is
  detected. Problems caused by an interaction of d
  1 (or more) elements may not be detected.
• Choosing the degree of coverage defines the
  trade-off in risk we are making
• Fewer test configurations versus potential
  uncovered interactions.

25
A Large Covering Array
1
1
1
1
1
1
1
1
1
1
1
1
1

• This is a strength 2 covering array for 13
  parameters, each with 3 values.
• 15 test configurations.

1
2
2
2
1
2
2
2
1
2
2
2
1
1
3
3
3
1
3
3
3
1
3
3
3
1
2
1
2
3
2
1
2
3
2
1
2
3
1
2
2
3
1
2
2
3
1
2
2
3
1

2
3
1
2
2
3
1
2
2
3
1
2

A single test configuration
3
1
3
2
3
1
3
2
3
1
3
2
1
3
2
1
3
3
2
1
3
3
2
1
3

3
3
2
1
3
3
2
1
3
3
2
1

1
Each grid entry represents a specific parameter
value
2
3
Parameter
1
2
3
26
Some results

• Results from the previous covering array example
• 13 components, 3 types for each component.
• Number of potential test configurations
  1,594,323.
• Number of degree 2 interaction elements 702.
• Minimum number of configurations for 100
  coverage of degree 2 interaction elements 15.
• Achieving coverage of interaction elements
  results in a number of test configurations that
  is proportional to.
• The logarithm of the number of components.
• The maximum number of types for any component,
  raised to the power of the interaction coverage
  degree.

27
Number of configurations to coverinteraction
elements of degree 2
28
Our example again...
Payment Server
BusinessWeb Server
Client browser
Type MasterCard,Visa,American Express
Business Database
Type Netscape,Explorer
Type iPlanet,Domino,Apache
Type DB/2,Oracle,Access
29
Test configurations for degree 2 coverage
Data Base
Configuration
Payment
Web Server
Browser
1
MasterCard
DB/2
Netscape
iPlanet
2
Oracle
Netscape
Visa
Domino
3
Access
Netscape
AmEx
Apache
4
Access
Explorer
Visa
iPlanet
5
DB/2
AmEx
Domino
Explorer
6
Oracle
Explorer
MasterCard
Apache
7
Oracle
(dont care)
AmEx
iPlanet
8
Access
MasterCard
Domino
(dont care)
9
DB/2
Visa
Apache
(dont care)
30
TConfig Test Configuration Generator
Demo
31
Summary

• We have defined how to measure coverage of
  potential system interactions.
• The problem has been compared to a well-known NP
  complete problem, but is distinguishable from it.
• Strategy for choosing test configurations
• Maximize coverage of interaction elements for a
  given degree.
• Choose interaction degree based on
• Degree of interaction risk that can be tolerated.
• Test budget constraints.

32
Further work

• Comparison with other problems based on integer
  programming or constraints.
• Initial results generation of configurations is
  slower, and only small problems can be handled
• No correspondence to an NP-complete problem.
• Improved algorithms for generation of covering
  arrays
• Parameters have widely varying numbers of values.
• Construction of covering arrays for strength ? 3.
• Continuation of partially constructed arrays.