Modeling the Covering Test Problem

1
Modeling the Covering Test Problem

• Brahim Hnich, Steven Prestwich, and Evgeny
  Selensky
• Cork Constraint Computation Center
• UCC

Supported by SFI
2
Overview

• Motivation
• Problem description
• CP models
• Naïve model
• Alternative model
• Integrated model
• Symmetry
• Results
• Local search approach

3
Motivation

• Software and hardware testing important activity
  in product development process
• E.g., s/w testing consumes up to 50 of overall
  cost
• Exhaustive testing is infeasible
• Large number of possible test cases
• 10 on/off switches lead to 210 possible
  combinations

4
Motivation

• To be practical
• small number of possible test cases such that
• A small number (3) on/off switches get exercised
  in all possible 23 combinations
• The question becomes
• How many test cases do we need?
• This problem is an instance of the covering
  arrays problem

5
Covering array

• A covering array ltt,k,ggt of size b is
• An kxb array (k vectors of length b)
• Each entry is drawn from 0,,g-1
• The projection of any t coordinates contains all
  gt possibilities

6
Covering array example

• A covering arraylt3,5,2gt of size 10 is

7
Covering array example

• A covering arraylt3,5,2gt of size 10 is

8
Covering array example

• A covering arraylt3,5,2gt of size 10 is

9
Covering array example

• A covering arraylt3,5,2gt of size 10 is

10
Covering array example

• A covering arraylt3,5,2gt of size 10 is

11
Covering array example

• A covering arraylt3,5,2gt of size 10 is

12
Covering array example

• A covering arraylt3,5,2gt of size 10 is

13
Covering array example

• A covering arraylt3,5,2gt of size 10 is

14
Covering array example

• A covering arraylt3,5,2gt of size 10 is

15
CP models naïve model
k
Mb,k
b
16
Coverage constraints

• For every t columns c1,,ct
• For every row r
• For each possible combination p
• Introduce a Boolean variable B
• Use reification constraints to state that
  combination p is true iff Boolean variable B is
  set to 1
• B1 iff (Mr,c1 0 and Mr,c20 and . And
  Mr,ct0)
• Enforce that the sum of the Booleans associated
  with each combination p is at least 1
• Too many auxiliary variables
• Inefficient propagation

17
Row and column symmetry

• The naïve matrix model has both row and column
  symmetry
• Any permutation of the rows does not affect the
  set of solutions
• Any permutation of the columns does not affect
  the set of solutions
• Reduce this symmetry double-lex
• Lexicographically order the rows
• Lexicographically order the columns

18
CP models alternative model
k choose t
b
19
Coverage constraints

• For each column
• State a global cardinality constraint enforcing
  that each value occurs between 1..b-2t 1 times
• Global constraint
• Efficient propagation
• Effective pruning

20
Intersection constraints

• Because our columns intersect
• E.g. lt1,2,3gt and lt1,2,4gt
• Their values should agree

0 1 2 3 4 5 6 7
0 1 2 3 4 5 6 7
21
Symmetry in alternative matrix

• Rows are still symmetric
• Lexicographic ordering the rows
• However, columns are no longer symmetric
• But, the column symmetry in the naïve model is
  now a complex symmetry
• A complex combination of partial column symmetry
  and value symmetry
• Difficult to break!

22
Comparison

• Naïve model
• Coverage constraints?
• Symmetry
• Row symmetry ?
• Column symmetry ?

• Alternative model
• Coverage constraints?
• Intersection constraints ?
• Symmetry
• Row symmetry?
• Complex symmetry ?

23
Integrated model

• Naïve model
• Coverage constraints?
• Symmetry
• Row symmetry ?
• Column symmetry ?

• Alternative model
• Coverage constraints?
• Intersection constraints ?
• Symmetry
• Row symmetry?
• Complex symmetry ?

24
Integrated model

• Naïve model
• Coverage constraints?
• Symmetry
• Row symmetry ?
• Column symmetry ?

• Alternative model
• Coverage constraints?
• Intersection constraints ?
• Symmetry
• Row symmetry?
• Complex symmetry ?

Channelling constraints ? Increased number of
variables ?
25
Integrated model

• Naïve model
• Symmetry
• Row symmetry ?
• Column symmetry ?

• Alternative model
• Coverage constraints?

Channelling constraints ? Increased number of
variables ?
26
CP models channelling
27
Experiments

• Moderate sized instances
• Naïve model is very inefficient
• The integrated model is al least one order of
  magnitude faster than the alternative model
• New results for t3 and k lt12
• Larger Instances
• Beyond 2000 variables
• Amount of search is computationally prohibitive

28
A local search approach

• An almost SAT-encoding of the integrated model
• The upper bound on the coverage constraints are
  hard to express (implied constraint)
• No symmetry breaking
• No at most one clauses
• details in the paper

29
Local search results

• Run model on Walksat
• reproduced bounds found by solver
• Solves several larger instances with better
  results than the state of the art
• No proof of optimality though!

30
Conclusion

• The covering test problem
• A core problem in combinatorial software testing
• CP approach
• Integrated model was able to prove optimality and
  improve bounds for moderate size instances
• Local search approach
• Improved bounds for larger instances
• Future
• Push the models further to solver larger
  instances