Modeling the Covering Test Problem Brahim Hnich, Steven Prestwich, and Evgeny Selensky Cork Constraint Computation Center UCC Supported by SFI

Overview Motivation Problem description CP models Naïve model Alternative model Integrated model Symmetry Results Local search approach

Motivation Software and hardware testing: important activity in product development process E.g., s/w testing consumes up to 50% of overall cost Exhaustive testing is infeasible Large number of possible test cases 10 on/off switches lead to 2 10 possible combinations

Motivation To be practical small number of possible test cases such that A small number (3) on/off switches get exercised in all possible 2 3 combinations The question becomes How many test cases do we need? This problem is an instance of the covering arrays problem

Covering array A covering array of size b is: An kxb array (k vectors of length b) Each entry is drawn from {0,…,g-1} The projection of any t coordinates contains all g t possibilities

Covering array: example A covering array of size 10 is:

Covering array: example A covering array of size 10 is:

Covering array: example A covering array of size 10 is:

Covering array: example A covering array of size 10 is:

Covering array: example A covering array of size 10 is:

Covering array: example A covering array of size 10 is:

Covering array: example A covering array of size 10 is:

Covering array: example A covering array of size 10 is:

Covering array: example A covering array of size 10 is:

CP models: naïve model {0,…,g-1} … … … b k M[b,k]

Coverage constraints For every t columns c1,…,ct For every row r For each possible combination p Introduce a Boolean variable B Use reification constraints to state that combination p is true iff Boolean variable B is set to 1 B=1 iff (M[r,c1]= 0 and M[r,c2]=0 and …. And M[r,ct]=0) Enforce that the sum of the Booleans associated with each combination p is at least 1 Too many auxiliary variables Inefficient propagation

Row and column symmetry The naïve matrix model has both row and column symmetry Any permutation of the rows does not affect the set of solutions Any permutation of the columns does not affect the set of solutions Reduce this symmetry: double-lex Lexicographically order the rows Lexicographically order the columns

CP models: alternative model …… 0{0..7}{0,…,g t -1}……… 0{0..7} ……… 1 ……… 2 ……… 3 ……… 4 ……… 5 ……… 6 ……… 7 ….…… b k choose t

Coverage constraints For each column State a global cardinality constraint enforcing that each value occurs between 1..b-2 t +1 times Global constraint Efficient propagation Effective pruning

Intersection constraints Because our columns intersect E.g. and Their values should agree

Symmetry in alternative matrix Rows are still symmetric Lexicographic ordering the rows However, columns are no longer symmetric But, the column symmetry in the naïve model is now a complex symmetry A complex combination of partial column symmetry and value symmetry Difficult to break!

Comparison Naïve model Coverage constraints  Symmetry Row symmetry Column symmetry Alternative model Coverage constraints Intersection constraints  Symmetry Row symmetry Complex symmetry 

Integrated model Naïve model Coverage constraints  Symmetry Row symmetry Column symmetry Alternative model Coverage constraints Intersection constraints  Symmetry Row symmetry Complex symmetry 

Integrated model Naïve model Coverage constraints  Symmetry Row symmetry Column symmetry Alternative model Coverage constraints Intersection constraints  Symmetry Row symmetry Complex symmetry  Channelling constraints  Increased number of variables 

Integrated model Naïve model Symmetry Row symmetry Column symmetry Alternative model Coverage constraints Channelling constraints  Increased number of variables 

CP models: channelling Column Column 1Column 2Column

Experiments Moderate sized instances Naïve model is very inefficient The integrated model is al least one order of magnitude faster than the alternative model New results for t=3 and k <12 Larger Instances Beyond 2000 variables Amount of search is computationally prohibitive

A local search approach An almost “SAT-encoding” of the integrated model The upper bound on the coverage constraints are hard to express (implied constraint) No symmetry breaking No at most one clauses … details in the paper

Local search: results Run model on Walksat reproduced bounds found by solver Solves several larger instances with better results than the state of the art No proof of optimality though!

Conclusion The covering test problem A core problem in combinatorial software testing CP approach Integrated model was able to prove optimality and improve bounds for moderate size instances Local search approach Improved bounds for larger instances Future Push the models further to solver larger instances