1. A Composition-Based Method for Combinatorial Test Design Eitan Farchi, IBM Research, Haifa Seri Khoury, Technion Amir Shwartz and Anna Zamansky, University of Haifa

2. Composition Model Model Module A Module B Test Plan Test Plan
In this work we considered sequential scenarios, where
One realization of this idea is via composition, which naturally arises in sequential scenarios, in which the output of one system fragment is used as an input of another fragment
Test Plan
Test Plan

3. Example: Unix Shell Commands
ls - listing the contents of a directory
Many optional parameters:
-laxo lists files with permissions
-l shows hidden files
-a displays files in a line format …

4. Example: Unix Shell Commands
grep – searching for lines containing a match to a given pattern.
grep -w ”ctd” myfile.txt
search the file myfile.txt for lines containing ”ctd”,
selecting only those lines that form whole words

5. Example: Unix Shell Commands
Consider now their composition
ls -l | grep -w ”ctd”
search the file names of the current directory for ”ctd”

6. Example: Unix Shell Commands
We have:
Models of ls and grep
Test plans for ls and grep
Idea of “Composition” :
Reuse resources created for the individual system calls for testing the
sequential scenarios.
How expensive is it?

7. Formulate a composition-oriented CTD problem
Main contribution
Formulate a composition-oriented CTD problem
Propose an efficient algorithm to solve it

8. Making a distinction between input and output
Test t: (a1, a2, a3, b1, b3)
t: A → B A B

9. A test plan P: A → B is a triple (E,R,T):
E is the system model: a set of executable tests t: A → B
R is a set of coverage requirements
T is the set of tests: a subset of E
P is complete if every coverage requirement is covered by some test t ∊T.

10. Composition of models E1: A → B E2: B → C A B B C Open for Permission
File
read
user
small
write
admin
large
File
Result
small
success
large
failure A C
E1○ E2: A → C
Open for
Permission
read
user
write
admin
Result
success
failure

11. T1 ○ T2: (read, user, success)
Composition of tests
open-for permission file file status
T1 ○ T2: (read, user, success)
T1: A → B T2: B → C
(read, user, small) (small, success)

12. T1 ○ T2: not defined T1: A → B T2: B → C
Composition of tests
T1 ○ T2: not defined
T1: A → B T2: B → C
(read, user, small) (large, success)
open-for permission file file status

13. Can we build composable test plans? Complete test plan P1 = (E1,R1,T1)
The problem
P1 and P2 are not composable: T1 ○ T2 may not be a complete test plan for E1 ○ E2!
Can we build composable test plans?
Complete test plan P1 = (E1,R1,T1)
Complete test plan P2 = (E2,R2,T2)

14. The CTD problem
Given a system model E and a set of coverage requirements R, find a complete and sufficiently small test plan P = (E, R, T).
Greedy algorithm based on BDDs by Segall, Tzoref-Brill & Farchi (implemented in IBM Functional Coverage Unified Solution (FoCuS))

15. The Composition-Oriented CTD problem
Given system models E1: A → B and E2: B → C and corresponding coverage requirements R1 and R2, find complete and sufficiently small test plans for them which are also composable.
The main advantage
in thinking in terms of composition is that it facilitates reuse of
resources in CTD: given composable test plans P1 and P2 for
two different system modules, we can ”get for free” a test plan
for their sequential composition if we solve the CTD problem
in a slightly more sophisticated way.

16. Outputs: composable and complete test plans for these models
The algorithm
An extension of the greedy algorithm based on BDDs by Segall, Tzoref-Brill & Farchi (implemented for FoCuS)
Inputs: two system models and their corresponding coverage requirements
Outputs: composable and complete test plans for these models
Following [5], R contains a set of tuples of parameters
to be covered (as opposed to tuples of values as defined in
Definition 3). The algorithm maintains for each tuple t 2 R a
BDD uncov(t), which captures the assignments to t that are not
covered by the tests that were selected so far. In each iteration,
the algorithm uses the ChosenTest procedure (Algorithm 2)
to find a test from the system model E1, covering as many
tuples as possible in R1. After finding such test, the algorithm
uses the InnerJoin procedure (Algorithm 3) to try and find a
matching from E2 that covers as many tuples as possible in R2.
If such matching was found, it is added also to T2 and cleaned
from its uncov BDD (line 10,11). so the algorithm would not
consider it further, If there is no such matching, the algorithm
looks for a random matching from E2 (line 13)

17. Evaluation

18. Presented a formulation of a version of CTD problem relevant when composition of different systems components is to be tested.
Presented a BDD-based algorithm for solving the new composition-oriented problem
Preliminary evaluation shows that algorithm performs as well the FoCuS BDD-based CTD algorithm, but we get composability “for free”
Summary

19. Future Work Evaluation using large-size examples
How to systematicall increase reusability of CTD artifact?
What other operations other than composition are naturally defined on test plans?
Thank you for your attention!
Future
Work