1. Systematic Software Testing Techniques: Combinatorial Testing
Dr. Renée Bryce
Professor
University of North Texas

2. Presentation outline Introductions Motivation
Background on combinatorial testing
Examples, definitions, fun facts, and an example algorithm Try it! Create your own Pairwise Combinatorial Test Suite on paper
Sneak peek at the ACTS tool (You’ll use this tool when Professor Wong returns next week.)
BONUS QUESTION! Do you have an examples of where you could apply CT?
Prioritized Combinatorial Testing
Example algorithm
Example applications
Test Suite Reduction
Definitions and examples
Try it! Reduce a test suite using the HGS algorithm.

3. Introductions
Briefly share your experiences with Software Testing

4. Motivation Costs of software defects
Software defects cost ~$59 billion per year [1]
One contributor to software defects
Many system components are tested individually, but often unexpected interactions between components cause failures.
[1] National Institute of Standards and Technology, The Economic Impacts of Inadequate Infrastructure for Software Testing, U.S. Department of Commerce, May 2002.

5. Combinatorial TestING Example
Hardware
Operating
System
Network
Connection
Memory PC
Windows XP
Dial-up
64MB
Laptop
Linux
DSL
128MB
PDA
FreeBSD
Cable
256MB
Four factors (components) have three levels (options) each
Sample test
Test
No.
Hardware
Operating
System
Network
Connection
Memory 1 PC
Windows XP
Dial-up
64MB
Pairs covered
1. (PC, Windows XP)
4. (Windows XP, Dial-up)
2. (PC, Dial-up)
5. (Windows XP, 64MB)
3. (PC, 64MB)
6. (Dial-up, 64MB)

6. Combinatorial Testing Example
Hardware
Operating
System
Network
Connection
Memory PC
Windows XP
Dial-up
64MB
Laptop
Linux
DSL
128MB
PDA
FreeBSD
Cable
256MB
Four factors (components) have three levels (options) each
Test No.
Hardware
Operating
System
Network
Connection
Memory 1 PC
Windows XP
Dial-up
64MB 2
Laptop
FreeBSD
DSL
64MB 3
PDA
Linux
Cable
64MB 4
Laptop
Windows XP
Cable
128MB 5
Laptop
Linux
Dial-up
256MB 6 PC
Linux
DSL
128MB 7
PDA
Windows XP
DSL
256MB 8 PC
FreeBSD
Cable
256MB 9
PDA
FreeBSD
Dial-up
128MB

7. Fun Facts Combinatorial testing has been used in several fields:
Agriculture
Combinatorial chemistry
Genomics
Software/hardware testing
Study of Mozilla web browser found 70% of defects with 2-way coverage; ~90% with 3-way; and 95% with 4-way. [Kuhn et. al., 2002]
Combinatorial testing of 109 software-controlled medical devices recalled by US FDA uncovered 97% of flaws with 2-way coverage; and only 3 required higher than 2. [Kuhn et. al., 2004]

8. Covering arrays
A covering array, , is an N x k array. In every N x t subarray, each t-tuple occurs at least λ times. In our application, t is the strength of the coverage of interactions, k is the number of components (factors), and v is the number of options for each component (levels). In all of our discussions, we treat only the case when λ = 1, (i.e. that every t-tuple must be covered at least once).
Behind the scenes this combinatorial object is constructed to represent interaction test suites
No efficient exact method is known
Mathematicians and Computer Scientists have offered solutions from different view points. Their solutions have been measured by time to generate test suites and sizes of test suites.

9. Exercise 1
1. List all of the 2way combinations (pairs) for this input:
Hint: Example pairs are (0,3) (0,4) (0,5)…. (8,11)
2. Create a combinatorial test suite for the input above. No credit will be given for an exhaustive test suite.

10. Perceived benefits of greedy algorithms
Mathematical
Greedy
Search
Size of test suites
Accurate on special cases; but not as general as needed
Reasonably accurate
Most accurate (if given enough time)
Time to generate tests
Yes
Often time consuming (for good results)
Seeding/ Constraints
Difficult to accommodate seeds/constraints

11. History of One-test-at-a-time Greedy Algorithms
Pros
First tool to generate test suites (based on covering arrays)
Cons
Produces different test suites to the same inputs
Slow
Deterministic
Particularly good for mixed-level inputs
Faster
Overly large test suites for fixed-level inputs
Competitive results
Logarithmic guarantee on the size of test suites
AETG
TCG
DDA

12. Sample sizes of test suites
[1] R. Bryce, C.J. Colbourn. A Density-Based Greedy Algorithm for Higher Strength Covering Arrays, Journal of Software Testing, Verification, and Reliability, (March 2009), 19(1):37-53.
[2] R. Bryce, C.J. Colbourn. The Density Algorithm for Pairwise Interaction Testing, Journal of Software Testing, Verification and Reliability, (August 2007), 17(3):
*(Citeseer impact ranking of STVR: .36)

13. Framework of One-row-at-a-time Greedy Methods
Defines commonalities that all “one-row-at-a- time” greedy algorithms have in common
A process provides statistical feedback on the impact of different decisions that can be made in the framework
Experiments explore several thousand instantiations of the framework and provide a requisite of knowledge

14. Framework for greedy methods
Layer one: test suite repetitions
Layer two: multiple candidates
Layer three: factor ordering*
Layer four: level selection*
Pairs left to cover:
Test No.
Hardware
Operating
System
Network
Connection
Memory 48 1 PC
Windows XP
Dial-up
64MB 42 2 PC
Linux
DSL
128MB 36 3 PC
FreeBSD
Cable
256MB 4 ? ? ? ?

15. Framework for greedy methods
Layer one: test suite repetitions
Layer two: multiple candidates
Layer three: factor ordering*
Layer four: level selection*
Factor 1 ? PC
Laptop
PDA
Involved in 9 uncovered pairs
Involved in 9 uncovered pairs
Involved in no uncovered pairs

16. Framework for greedy methods
Layer one: test suite repetitions
Layer two: multiple candidates
Layer three: factor ordering*
Layer four: level selection*
Laptop ? PC
Laptop
PDA
Involved in 9 uncovered pairs
Involved in 9 uncovered pairs
Involved in no uncovered pairs

17. Framework for greedy methods
Layer one: test suite repetitions
Layer two: multiple candidates
Layer three: factor ordering*
Layer four: level selection*
Laptop ?
Factor 3
Dial-up
DSL
Cable
Will cover 1 new pair
Will cover 1 new pair
Will cover 1 new pair

18. Framework for greedy methods
Layer one: test suite repetitions
Layer two: multiple candidates
Layer three: factor ordering*
Layer four: level selection*
Laptop ?
DSL
Dial-up
DSL
Cable
Will cover 1 new pair
Will cover 1 new pair
Will cover 1 new pair

19. Framework for greedy methods
Layer one: test suite repetitions
Layer two: multiple candidates
Layer three: factor ordering*
Layer four: level selection*
Laptop
Factor 2
DSL ?
WinXP
Linux
FreeBSD
Will cover 2 new pair
Will cover 1 new pair
Will cover 2 new pairs

20. Framework for greedy methods
Layer one: test suite repetitions
Layer two: multiple candidates
Layer three: factor ordering*
Layer four: level selection*
Laptop
WinXP
DSL ?
WinXP
Linux
FreeBSD
Will cover 2 new pair
Will cover 1 new pair
Will cover 2 new pairs

21. Framework for greedy methods
Layer one: test suite repetitions
Layer two: multiple candidates
Layer three: factor ordering*
Layer four: level selection*
Laptop
WinXP
DSL
Factor 4
64MB
128MB
256MB
Will cover 2 new pair
Will cover 2 new pair
Will cover 3 new pairs

22. Framework for greedy methods
Layer one: test suite repetitions
Layer two: multiple candidates
Layer three: factor ordering*
Layer four: level selection*
Laptop
WinXP
DSL
64MB
64MB
128MB
256MB
Will cover 2 new pair
Will cover 2 new pair
Will cover 3 new pairs

23. Framework for greedy methods
Layer one: test suite repetitions
Layer two: multiple candidates*
Layer three: factor ordering
Layer four: level selection
Number of newly covered pairs
3 candidate rows
PDA
Linux
Cable
128MB 6 5
PDA
Win XP
Cable
64MB 5
PDA
Linux
Cable
128MB

24. Framework for greedy methods
Layer one: test suite repetitions*
Layer two: multiple candidates
Layer three: factor ordering
Layer four: level selection
Test suite A
Test suite B
Test suite C
Select the smallest test suite generated.

25. Framework experiment - ANOVA results for several inputs
[1] R.Bryce, C.J. Colbourn, M.B. Cohen. A Framework of Greedy Methods for Constructing Interaction Tests. The 27th International Conference on Software Engineering (ICSE), St. Louis, Missouri. (May 2005), pp (13% acceptance rate)

26. Sneak Peek at ACTS
Professor Wong will cover this next week, but here is a quick glance.
Learn more at:
Download it at**:
** If you use ACTS at your company, please NIST for permission first

27. Test Suite Prioritization and Reduction by Combinational-based Criteria
Dr. Renée Bryce
Associate Professor
University of North Texas

28. Prioritized combinatorial testing
What if parts of a system are more important to test earlier?
What if a tester learns during testing and wants to regenerate a test suite with new priorities?
What if a tester has time to run pair-wise coverage and time to run some three-way tests?

29. A variation of the covering array
An ℓ-biased covering array is a covering array CA(N; 2, k, v) in which the first rows form tests whose total benefit is as large as possible. That is, no CA(N; 2, k, v) has rows that provide larger total benefit.
Input
3 factors with varying numbers of associated levels (options) and weights assigned to each level

30. Algorithm Walkthrough
Input
3 factors with varying numbers of associated levels (options) and weights assigned to each level
Larger weight means higher priority should be given to testing earlier!
Step 1 – Calculate Factor Interaction Weights.
Factors will be assigned values in order of “highest priority”

31. Algorithm Walkthrough
Input
Input has been processed
Factor interaction weights have been calculated (to determine order to assign level values to factors).
Step 2 – Calculate Factor-Level Interaction Weights to select the level that covers the most uncovered weighted density.
For a factor, i, and a level, ℓ, that number of levels for a factor is called , and the factor interaction weight is called
Factor-Level Interaction Weight =
Remember, the factor interaction weight was 1.3 for factor 2.

32. Algorithm Walkthrough
For a factor, i, and a level, ℓ, that number of levels for a factor is called , and the factor interaction weight is called
Factor-Level Interaction Weight =
Step 2 (continued) - Calculate Factor-Level Interaction Weights to select the level that covers the most uncovered weighted density.
Weight between two levels divided by max factor-interaction weight
Weight between two levels divided by max factor-interaction weight + Weight of level multiplied by weight of level (of fixed factor)
Weights of levels between each multiplied

33. Sample Results Input Output
[1] R. Bryce, C.J. Colbourn. Prioritized Interaction Testing for Pairwise Coverage with Seeding and Avoids, Information and Software Technology Journal (IST, Elsevier), (October 2006), 48(10):

34. Test Suite Prioritization
Problem: Given T, a test suite, Π, the set of all test suites obtained by permuting the tests of T, and f, a function from Π to the set of real numbers, the problem is to find π∈Π such that ∀π′ ∈Π,f(π)≥f(π′). In this definition, Π refers to the possible prioritizations of T and f is a function applied to evaluate the orderings.
Convert the web logs to a user-session-based test suite.
POST/GET requests
The test suite is large!

35. Case Study: Prioritizing User-session-based Test Suites
Methodology: Convert web logs to user-session-based test suites, prioritize, and write to an XML format.
Algorithm: Efficiently prioritize by combinatorial-based coverage for large test suites
Empirical Studies: Families of empirical studies to analyze the effectiveness in relation to characteristics of the applications and test suites.

36. Research Questions
Can we improve the rate of fault detection for user-session-based testing with new prioritization criteria?
Which techniques are valuable in different scenarios?
i.e.: tests have a high/low Fault Detection Density
i.e.: predicted distribution of faults (deemed from prior versions of the software)
Can we fine tune the criteria?
i.e.: cost-based prioritization

37. Prioritization Metrics
Test length based on number of base requests:
order by the number of HTTP requests in a test case
Frequency-based prioritization:
order such that test cases that cover most frequently accessed pages/sequence of pages are selected for execution before test cases that exercise the less frequently accessed pages/sequences of pages.
Unique coverage of parameter-values:
order tests to cover all unique parameter-values as soon as possible
2-way parameter-value interaction coverage:
order tests to cover all pair-wise combinations of parameter-values between pages as soon as possible
Test length based on number of parameter-value:
order by number of parameter-values used in a test case
Random:
randomly permute the order of tests

38. Empirical Studies TerpCalc, TerpPaint, Terp Spreadsheet, and TerpWord
Online Bookstore
Online Course Project Manager (CPM)
Online Conference Management System
SchoolMate
Online Music Store
Metavist (sponsored by USDA)

39. Results for an on-line system for a Course Project Manager and 890 Test Cases
[1] R. Bryce, S. Sampath, A. Memon. Developing a Single Model and Test Prioritization Strategies for Event-Driven Software, Transactions on Software Engineering, (January 2011), 37(1):48-64.

40. Sample results
% of
test
suite
run
Most
frequent
requests
No. of
Requests
Long to
short
Short to
long
PVs
Long to
Short
1-way
2-way
Random 10
85.28
78.17
75.14
83.53
16.38
83.79
83.72
48.63 20
88.52
80.34
77.76
88.77
25.6
87.78
90.8
57.55 30
89.4
81.77
80.27
26.44
91.54
91.72
64.51 40
89.86
84.58
81.39
92.71
28.76
94.79
95.64
69.19 50
91.04
85.58
82.95
30.33
73.03 60
91.58
87.14
84.44
94.26
34.64
75.37 70
92.1
87.74
85.15
39.15
77.37 80
92.35
88.27
86.21
39.58
78.24 90
92.37
88.3
86.31
42.18
94.99
78.45
100
92.45
88.36
86.35
43.09
78.49

41. Test prioritization by interaction coverage
Test suite prioritization
GUI-based testing

42. Empirical Studies Traffic Collision Avoidance System GUI-based Testing
Word processor
Spreadsheet
Paint
Calculator
Web application Testing
Bookstore
Course Project Manager
Conference Management Software

43. Transfer of Work
Potential users that have contacted NIST to use our tool:
AT&T
BBC (for Winter Olympics website)
Booz Allen Hamilton
Angel.com
U.S. Army Test and Evaluation Research Laboratory, Aberdeen Proving Ground
A2Z Research and Development
NASA IV&V

44. Transfer of Work
[1] S. Sampath, R. Bryce, S. Jain, S. Manchester. A Tool for Combinatorial-based Prioritization and Reduction of User-Session-Based Test Suites, International Conference on Software Maintenance (ICSM) - Tool Demonstration Track, Williamsburg, VA, September 2011

45. Test Suite Reduction
Problem: Given T, a test suite with test cases { }, a set of testing requirements,{ }, that must be satisfied to provide the desired test coverage of the program, and subsets { } of T, one associated with each of the s such that any one of the tests belonging to Ti satisfies . Find the minimal cardinality subset of T that exercises all of the requirements exercised by the original test suite T.
Original Test Suite
(Too large for our budget)
Reduced Test Suite
(Fits into budget) 45

46. Reduction Example Original Test Suite
{t1,t2,t3,t4}
Requirements covered by the test suite
{r1,r2,r3,r4}
Problem: Reduce the test suite such that it maintains coverage of these requirements

47. Test Suite Reduction Example
Requirement Ti 1
{t3,t4} 2
{t4} 3
{t1, t2, t3, t5} 4
{t1, t2, t3}
In this example, there are three possible solutions. We highlighted 1: {t1, t4}

48. Test Suite Reduction Example
HGS Algorithm
Select t5 since it is of cardinality 1
2. Consider unmarked Tis of cardinality 2, that is T4, T5, T6.
Select the test that appears in the most Tis. That is a tie between t1 and t6.
Break the tie by examining sets of cardinality (m+1), That is sets T3 and T7.
Break tie between t1 and t6, by selecting t1 as it is in T3. T
Requirement Ti 1
{t1, t5} 2
{t5} 3
{t1, t2, t3} 4
{t3, t6} 5
{t1, t4} 6
{t1, t6} 7
{t3, t4, t7} 8
{t2, t3, t4, t7}
3. T4 is of cardinality 2, there is a tie between t3 and t6, so we look at sets of size cardinality (m+1). We choose t3.
Reduced Test Suite:
{t5,
t1,
t3}

49. Exercise T Requirement Ti 1 {t1, t5} 2 {t5} 3 {t1, t2, t3} 4 {t3, t6}
Reduce this test suite using the HGS algorithm: T
Requirement Ti 1
{t1, t5} 2
{t5} 3
{t1, t2, t3} 4
{t3, t6} 5
{t1, t4} 6
{t1, t6} 7
{t3, t4, t7} 8
{t2, t3, t4, t7}

50. Test Suite Reduction Example
HGS Algorithm
Select t5 since it is of cardinality 1
2. Consider unmarked Tis of cardinality 2, that is T4, T5, T6.
Select the test that appears in the most Tis. That is a tie between t1 and t6.
Break the tie by examining sets of cardinality (m+1), That is sets T3 and T7.
Break tie between t1 and t6, by selecting t1 as it is in T3. T
Requirement Ti 1
{t1, t5} 2
{t5} 3
{t1, t2, t3} 4
{t3, t6} 5
{t1, t4} 6
{t1, t6} 7
{t3, t4, t7} 8
{t2, t3, t4, t7}
3. T4 is of cardinality 2, there is a tie between t3 and t6, so we look at sets of size cardinality (m+1). We choose t3.
Reduced Test Suite:
{t5,
t1,
t3}