1. Combinatorial Testing Using Covering Arrays - Going Beyond Pairwise Testing Raghu Kacker, NIST Yu Lei, UTA 5/15/06

2. Combinatorial Testing Using Covering Arrays 2 Outline  Introduction  The IPO Strategy  FireEye: An N-Way Testing Tool  Related Work  Conclusion

3. Combinatorial Testing Using Covering Arrays 3 Why Software Testing?  Modern society is increasingly dependent on the quality of software systems.  A NIST study reports that software failure costs the US economy billions of dollars every year.  Testing (or dynamic analysis) is the most widely used approach to ensuring software quality  Other approaches like static analysis and formal verification are more difficult to apply and do not seem to scale

4. Combinatorial Testing Using Covering Arrays 4 Testing Process The testing process consists of three stages:  Test Generation: Generate test data  For model-based testing, a model (or abstraction) of the system has to be built.  Test Execution: Test setup and the actual test runs  Test Evaluation: Check if the output is in line with expectations

5. Combinatorial Testing Using Covering Arrays 5 All About Trade-Off  Testing is labor intensive and can be very costly  estimated to often consume more than 50% of the development cost  Exhaustive testing is often impractical due to resource constraints  From a certain perspective, testing is basically about making a good trade-off between test effort and quality assurance.

6. Combinatorial Testing Using Covering Arrays 6 Combinatorial Testing  Generates tests from an input parameter model by combining the values of the parameters.  Requires lightweight specification and no knowledge about the implementation structure  Can be virtually applied to any software and at different levels of abstraction  Can be implemented as a push-button feature  Usually performed to achieve t-way coverage, i.e. guarantees to cover every t-way interaction  Motivation: Not every parameter contributes to every fault  Can dramatically reduce the number of tests while still preserving important fault detection capabilities.

7. Combinatorial Testing Using Covering Arrays 7 The ASTUCA Project  Develop new methods and tools for efficient t-way testing for up to 6-way testing  Create new algorithms for efficient test set construction  Provide adequate support for parameter relations and constraints  Explore integration with other software testing tools  Conduct empirical evaluation of t-way testing in an industrial setting  A collaborative effort among the following institutions:  The US National Institute of Standards and Technology  George Mason University  The University of Texas at Arlington

8. Combinatorial Testing Using Covering Arrays 8 State-of-the-Art  Existing work has mainly focused on pairwise testing  Many failures are caused by the interaction involving more than two parameters  For certain software, pairwise testing discovers a relatively low percentage of faults e.g., For the RAX in NASA Deep Space 1 mission, pairwise testing only discovers 54 percent of interface faults, and 47 percent of engine faults.  Increased coverage leads to a higher level of assurance  Many applications, e.g., security protocols, have strict requirements on test coverage

9. Combinatorial Testing Using Covering Arrays 9 Technical Challenges  The computational complexity for t-way testing grows rapidly as the value of t increases  New algorithms must strike a balance between the time and space requirements and the optimality of the resulting test sets  The number of tests also grows rapidly as the value of t increases  Impractical to manually execute and inspect the results of a large number of test runs  Test generation, test execution, and test evaluation must integrate together to enable test automation

10. Combinatorial Testing Using Covering Arrays 10 Terminology  N-Way Test set -> N-Way Covering array  Tests -> Rows  Parameters -> Factors or Columns  Values -> Levels

11. Combinatorial Testing Using Covering Arrays 11 Outline  Introduction  The IPO Strategy  FireEye: An N-Way Testing Tool  Related Work  Conclusion

12. Combinatorial Testing Using Covering Arrays 12 The Framework (1)  Builds a t-way test set in an incremental manner  A t-way test set is first constructed for the first t parameters,  Then, the test set is extended to generate a t-way test set for the first t + 1 parameters  The test set is repeatedly extended for each additional parameter.  Two steps involved in each extension for a new parameter:  Horizontal growth: extends each existing test by adding one value of the new parameter  Vertical growth: adds new tests, if necessary

13. Combinatorial Testing Using Covering Arrays 13 The Framework (2) Strategy In-Parameter-Order begin /* for the first t parameters p 1, p 2, …, p t */ T := {(v 1, v 2, …, v t ) | v 1, v 2, …, v t are values of p 1, p 2, …, P k, respectively} if n = t then stop; /* for the remaining parameters */ for parameter p i, i = t + 1, …, n do begin /* horizontal growth */ for each test (v 1, v 2, …, v i-1 ) in T do replace it with (v 1, v 2, …, v i-1, v i ), where v i is a value of p i /* vertical growth */ while T does not cover all the interactions between p i and each of p 1, p 2, …, p i-1 do add a new test for p 1, p 2, …, p i to T; end

14. Combinatorial Testing Using Covering Arrays 14 Example (1) Consider a system with the following parameters and values:  parameter A has values A1 and A2  parameter B has values B1 and B2, and  parameter C has values C1, C2, and C3

15. Combinatorial Testing Using Covering Arrays 15 Example (2) A B A1 B1 A1 B2 A2 B1 A2 B2 A B C A1 B1 C1 A1 B2 C2 A2 B1 C3 A2 B2 C1 ABC A1B1C1 A1B2C2 A2B1C3 A2B2 C1 A2B1C2 A1B2C3 Horizontal GrowthVertical Growth

16. Combinatorial Testing Using Covering Arrays 16 Comparison to AETG (1)  A commercial tool developed by Telcordia, and protected by a US patent  Starts with an empty set and adds one (complete) test at a time  Each test is locally optimized to cover the most number of missing pairs:  Generate a random order of the parameters  Use a greedy algorithm to construct a test that covers the most uncovered pairs  Repeat the above two steps for a given number of times (suggested 50), and select the best one

17. Combinatorial Testing Using Covering Arrays 17 Comparison to AETG (2) ABC A1B1C1 ABC A1B1C1 A1B2C2 ABC A1B1C1 A1B2C2 A2B1C3 A2B2 C1 A2B1C2 A1B2C3 Adds the 1st testAdds the 2nd testAdds the last test ABCABC

18. Combinatorial Testing Using Covering Arrays 18 IPO vs AETG  IPO is deterministic, whereas AETG is inherently non-deterministic  IPO has a lower order of complexity, both in terms of time and space, than AETG  IPO constructs a test set one parameter at a time and in a more incremental nature.  The results generated by IPO are still competitive to those generated by AETG.  IPO is more flexible than AETG  IPO can take a test set for a subsystem (i.e., for a subset of parameters) and then extend it to a complete set for the entire system

19. Combinatorial Testing Using Covering Arrays 19 Outline  Introduction  The IPO Strategy  FireEye: A Prototype Tool  Related Work  Conclusion

20. Combinatorial Testing Using Covering Arrays 20 Major Features  Uses Java as the programming language  Relatively easier to program, leading to reduced development time and ease of maintenance  Supports the concept of “write-once-run-everywhere”  Data structures are carefully designed to optimize the runtime performance  A hierarchical structure is used to manage the possible interactions  Allows an incomplete test set to be extended to a complete one  Add new tests or parameters, if necessary, to achieve t- way coverage

21. Combinatorial Testing Using Covering Arrays 21 Initial Results (1) [System] Name: Test Configuration for TCAS [Parameter] -- only compare with MINSEP and MAXALTDIFF Cur_Vertical_Sep : 299, 300, 601 High_Confidence : TRUE, FALSE Two_of_Three_Reports_Valid : TRUE, FALSE -- Low and High, only compare with Other_Tracked_Alt Own_Tracked_Alt : 1, 2 Other_Tracked_Alt : 1, 2 -- only compare with OLEV Own_Tracked_Alt_Rate : 600, 601 Alt_Layer_Value : 0, 1, 2, 3 -- compare with each other (also see NOZCROSS) and with ALIM Up_Separation : 0, 399, 400, 499, 500, 639, 640, 739, 740, 840 Down_Separation : 0, 399, 400, 499, 500, 639, 640, 739, 740, 840 Other_RAC : NO_INTENT, DO_NOT_CLIMB, DO_NOT_DESCEND Other_Capability : TCAS_TA, OTHER Climb_Inhibit : TRUE, FALSE

22. Combinatorial Testing Using Covering Arrays 22 Initial Results (2) All the experiments are performed on a desktop with 1.2GHZ CPU and 1GB memory.

23. Combinatorial Testing Using Covering Arrays 23 Outline  Introduction  The IPO Strategy  FireEye: An N-Way Testing Tool  Related Work  Conclusion

24. Combinatorial Testing Using Covering Arrays 24 Classification  Search-Based methods that are mainly developed by computer scientists  AETG (from Telcordia), TCG (from JPL/NASA), DDA (from ASU), PairTest  Algebraic methods that are mainly developed by mathematicians  Orthogonal Arrays  Recursive Construction

25. Combinatorial Testing Using Covering Arrays 25 Orthogonal Arrays  Orthogonal arrays can be constructed very fast and are always optimal  Any extra test will cause a pair to be covered for more than once  However, there are several limitations:  Orthogonal arrays do not always exist  Every parameter must have the same number v of values  Existing methods often require v be a prime power.  Every t-way interaction must be covered at the same number of times

26. Combinatorial Testing Using Covering Arrays 26 Recursive Construction  Covering arrays are a more general structure, which requires every t-way interaction be covered at least once  Constructing a covering array from one or more covering arrays with smaller parameter sets  Recursive construction can be fast, but it also has restrictions on the number of parameters and the domain sizes

27. Combinatorial Testing Using Covering Arrays 27 Search-Based vs Algebraic Methods  Search-based methods:  Advantages: no restrictions on the input model, and very flexible, e.g., relatively easier to support parameter relations and constraints  Disadvantages: explicit search takes time, the resulting test sets are not optimal  Algebraic methods:  Advantages: very fast, and often produces optimal results  Disadvantages: limited applicability, difficult to support parameter relations and constraints  The advantages and disadvantages of the two types of methods seem to complement with each other

28. Combinatorial Testing Using Covering Arrays 28 Outline  Introduction  The IPO Strategy  FireEye: An N-Way Testing Tool  Related Work  Conclusion

29. Combinatorial Testing Using Covering Arrays 29 Conclusion  Combinatorial testing is a well-defined problem and has been used widely in practice.  The IPO strategy has a lower order of complexity than AETG, and still produces competitive results.  Algebraic methods, if applicable, are fast and can be optimal, whereas search-based algorithm are very flexible.  Going beyond 2-way testing presents challenges and opportunities to the area of combinatorial testing.

30. Combinatorial Testing Using Covering Arrays 30 References 1. Boroday S. Y. and Grunskii I. S., “Recursive generation of locally complete tests,” Cybernetics and Systems Analysis 28 (1992), 20-25. 2. K. A., Bush, “Orthogonal arrays of index unity,” Annals of Mathematical Statistics, 23 (1952), 426-434. 3. D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, “The AETG System: An Approach to Testing Based on Combinatorial Design,” IEEE Transactions on Software Engineering, 1997, Vol. 23, No. 7.The AETG System: An Approach to Testing Based on Combinatorial Design 4. M. B. Cohen, C. J. Colbourn, P. B. Gibbons and W. B. Mugridge, “Constructing test suites for interaction testing,” In Proc. of the Intl. Conf. on Software Engineering, (ICSE 2003), 2003, pp. 38-48, Portland. 5. R. Kuhn, D. Wallace, A. Gallo, “Software Fault Interactions and Implications for Software Testing,” IEEE Transactions on Software Engineering, June 2004, Vol. 30, No. 6.Software Fault Interactions and Implications for Software Testing 6. Alan Hartman, Leonid Raskin, “Problems and algorithms for covering arrays,” Discrete Mathematics 284(1-3): 149-156 (2004)Leonid RaskinDiscrete Mathematics 284 7. Y. Lei and K. C. Tai, “In-parameter-order: a test generation strategy for pairwise testing,” Proceedings Third IEEE Intl. High-Assurance Systems Engineering Symosium., 1998, pp. 254-261.In-parameter-order: a test generation strategy for pairwise testing 8. K. C. Tai and Y. Lei, “A Test Generation Strategy for Pairwise Testing,” IEEE Transactions on Software Engineering, 2002, Vol. 28, No. 1.A Test Generation Strategy for Pairwise Testing