Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Pairwise Testing Comes of Age George B. Sherwood, Ph.D. Thursday May 8, 1:30 p.m. T14 Pairwise Testing

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Pairwise Testing Comes of Age

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Évariste Galois French mathematician, studied abstract algebra & group theory Radical Republican during the reign of Louis Philippe Short life with tragic ending: 1829 Father committed suicide Failed entry to École Polytechnique 1830 Expelled from École Normale for criticizing director 1831 Imprisoned for wearing illegal uniform 1832 Rejected by Stéphanie- Felice du Motel Shot in duel at age 20

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. 21 st Century Testing Increasing dependency on software, networks and distributed applications Increasing expectation and need for quality Facts of life: More features and complexity Shorter development cycles More reuse of components

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Why Pairwise Testing? Statistical coverage to find faults ̶ Improved quality Small number of test cases ̶ Reduced cost

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Pairwise Testing Example Test 3 new applications on a networked computer 4 test configuration factors with 3 values 3 4 =81 configurations in total – too many 3 configurations cover all values individually – but only a few interactions

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Pairwise Testing Includes all pairs of interactions between test factors Provides a small set of test cases – a practical alternative to testing all combinations Started almost 20 years ago Increasingly popular way to improve quality and reduce cost

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Galois Field Example Modulo 3 arithmetic – GF(3) Usual + or x, divide by 3 & take remainder Use GF(3) tables to make orthogonal array

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Orthogonal Array OA(9;3 3 ) Replace each element in the x table with the corresponding column from the + table

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Orthogonal Array OA(9;3 4 ) Add one column which pairs a different value with every 3 rows

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Networked Computer Example 4 test configuration factors with 3 values Replace numbers in OA(9;3 4 ) template with test factor values

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Networked Computer Example Only 9 configurations are needed to cover all 54 pairs of values using OA(9;3 4 )

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Design of Experiments Ronald A. Fisher analyzed data from crop yield experiments Developed new statistical methods to study experimental data Pioneered the mathematical principles of experimental design

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Orthogonal Arrays Orthogonal Array designs are used in agricultural & manufacturing experiments, clinical trials, market research… Results can show effects of factors and how they interact Calyampudi Radhakrishna Rao introduced Orthogonal Arrays

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. OAs for Manufacturing Genichi Taguchi adapted Fisher’s experimental design to improve quality of manufactured goods Used OAs to select test parameters in creative ways Worldwide influence in improving manufactured quality

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. OAs for Software Test Keizo Tatsumi, Robert Mandl & others applied these ideas to software testing Taguchi visited Bell Labs and collaborated with Madhav Phadke Orthogonal Array Test System (OATS) offered statistical coverage where exhaustive testing was not possible

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. StarLAN and StarGROUP Client installation test 101 combinations of PCs and OS versions 15 network cards Tried to use OATS to reduce number of test configurations OATS could not handle our complex configuration constraints correctly Generated impossible tests, as in previous example

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Networked Computer Example Two of the 9 configurations from OA(9;3 4 ) are invalid

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. What to do? Skip the bad cases? Then for each case there is no coverage for 5 good pairs, along with the invalid one Replace values ‘Linux’, ‘OS X’ or ‘Int Explorer’ to avoid bad pairs? Still miss good pairs Solution: Computer search for small number of cases to cover all allowed pairs.

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Networked Computer Example 11 configurations cover 52 allowed pairs

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Constrained Array Test System Search among allowed test cases to handle constraints Find next “best” test case to minimize uncovered pairs Secondary result: fewer test cases than OATS

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Types of Arrays

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Early Tools

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Pairwise Selection of Input Data Identify equivalence classes Analyze functional requirements, use cases Partition factor value combinations for equivalent processing and expected results Generate test cases for different classes separately All pairs from each class go to same resulting state Example: Don’t mix error combinations with normal combinations

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Uncle Katzendogg’s Guaranteed Long-term Weather Forecasts

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Weather Forecast Example 4 test factors: Month, Day, Year, Zip code 3 sets of test cases leading to different states: 1.Valid dates with valid zip codes ̶ Forecast returned 2.Invalid dates with valid zip codes ̶ Date error returned 3.Valid dates with invalid zip codes ̶ Zip code error returned

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Weather Forecast Valid Input Factor Values

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Weather Forecast Valid Input Test Cases

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Weather Forecast Invalid Date Factor Values

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Weather Forecast Invalid Date Test Cases

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Weather Forecast Invalid Zip Code Test cases Factor values

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Pairwise Testing Today Test demands continue to grow Pairwise testing is a popular response for effective coverage with a limited number of test cases Improved training is needed, e.g. to handle constraints, equivalence classes Careless use has led to controversy

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Pairwise.org Tools List – part 1

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Pairwise.org Tools List – part 2

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Bach & Schroeder “Pairwise Testing: A Best Practice that Isn’t” Useful criticism of careless pairwise testing MS Word options example –

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Pairwise Testing Efficiency Pairwise generators are more efficient than random test case selection Significant improvements in pairwise efficiency are possible Practitioners need pairwise testing to meet modern testing challenges

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. How do Searches Work? Uncovered pairs Factor values ? Test cases Go to the space of all possible test cases Look for small number of test cases (short path) to minimize the uncovered pairs

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Search Variations Several algorithms for different tradeoffs Test practitioner goals: Few test cases Quick responses Practicality – mixed values, constraints Researcher goals: Fewer test cases More factors New methods, better solutions

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Finding the Minimal Set of Test Cases is Hard ? Mixed covering array constructions can complement searches using less processing Typically searches are CPU bound Improvement diminishes with processing time

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Change Mixed CA Values Without Searching Example: Change OA(9;3 4 ) to CA(12; ) Reduce the number of values in two of the factors Increase the number of values in one factor

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. OA(9;3 4 ) to CA(9; ) Replace all the 2s in any 2 factors with unassigned (*) values

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. CA(9; ) to CA(12; ) Add 3 rows with 3s to one of the factors with 3 values For each of the other factors, pair its values with the 3s in the new rows

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. More CA Factors from Products and Searches Product sequence: OA(9;3 4 ), CA(15;3 15 ), CA(21;3 57 ), … Nurmela’s tabu search: CA(15;3 20 ) Products using tabu search result: CA(15;3 20 ), CA(21;3 74 ), CA(27;3 282 ), … Combined methods give better results

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Optimal Mixed CAs & More Factors Add n test cases to OA(25;5 6 ) to get optimal array Each factor of 5 values can be replaced by the indicated factors without additional test cases Example: Replace 3 factors in CA(35; ) to get CA(35; )

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Mixed Covering Arrays Compared with OA templates More choices for numbers of values Fewer test cases typically Compared with greedy search Faster responses (less processing) Fewer test cases when CAs are optimal Search needed for constraints

Testcover.com Copyright © 2008 Testcover.com, LLC. All rights reserved. Future Pairwise Trends Platforms: Faster, distributed processing Tool Software: Smarter searches, less processing Fewer test cases Improved test efficiency Faster responses What-if test designs, test changes Use with agile, incremental cycles