Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. The Bug Stops Here Improving Test Design with a Novel Test Case Generator Technical Presentation Salon 6 Wednesday May 18, 1:45 – 2:45

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. How to Spin a Web of Effective Test Cases Test case generator makes “all pairs” testing simpler, quicker, more practical New technology improves test case coverage and fault discovery Service continually integrates research advances to reduce number of test cases

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Testing Breakthrough Needed Increasing dependency on software, networks and distributed applications Increasing expectation and need for quality Facts of life: More features and complexity Shorter development cycles More reuse of components

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. This Presentation Describes approaches to software test case generation Test configurations System-level functional tests – integration tests, functional verification tests, regression tests etc. Does not address Unit testing (e.g. code coverage) Stability and load testing

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. This Presentation Explains how the Testcover.com Covering Test Case Generator Service improves test case coverage and fault discovery Shows how to use the service with examples Tells how to receive 1 month service free Is available on the web at Testcover.com

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Founded in 2003 to aid testers in the selection of efficient sets of test cases Covering Test Case Generator Service On the web at Testcover.com Technology based on: advances in covering array construction practical software engineering experience Testcover.com Dedicated to Excellence in Test Design

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example Test three new applications on a networked computer Test factors and their values:

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example A simple approach: One test configuration for each factor value Select a most common or default value for each factor Vary each factor over all its values while the other factors are set to their default values

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example A simple approach: How many test configurations? 12

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example A simple approach: Small number of test configurations All individual factor values are covered Many pairs of factor values are not: Application 2 with Cable Connection Firefox Browser with ME Operating System Application 3 with Medium Resolution More…

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example A better approach: Include all pairs of factor values Test all pair-wise interactions How many pairs of factor values? 101 How many test configurations? 16

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example Configuration Example How many pairs of factor values?

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example Configuration Example How many pairs of factor values?

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example Configuration Example How many pairs of factor values?

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example Configuration Example How many pairs of factor values?

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example Configuration Example How many pairs of factor values?

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example Configuration Example How many pairs of factor values?

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example Configuration Example How many pairs of factor values?

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example Configuration Example How many pairs of factor values?

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example Configuration Example How many pairs of factor values?

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example Configuration Example How many pairs of factor values?

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example 16 configurations cover all 101 pairs

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Combo Countdown Coverage metric Number of untested pairs after each test case Stops at 0 or the number of untested pairs when there are constraints Illustrates diminishing returns Supports risk assessment

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Test Definitions Test factor – any variable whose values are to be controlled during testing Test factor value – specific value taken by one of the test factors Test case – set of test factor values with one allowed value for each of the test factors Combination – association of some number of factor values (pairs, triples, etc.)

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Array Definitions Size – number of test cases, number of rows in array Degree – number of test factors, number of columns in array Order – number of values for a test factor Strength – size of combinations (2: pairs, 3: triples, etc.)

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Mixed arrays Mixed array has factors with different orders Example: factors with 5 values 3 factors with 4 values 2 factors with 3 values 1 factor with 2 values Degree = = 8 There is a array with strength 2 and size 25

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Early Approaches Orthogonal arrays used in agricultural and medical experiments Dr. Genichi Taguchi used orthogonal arrays to select test parameters Statistical coverage where exhaustive testing is not possible

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Real-world Constraints What if the applications in the configuration example are ported to Linux and Macintosh OSX? Internet Explorer is supported on Windows only There are Browser-Operating System pairs that must be excluded

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example The wrong answer:

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Constrained Array Test System Motivated by testing Local Area Network product configurations Developed in 1990 to handle constraints Employed greedy search algorithm Used internally at AT&T as best practice Described in STAR proceedings May 1994

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Current Challenges Current Challenges 1.Find small array quickly 2.Adapt to system constraints simply 3.Support multiple system states 4.Allow easy use Testcover.com addresses these challenges Let’s see how…

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Sample Arrays SizeArraySizeArraySizeArray

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Sample Mixed Arrays SizeDegreeArraySizeDegreeArray

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Challenge 1: Fewer Test Cases Small arrays result from multiple techniques Testcover.com continually integrates research advances to reduce the numbers of test cases New technology provides quicker results The results given here are subject to improvement without notice!

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Challenge 2: Handle Constraints Direct Product Block notation enables simple descriptions of complex constraints and dependencies Let’s see how DPB notation works with the Configuration Example

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example No constraints – one block

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Direct Product Block Notation Block represents all possible test cases for selected values of the test factors Multiple blocks represent all allowed test cases – as a set of products Simpler than enumerating all disallowed combinations

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Configuration Example Browser/OS constraints – two blocks

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Challenge 3: Multiple States Calendar example illustrates: Direct Product Block notation Constraints – different month lengths Multiple states using different partitions of test cases – Normal (good dates) Error handling (too long months)

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Calendar Example Request Part 1

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Calendar Example Request Part 2

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Calendar Example Results Part 1

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Calendar Example Results Part 2

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Calendar Example Results Part 3

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Challenge 4: Ease of Use Direct Product Block notation Simple to learn Flexible and efficient Internet-based service Easy access without special installation Independent of development environment Enhancements without upgrade work Practical for everyday use

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Challenge 4: Ease of Use Advanced Web Search example illustrates: Practical for real HTML form Direct Product Block notation Constraints – language button use Small number of test cases

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Advanced Web Search Example See HTML form

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Advanced Web Search Example Request part 1

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Advanced Web Search Example Request part 2

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Advanced Web Search Example Request part 3

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Advanced Web Search Example Array Degree pairs, 6 not covered (constraints) Size 63 (compared to 8 x 7 = 56)

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. How to Spin a Web of Effective Test Cases New technology improves test case coverage and fault discovery Service continually integrates research advances to reduce number of test cases Exclusive Direct Product Block notation simplifies use

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Exhibit at Booth 8 See these and other examples run Receive your brochure and gift Register for 1 free month of service

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. STAREAST 2005 First Month Free 4 Month Subscription for the Price of 3 For STAREAST 2005 Attendees Only Register at Booth 8 May 18 – May 19 Try Testcover.com Service Free for 1 Month Your Choice: Pay invoice for 4-Month Subscription Cancel On-line 1 Month Free Service either way – $100 Value

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Back-up Slides

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Test Case Objectives Run enough of the “right” test cases to exercise system interactions completely Avoid wasting time and money on unnecessary test cases Insure system behavior is thoroughly verified Allow for complex and changing system requirements

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Orthogonal Arrays & Covering Arrays Every pair of columns in an orthogonal array contains all pairs of values an equal number of times Every pair of columns in a covering array contains all pairs of values at least once Orthogonal arrays are covering arrays Both types can be mixed (different orders)

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Orthogonal Arrays & Covering Arrays Orthogonal arrays are balanced – suggest which factor values give best measured results Tuning complex systems Medical & agricultural trials Covering arrays are not required to be balanced – can be smaller for pass/fail type tests

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Covering Arrays vs. Random Values Goal: Validate operation quickly – integration, function, regression – Arrays provide coverage with few test cases Goal: Validate continuous operation – stability, load, stress – Random values can provide many different test cases

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Strength t arrays are defined similarly – Every set of t columns in an array contains all t-tuples of values (3-tuples, 4-tuples, etc.) Size (number of test cases) increases with strength – Minimum size is product of the t largest orders Example: arrays Strength:234 Minimum Size: Higher Strength Arrays

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Higher Strength Arrays Higher strength arrays can cover more combinations of factor values, with a cost of more test cases Generally finding higher strength arrays requires more processing See Sherwood, Martirosyan, and Colbourn, Sections 4 & 5.

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. RDF/XML Results The Resource Description Framework (RDF) is a general- purpose language endorsed by the World Wide Web Consortium (W3C) for representing information on the web. RDF is based on the idea of identifying things using Uniform Resource Identifiers (URIs) and describing resources in terms of simple properties. RDF represents simple statements about resources as a graph of nodes and arcs representing the resources, and their properties and values. RDF also provides an XML-based syntax (called RDF/XML) for recording and exchanging these graphs. In addition to HTML results, the Testcover.com test case generator provides RDF/XML results for use by other applications.

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Sources & Further Reading Companies Seek to Hold Software Makers Liable for Flaws, David Bank, The Wall Street Journal, page B1, February 24, To Darned Big to Test, Keith Stobie, ACM Queue, vol. 3, no. 1, pages 30-37, February Products of Mixed Covering Arrays of Strength Two, C. J. Colbourn, S. S. Martirosyan, G. L. Mullen, D. Shasha, G. B. Sherwood, and J. L. Yucas, preprint, Covering Arrays of Higher Strength From Permutation Vectors, G. B. Sherwood, S. S. Martirosyan, and C. J. Colbourn, preprint, On the Construction of Orthogonal Arrays and Covering Arrays Using Permutation Groups, George Sherwood, March June 2004.

Testcover.com Copyright © 2005 Testcover.com, LLC. All rights reserved. Sources & Further Reading Effective Testing of Factor Combinations, George Sherwood, Third International Conference on Software Testing, Analysis & Review, May 8-12, 1994, Washington, DC. Orthogonal Arrays Theory and Applications, A. S. Hedayat, N. J. A. Sloane, and John Stufken, 1999 Springer-Verlag. Quality Engineering Using Robust Design, M. S. Phadke, 1989, Prentice Hall. M. J. Suárez-Cabal and J. Tuya. Using an SQL Coverage Measurement for Testing Database Applications. ACM Software Engineering Notes. 29(6): , 2004.