1. Orthogonal Array Testing Strategy

2. Abstract
Orthogonal Array Testing Strategy (OATS) is a systematical, statistical way of testing pair-wise interactions by deriving suitable small set of test cases from a large number of scenarios. The testing strategy can be used to reduce the number of test combinations and provide maximum coverage with a minimum number of test cases. OATS utilizes an array of values representing variable factors that are combined pair-wise rather than representing all combinations of factors and levels.
Orthogonal Array Testing (OATS) is a testing approach that utilizes testing interactions in pairs (aka pair-wise), derived from the overall large picture of test scenarios identified. The expected results from OATS utilization is to identify a smaller number of test cases for text execution.

3. Abstract
Being intelligent about which test cases are selected can make the difference between “endlessly executing” over “executing a concise, defined set of test cases”. OATS steps involve identifying, determining, finding, mapping, choosing, and transcribing. As an end result, orthogonal array testing provides a means to select a test set that guarantees, creates, exercises, and is simplistic, deriving a suitable small set of test cases from a large number of scenarios with broad coverage.

4. Outline
Definition for Orthogonal Array Testing Strategy (OATS)
The OAT Theory
Simple OAT Example
Pairwise Combination of Parameters
OATS Example 2
Why Use Orthogonal Array Testing?
Advantages of OATS
Disadvantages of OATS

5. Selecting the Right Test Set with OATS Mistakes to Avoid
Outline
Selecting the Right Test Set with OATS
Mistakes to Avoid
Applying OATS manually
Focusing OATS on the wrong areas of applications
Using OATS for minimal testing efforts
Using OATS for High-Risk Applications
Picking the Wrong Parameters to Combine

6. Outline
How to Utilize and Apply OATS?
What Can OATS Provide?
Definition of Pairwise
Pairwise Facts
Where to get Pairwise?
Summary
References
Resourceful Websites

7. Definition of Orthogonal Array Testing Strategy
Orthogonal Array Testing Strategy (OATS) is a systematical, statistical way of testing pairwise interactions by deriving a suitable small set of test cases from a large number of scenarios.

8. The OAT Theory
OAT can be used to reduce the number of combinations and provide maximum coverage with a minimum number of test cases.
OAT is an array of values in which each column represents a variable - factor that can take a certain set of values called levels. Each row represents a test case.
In OAT, the factors are combined pairwise rather than representing all possible combinations of factors and levels.
Orthogonal arrays are two dimensional arrays of numbers which possess the interesting quality that by choosing any two columns in the array you receive an even distribution of all the pairwise combinations of values in the array.

9. Consider three parameters – A, B, and C.
Simple OAT Example
Consider three parameters – A, B, and C.
Parameters A, B, and C each have positive values – 1, 2, and 3.
Testing all combinations of the three parameters would involve executing a total of 27 test cases.
3 x 3 x 3 = 27

10. Simple OAT Example (cont’d)
With the way programming works, a fault will most likely depend on the values of two parameters, not three.
The fault might occur for each of these three test cases: A=1, B=1, C=1, A=1, B=1, C=2, and A=1, B=1, C=3.
Not necessary to run all 27 scenarios, 9 test cases is suffice.
The 9 scenarios outlined in Table 1 (on the next screen) address all possible pairs within the three parameters.

11. Pair-wise Combination of Parameters
All possible pairwise combinations between parameters A and B, B and C, and C and A are displayed in Table 1.
9 scenarios provides the same coverage as executing all 27 scenarios.
This same concept is applied to more complex scenarios where testing an application might require 10,000+ test cases and utilizing OATS, it can be reduced significantly in the number of test scenarios, such as down to 1,000 or less test cases to execute.

12. Orthogonal Array – Example 2 (More Complex)
Orthogonal Array has 3 columns, representing 3 variables.
Utilize a three level-alphabet – Red, Green, and Blue.
Next, organize combinations into a table (next slide).

13. Orthogonal Array – Example 2
All Combinations for Three Variables of Three Levels Each A B C 1
Red 2
Green 3
Blue 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Orthogonal Array has 3 columns, representing 3 variables.
Utilize a three level-alphabet – Red, Green, and Blue.

14. Orthogonal Array – Three Variables of Three Levels Each
For each pair of columns, AB, AC, and BC, each pair of colors appears exactly three times.
To save testing effort, the appearance of each pair has been reduced to once.
For each pair of columns, AB, AC, and BC, each pair of colors appears exactly three times.
To save testing effort, the appearance of each pair has been reduced to once.

15. Orthogonal Array – Example 2
All-Pairs Array, Three Variables of Three Levels Each A B C 2
Red
Green 4 9
Blue 12 14 16 19 24 26
Orthogonal Array has 3 columns, representing 3 variables.
Utilize a three level-alphabet – Red, Green, and Blue.

16. Why Use Orthogonal Array Testing?
Being intelligent about which test cases you choose can make all the difference between:
Endlessly executing tests that just aren’t likely to find bugs and don’t increase your confidence in the system.
Executing a concise, well-defined set of tests that are likely to uncover most (not all) of the bugs and that give you a great deal more comfort in the quality of your software.
Test case selection poses an interesting dilemma for the software professional.
Almost everyone has heard:
that you can’t test quality into a product
that testing can only show the existence of defects and never their absence
that exhaustive testing quickly becomes impossible — even in small systems.
However, testing is necessary.

17. Utilizes variable pair combination
Advantages of OATS
Utilizes variable pair combination
Provides uniformly distributed coverage
Useful for integration testing of software components
Useful for testing configurable options on an application
web page allowing font style
background color
page layout selections
There are many advantages the OATS technique can provide.

18. Arrays can be difficult to construct.
Disadvantages of OATS
Can only be applied at the initial stage of the product/process design system.
There are some situations whereby OA techniques are not applicable, such as a processes involving influencing factors that vary in time and cannot be quantified exactly.
Can be overwhelming to use the first time; improves with use on each project.
Arrays can be difficult to construct.
As with advantages of a strategy, there are also disadvantages.

19. Selecting the Right Test Set with OATS
OATS provides a means to select a test set that:
Guarantees testing the pair-wise combinations of all the selected variables.
Creates an efficient and concise test set with many fewer test cases than testing all combinations of all variables.

20. Selecting the Right Test Set with OATS (cont’d)
3. Creates a test set that has an even distribution of all pair-wise combinations.
4. Exercises some of the complex combinations of all the variables.
5. Is simpler to generate and less error prone than test sets created by hand.
Continuation of OATS providing a means to select a test set.

21. Applying OATS manually.
Mistakes to Avoid
Applying OATS manually.
Focusing the testing effort on the wrong area of the application.
Using OATS for minimal testing efforts.
Using OATS for high-risk applications.
Picking the wrong parameters to combine.
There are common mistakes often made when utilizing OATS.
By applying OATS manually, it refers to trying to figure out all of the possible combinations yourself and entering the data manually without the assistance of a tool, such as All Pairs.

22. Applying OATS manually
Manual entry of OATS test parameters is time consuming and prone to inaccuracies.
Utilize free orthogonal array freeware, to avoid manually applying OATS.
Orthogonal freeware can generate various array sizes.
We applied OATS manually during our first orthogonal testing effort.
We determined the variously sized orthogonal arrays based on our test parameters, and then manually replaced and plugged in the actual values into the table, using an Excel spreadsheet.
Doing all this manually was very time consuming, tedious, and prone to inaccuracies—and it required quite a bit of maintenance.
We didn’t have the extra budget to buy commercial test case generating tools, but had we known then about the availability of orthogonal array freeware, we could have used it to automatically generate various array sizes.
These automatically generated arrays could have been used in turn to write a program that automatically replaced the numbers in the table with actual test parameter values.

23. Focusing the testing effort on the wrong area of the application
Focus OATS on the critical areas of the application.
Evaluate where applying OATS is most feasible.
It is important to evaluate where the application testing effort will be most effective and efficient to apply.
Focus implementing the OAT testing effort from the perspective of what component(s) of the application would benefit most.
Focus on utilizing OAT on the critical areas of the application.
For example, one test engineer I know returned from a training class on the OATS technique, impatient to apply it right away. He spent days devising an incredibly elaborate orthogonal array testing system to verify possible query combinations applied through a GUI.
This specific application used a GUI front end that allowed the user to select various query combinations by selecting parameters on various pull-down menus—basically an in-house modification of a commercial off-the-shelf (or COTS) system that allowed for various application queries. The COTS tool had already been tested thoroughly by the vendor, but in his eagerness to apply his newly learned technique the test engineer lost sight of the big picture.
He applied his elaborate OATS scheme to the wrong area of the system: the already-tested COTS front end.
It would have been good enough to simply run a variety of GUI tests to verify that the pull-down selections worked and produced correct results. In this case using an elaborate OATS system was excessive, and the testing effort would have been better focused on verifying the accuracy of the SQL queries (the actual code) and on verifying the data migration.
The OATS testing method is a great resource, but it’s important to evaluate where in your application testing effort it will be most effective and efficient to apply, and how much of it to use.

24. Using OATS for minimal testing efforts
There are times when OATS might ask for more test cases than is actually necessary.
Determine if budget and time allows for a “full testing effort” or a “good enough testing effort”.
If utilize OATS-generated test case combination, omit unnecessary tests.
In some cases OATS might ask for more test cases than is necessary for a “good enough” testing effort or for the testing time and budget allotted.
Here again, it’s important to evaluate whether the OATS testing technique is appropriate, or whether other testing techniques such as boundary values, equivalence classes, etc., would be sufficient.
If the test engineer decides to go forward with the implementation of orthogonal array testing despite timing or budget concerns, it’s important to evaluate the OATS-generated test case combinations and omit any excessive test input combinations.

25. Using OATS for high-risk applications
Testing medical device systems or other life-critical applications require more than just OATS.
Other testing techniques must be used to complement OATS.
OATS is used to minimize the test combination inputs.
Not a technique for exhaustive testing.
If you are testing medical device systems or any other life-critical applications, don’t pin your hopes on orthogonal array testing to do the job all by itself.
You’ll need to complement this testing technique with other well-established testing techniques.
It’s important to keep in mind that the goal of OATS is to allow for minimizing the test combination inputs—it’s not a technique that allows for exhaustive testing.

26. Picking the Wrong Parameters to Combine
Understand how the application functions.
Enlist the help of:
Subject Matter Experts (SMEs)
Developers
Business Analysts
Pick the right parameter combinations based on feedback from SME, Developers, and BAs.
Ask my audience members to come up with an example application on which they think they could apply OATS.
I then ask for volunteers to present their example application and how they would specifically apply OATS.
The most common issue that surfaces during this exercise is that people pick the wrong parameters to combine.
Picking those parameters can be an elaborate effort—especially when a wide variety or huge number of parameter combinations is involved.
Enlist the help of a subject matter expert, a developer who understands the application code, and a test engineer.

27. How to Utilize and Apply OATS?
Steps to use OATS are outlined below:
Identify independent variables
Determine the maximum number of values
Find a suitable array with the smallest number of runs
Map the factors and values onto the array
Choose values for any remaining level
Transcribe the runs into test cases
Steps to use OATS are outlined below:
Identify independent variables to map the factors of the array.
Determine the maximum number of values that each independent variable will take on to map the levels of the array.
Find a suitable array with the smallest number of runs. A suitable array is one that has at least as many factors as needed from step 1 and has at least as many levels for each of the factors determined in step 2.
Map the factors and values onto the array.
Choose values for any remaining levels.
Transcribe the runs into test cases, adding any particularly suspicious combinations that are not generated.

28. OATS provides a means to select a test set that:
What Can OATS Provide?
OATS provides a means to select a test set that:
Guarantees testing the pair-wise combinations of all selected variables.
Creates an efficient and concise test set with fewer test cases.
Creates a test set that has an even distribution of all pair-wise combinations
Orthogonal Array Testing can provide means to select test sets.

29. What Can OATS Provide? (cont’d)
OATS provides a means to select a test set (cont’d):
4. Exercises some of the complex combinations of all variables.
5. Is simpler to generate and less error prone than test sets created by hand.

30. Definition of Pairwise Testing
Recall Orthogonal Array Testing Strategy is a systematical, statistical way of testing pairwise interactions by deriving a suitable small set of test cases from a large number of scenarios.
Pairwise testing is a type of combination testing method, where each pair of input parameters to a system (usually software algorithms), tests all possible combinations of the parameters.

31. Pairwise testing is also known as Orthogonal Array Testing.
Pairwise Facts
Pairwise testing is also known as Orthogonal Array Testing.
The goal of using orthogonal arrays in testing is:
to get the biggest bang for the testing buck
performing the smallest number of tests likely to expose defects
To keep the test scenario numbers low, the desired combinations are pairs of variables in each of their possible states, rather than combinations of three or more variables. This makes pair-wise testing a kind of subset of orthogonal array testing.
Pairwise and Orthogonal Array Testing are one in the same.
The overall goal is to execute the smallest number of tests to identify defects.

32. All Pairs can be downloaded from http://www.satisfice.com/tools.shtml
Where to get All Pairs?
All Pairs software will test for all possible discrete combinations of the parameters identified for testing.
All Pairs can be downloaded from
Let's say you must test something that involves a lot of different variables.
Configuration testing is like that: different printers, different print options, different documents.
Who knows what combination will fail?
But you can't try all combinations of all variables together, because that would be too many tests.
So instead, you could create tests that pair each value of each of the variables with each value of each other variable at least once.
Allpairs is a tool designed to find a reasonably small set of test cases to satisfy that coverage standard. For instance, to try all combinations of 10 variables with ten values each would require 10,000,000,000 test cases. Allpairs only requires 177 cases. Allpairs is a command-line executable based on a Perl script. Source is included.

33. Summary
Orthogonal Array Testing Strategy is a systematical, statistical way of testing pair-wise interactions by deriving suitable small set of test cases from a large number of scenarios.
OAT should be used to reduce the number of combinations and provide maximum coverage with a minimum number of test cases.
OAT guarantees testing the pair-wise combinations of all selected variables.
The importance of software in many industries has been growing exponentially in the last two decades, and the trend is accelerating.
Tasks that were once done through hardware are performed today much more simply and inexpensively through software.
New functions and features are added daily to these products that were once not even conceivable.
At the same time, the complexity, cost, and reliability issues of software are growing rapidly. To adapt existing Robust Design methods for software engineering and to invent new methods to improve quality, cost, and schedule is critical.  

34. Focusing the testing effort on the wrong area of the application.
Summary
Mistakes to Avoid.
Focusing the testing effort on the wrong area of the application.
Using OATS for minimal testing efforts.
Using OATS for high-risk applications.
Picking the wrong parameters to combine.
Pairwise testing is a type of combination testing method, where each pair of input parameters to a system (usually software algorithms), tests all possible combinations of the parameters.
The importance of software in many industries has been growing exponentially in the last two decades, and the trend is accelerating.
Tasks that were once done through hardware are performed today much more simply and inexpensively through software.
New functions and features are added daily to these products that were once not even conceivable.
At the same time, the complexity, cost, and reliability issues of software are growing rapidly. To adapt existing Robust Design methods for software engineering and to invent new methods to improve quality, cost, and schedule is critical.  

35. References
Bolton, M. (2004). Pairwise testing. Retrieved on December 1, 2009, from
Dustin, E. (2001). Orthogonally Speaking. Retrieved on November 17, 2009, from

36. References (cont’d)
Harrell, J. (2001). Orthogonal Array Testing Strategy (OATS) Technique. Retrieved on November 29, from
Learn Software Development. (2007). Orthogonal Array Testing Strategy (OATS). Retrieved on November 21, 2009, from

37. References (cont’d)
StickyMinds. (2001). Orthogonally Speaking. Retrieved on November 15, 2009, from
VTB. (2008). Orthogonal Array Testing Strategy (OATS). Retrieved on November 10, 2009, from

38. Resourceful Websites
These are some excellent websites to further explain the Orthogonal Array Testing Strategy and the AllPair Testing approach.