1. Ryan Lekivetz & Joseph Morgan, JMP
Validating systems with covering arrays and what to do when you get a failure
Ryan Lekivetz & Joseph Morgan, JMP
Abstract
ABSTRACT
JMP 12 introduced the first JMP Pro platform for Design of Experiments (DOE) with the Covering Array platform. Covering Arrays may be used to validate complex deterministic systems, as well as configurations of such systems, by revealing if any combination of system inputs, or any combination of configurations, induce a failure. However, the discovery of a failure does not necessarily indicate which level combination precipitated the failure. In this talk, we provide an overview of Covering Arrays and discuss how the Covering Array analysis tool that was also introduced in JMP12 can help identify the underlying cause of any discovered failures. 
Covering Arrays
Analysis
Examples
Final Thoughts
References
Template provided by ePosterBoards

2. Ryan Lekivetz & Joseph Morgan, JMP
Validating systems with covering arrays and what to do when you get a failure
Ryan Lekivetz & Joseph Morgan, JMP
Abstract
COVERING ARRAYS
Deterministic system (same input gives same output every time)
Failures more likely induced by small number of factors
Definition: A covering array CA(N; t, (v1*v2…*vk)) is an N x k array such that the i’th column contains vi distinct symbols. If for any t coordinate projection, all combinations of symbols exist, then it is a t-covering array. If N is minimal for fixed t, k, and (v1…vk), it is said to be optimal.
For a set of factors, a t-covering array has the property that for any subset of t factors, every possible level combination occurs at least once.
Covering arrays ensure that we’ve covered all t-way combinations (and more!)
Very economical with number of tests required vs. exhaustive testing
Test economy also means that when a failure occurs, we’re not sure of the cause
Covering Arrays
Analysis
Examples
Final Thoughts
References
Template provided by ePosterBoards

3. Ryan Lekivetz & Joseph Morgan, JMP
Validating systems with covering arrays and what to do when you get a failure
Ryan Lekivetz & Joseph Morgan, JMP
Abstract
ANALYSIS
If we have one or more failures, it’s natural to want to determine the cause (and fix it!)
After recording success/failure for each test, create a list of potential causes
JMP can generate this list automatically for us
Potential causes:
For a test that contains a failure, many possible combinations of factors exist in that test row
Can eliminate the combinations we’ve seen in tests that weren’t failures
Covering Arrays
Analysis
Examples
Final Thoughts
References
Template provided by ePosterBoards

4. Ryan Lekivetz & Joseph Morgan, JMP
Validating systems with covering arrays and what to do when you get a failure
Ryan Lekivetz & Joseph Morgan, JMP
JMP Installation
Categorical Platform
Abstract
OS/License/Installer
2 failures found
Variety of preferences
Many possible tests
2-coverage can be done in 16 runs
Covering Arrays
Analysis
Examples
DOE setup / data table
Preferences for platform
Final Thoughts
References
Analysis results
Analysis results
M. Drake, Discovery Summit Europe 2016
Template provided by ePosterBoards

5. Ryan Lekivetz & Joseph Morgan, JMP
Validating systems with covering arrays and what to do when you get a failure
Ryan Lekivetz & Joseph Morgan, JMP
Abstract
FINAL THOUGHTS
Effective means of validating complex systems
If all tests pass, ascertain that there is no fault due to t or few factors
If we do find failures, analysis can guide us towards the cause
Analysis can also accommodate designs with disallowed combinations
Covering Arrays
Analysis
Examples
Final Thoughts
References
D. Kuhn, D. R. Wallace, & A. M. Gallo, “Software Fault Interactions & Implications for Software Testing,” IEEE TSE, 30(6), 2004,
M. J. Drake. “Effective Test Design Using Covering Arrays.” Poster presented at Discovery Summit Europe 2016.
References
Template provided by ePosterBoards

8. Testing Preferences
Categorical platform preference set has lots of checkboxes (2-level), as well as a few options with 3 or more choices
712,483,534,798,848 possible combinations to test
Covering array starts with a 20-run design for strength 2
In a few seconds, Optimize finds a 16-run design, which is the minimum run size

9. Of the 946 (44 Choose 2) two-factor combinations in the 1 failure run, reduced to 12 possibilities.

10. Effectiveness of Covering Arrays
#factors
involved
in failure
Medical
devices
Browser
Server
NASA
GSFC 1 66 29 42 68 2 97 76 70 93 3 99 95 89 98 4
100 96 5 6
Cumulative percentage of faults triggered by interactions involving number of factors indicated in leftmost column (Kuhn et al. (2004)).
D. Kuhn, D. R. Wallace, & A. M. Gallo, “Software Fault Interactions & Implications for Software Testing,” IEEE TSE, 30(6), 2004,