1. Sergiy Vilkomir January 20, 2012
East Carolina University, Department of Computer Science, Software Testing Research Group
Research Seminar
Introduction to experimental investigation of effectiveness of software testing
Sergiy Vilkomir
January 20, 2012

2. Effectiveness of software testing
We investigate effectiveness of software testing criteria
A testing criterion is a rule, that tells us the requirements that a set of test cases should satisfy
Criterion = Approach = Strategy = Method
Effectiveness of testing criteria is the ability to detect failures in a software program.
STRG
January 20, 2012

3. Effectiveness of software testing
The measure of effectiveness: the probability that one test set, randomly taken according to criterion C, will detect a fault.
The definitions are very general and in practical investigations some specific areas are usually considered (specific criteria / types of software / types of faults)
STRG
January 20, 2012

4. Testing logical expressions
One of the specific areas - testing logical expressions in software
Often important for safety critical systems
Example:
if (P<140  T>280)  (P<150  T>260  N75)
then shut_down_reactor
Testing logical expressions has been extensively investigated
Many criteria have been suggested, e.g. control-flow criteria (CC, DC, MC/DC, RC/DC, etc)
STRG
January 20, 2012

5. Pair-wise testing
Combinations of either configuration parameter values or input parameter values are considered
All possible pairs of values must be covered
parameter = factor; value = level
Pair-Wise: A value of each level for each factor must be combined with a value from every levels for each other factor.
STRG
January 20, 2012

6. Pair-wise – 13 tests All pair are here 1 a 1 true 50 2 a 2 false 60 3 a 3 true 70 4 b 1 false 60 5 b 2 true 50 6 b 3 false 70 7 c 1 true 70 8 c 2 false 50 9 c 3 true d 1 false d 2 true d 3 * * 3 * 50
An example
4 input parameters
X - a, b, c, d
Y - 1, 2, 3
Z - true, false
S – 50, 60, 70
Total number of test combinations - 72
STRG
January 20, 2012

7. Pair-wise testing Provides a small set of test cases.
It is a practical alternative to testing all combinations.
Example 1:
3 parameters with 3 values each
All combinations: 27 test cases
Pair-wise: 9 test cases
Example 2:
75 parameters (35 – 2 values, 39 – 3 values, 1 – 4 values)
All combinations: 5.5 x test cases
Pair-wise: 28 test cases
STRG
January 20, 2012

8. Research area CC, DC, MC/DC, RC/DC Pair-wise testing
Software input domain;
Combinations of options, selections, and configurations
CC, DC, MC/DC, RC/DC
Logical expressions in software
STRG
January 20, 2012

9. Research Question
How effective is the pair-wise and t-way approach for testing logical expressions in software ?
It is a non-traditional area of application for the pair-wise approach
According to our knowledge, the only one detailed investigation of using pair-wise approach for testing logical expression was made by Kobayashi et al
(N. Kobayashi, T. Tsuchiya, and T. Kikuno, “Non-specification-based approaches to logic testing for software”, Information and Software Technology, 2002, Vol. 44, No. 2, pp. 123–121)
STRG
January 20, 2012

10. Our approach - experimental investigation
Step 1. Consider sets of logical expression as case studies Step 2. For each expression, generate (using our own tool) faulty expressions for all possible faults of 5 different types Step 3. Generate pair-wise test sets using external tools Step 4. Apply these test sets to all faulty expressions and determine (using our own tool) if faults were revealed Step 5. Evaluate effectiveness of pair-wise testing
STRG
January 20, 2012

11. Details- in the next two presentations
Two papers published and presented at international conferences
One paper submitted
New works in progress
Interest from other researches
STRG
January 20, 2012

12. Thank you! Questions?
STRG
January 20, 2012