Sergiy Vilkomir January 20, 2012 East Carolina University, Department of Computer Science, Software Testing Research Group Research Seminar Introduction to experimental investigation of effectiveness of software testing Sergiy Vilkomir January 20, 2012

Effectiveness of software testing We investigate effectiveness of software testing criteria A testing criterion is a rule, that tells us the requirements that a set of test cases should satisfy Criterion = Approach = Strategy = Method Effectiveness of testing criteria is the ability to detect failures in a software program. STRG January 20, 2012

Effectiveness of software testing The measure of effectiveness: the probability that one test set, randomly taken according to criterion C, will detect a fault. The definitions are very general and in practical investigations some specific areas are usually considered (specific criteria / types of software / types of faults) STRG January 20, 2012

Testing logical expressions One of the specific areas - testing logical expressions in software Often important for safety critical systems Example: if (P<140  T>280)  (P<150  T>260  N75) then shut_down_reactor Testing logical expressions has been extensively investigated Many criteria have been suggested, e.g. control-flow criteria (CC, DC, MC/DC, RC/DC, etc) STRG January 20, 2012

Pair-wise testing Combinations of either configuration parameter values or input parameter values are considered All possible pairs of values must be covered parameter = factor; value = level Pair-Wise: A value of each level for each factor must be combined with a value from every levels for each other factor. STRG January 20, 2012

Pair-wise – 13 tests All pair are here 1 a 1 true 50 2 a 2 false 60 3 a 3 true 70 4 b 1 false 60 5 b 2 true 50 6 b 3 false 70 7 c 1 true 70 8 c 2 false 50 9 c 3 true 60 10 d 1 false 50 12 d 2 true 70 12 d 3 * 60 13 * 3 * 50 An example 4 input parameters X - a, b, c, d Y - 1, 2, 3 Z - true, false S – 50, 60, 70 Total number of test combinations - 72 STRG January 20, 2012

Pair-wise testing Provides a small set of test cases. It is a practical alternative to testing all combinations. Example 1: 3 parameters with 3 values each All combinations: 27 test cases Pair-wise: 9 test cases Example 2: 75 parameters (35 – 2 values, 39 – 3 values, 1 – 4 values) All combinations: 5.5 x 1029 test cases Pair-wise: 28 test cases STRG January 20, 2012

Research area CC, DC, MC/DC, RC/DC Pair-wise testing Software input domain; Combinations of options, selections, and configurations CC, DC, MC/DC, RC/DC Logical expressions in software STRG January 20, 2012

Research Question How effective is the pair-wise and t-way approach for testing logical expressions in software ? It is a non-traditional area of application for the pair-wise approach According to our knowledge, the only one detailed investigation of using pair-wise approach for testing logical expression was made by Kobayashi et al (N. Kobayashi, T. Tsuchiya, and T. Kikuno, “Non-specification-based approaches to logic testing for software”, Information and Software Technology, 2002, Vol. 44, No. 2, pp. 123–121) STRG January 20, 2012

Our approach - experimental investigation Step 1. Consider sets of logical expression as case studies Step 2. For each expression, generate (using our own tool) faulty expressions for all possible faults of 5 different types Step 3. Generate pair-wise test sets using external tools Step 4. Apply these test sets to all faulty expressions and determine (using our own tool) if faults were revealed Step 5. Evaluate effectiveness of pair-wise testing STRG January 20, 2012

Details- in the next two presentations Two papers published and presented at international conferences One paper submitted New works in progress Interest from other researches STRG January 20, 2012

Thank you! Questions? STRG January 20, 2012