PJSISSTA '001 Black-Box Test Reduction Using Input-Output Analysis ISSTA ‘00 Patrick J. Schroeder, Bogdan Korel Department of Computer Science Illinois Institute of Technology Chicago, IL USA

PJSISSTA '002 1) Motivation 2) Black-Box Testing 3) Input-Output Analysis 4) Determining Input-Output Relationships 5) Conclusion Presentation Overview

PJSISSTA '003  Experience in Industry as a black-box tester of large, complex, telecommunications systems  Root cause analysis of customer-found faults often discovered ineffective or redundant black-box tests  Additional information from the software implementation often improves black-box testing (after the initial release!) Motivation for Research

PJSISSTA '004 Black-Box Testing  Black-box testing is testing from a functional or behavioral perspective to ensure a program meets its specification  Testing usually conducted without knowledge of software implementation - system treated as a “black box”  Black-box test design techniques include: equivalence partitioning, boundary value analysis, cause-effect graphing, random testing

PJSISSTA '005 Observations on Black-Box Testing  Testing from the black-box perspective generates a large number of tests (e.g. AT&T Definity PBX > 3 million tests)  All testers are faced with resource limitations; the number of tests must be reduced  Test reduction done randomly or based on “engineering judgement” may lead to ineffective or redundant tests

PJSISSTA '006 Combinatorial Testing  Combinatorial testing: For programs with multiple inputs, one must decide how to test different combinations of selected test data values  The most thorough approach is to test all combinations - this is quite often impossible due to the large number of tests  However, not to consider testing different input combinations could lead to an unacceptable number of undetected software faults

PJSISSTA '007 Combinatorial Explosion If n = 10 and 10 test data values are selected for each variable, the total number of combinatorial test cases is X 1 X 2 X 3... X n Program P

PJSISSTA '008 Current Approaches  Testers may try to identify “important” combinations to test using specifications, historical fault data, and engineering judgement  Orthogonal arrays, or other combinatorial design techniques, may be used to tests pair-wise or n-way combinations of inputs  Difficult to determine impact on fault-detection capability of the test set using these techniques

PJSISSTA '009 Definition of the Problem  In combinatorial testing, black-box testers face the situation where the number of possible test cases far exceeds the time and resources available to execute them  How can we reduce the number of combinatorial tests while maintaining the fault-detection capability of combinatorial testing?

PJSISSTA '0010 Proposed Technique  Observations:  Current techniques do not consider program output variables  For a class of programs, not all program inputs affect all program outputs  We propose the use of Input-Output (IO) Analysis to determine which program inputs affect program outputs  This information can be used to reduce the number of combinatorial tests while “maintaining” the fault-detection capability of the test set

PJSISSTA '0011 Simple Example using IO Analysis Input data selected using equivalence partitioning: A={1, 3} B={N,S,E,W} C={TDC, BDM} W Z Total Number of Combinatorial Tests: 2 * 4 * 2 = 16

PJSISSTA '0012 IO Analysis Applied A={1, 3} B={N,S,E,W} C={TDC, BDM} W Z

PJSISSTA '0013 IO Analysis Applied T(W) (1, N, TDC) (1, N, BDM) (3, N, TDC) (3, N, BDM) T(Z) (1, N, TDC) (1, S, TDC) (1, E, TDC) (1, W, TDC) A={1, 3} B={N,S,E,W} C={TDC, BDM} W Z T(W)  T(Z) (1, N, TDC) (1, N, BDM) (3, N, TDC) (3, N, BDM) (1, S, TDC) (1, E, TDC) (1, W, TDC)

PJSISSTA '0014 Optimization Problem Create the smallest possible test set which covers all data combinations in the individual output test sets Test set optimized using “Don’t Care” (DC) conditions Toptimized (1, N, TDC) (1, S, BDM) (3, E, TDC) (3, W, BDM) T(W) (1, DC, TDC) (1, DC, BDM) (3, DC, TDC) (3, DC, BDM) T(Z) (DC, N, DC) (DC, S, DC) (DC, E, DC) (DC, W, DC)

PJSISSTA '0015 Determining Input-Output Relationships  Information may be gathered in a variety of ways (e.g. from specifications, code)  Manual collection of information difficult and time consuming  Automated collection possible through:  Static Analysis  Dynamic Analysis  Execution-oriented Analysis

PJSISSTA '0016 Automated Analysis Techniques  Static techniques analyze a program’s source code - static program dependencies used to determine IO relationships  Dynamic techniques analyze a program’s execution trace across a set of test data - dynamic dependencies used to determine IO relationships  Execution-oriented techniques manipulate individual inputs across multiple program executions in a test harness to determine IO relationships

PJSISSTA '0017 Technique Advantages/Disadvantages  Static techniques reliably determine the absence of relationships between inputs and outputs; however, they may identify relationships that are not actually present  Dynamic analysis and execution-oriented analysis reliably determine the presence of input-output relationships; however, they cannot guarantee that all relationships will be detected  Dynamic analysis and execution-oriented analysis may be used when static analysis, perhaps in over-estimation, indicates that all program inputs affect all program outputs

PJSISSTA '0018 Experimental Study Results 1) Warehouse Management System Combinatorial Tests: 3,125 Using IO Analysis 40 2) DACS Configuration System Combinatorial Tests22,500 Using IO Analysis 158 3) Liquidity Ratio Spreadsheet Combinatorial Tests 1.95 Million Using IO Analysis 625

PJSISSTA '0019 Conclusion  For the class of program and type of problem defined, input- output analysis can reduce the number of combinatorial tests while “maintaining” the fault-detection capability of the test set  Research continues on identifying program types and testing situations where our approach is applicable  Additional experimentation with execution-oriented analysis to provide program dependencies, and other information useful in black-box testing, is underway