Test Suite Reduction for Regression Testing of Simple Interactions between Two Software Modules Dmitry Kichigin

Software regression integration testing Software integration testing – testing of interactions between software elements being integrated Software regression testing – retesting of software to verify that modifications have not caused unintended effects and that the software still complies with its specified requirements Problem: high cost of regression testing Solutions: Automation of testing Test suite reduction

“ seeks to reduce the number of test cases in a test suite while retaining a high percentage of the original suite’s fault detection effectiveness “ This reduces: The costs associated with test suite maintenance The costs of software execution on the tests Result: reduction of overall costs associated with regression testing

Traditional approach Involves a static analysis and/or instrumentation of a source code and consists of 3 main steps: 1.Instrumentation of software source code 2.Execution of software on a test suite and collecting coverage information 3.Using coverage information to reducing the test suite Existing methods: - Interface Mutation (Mutation Analysis) - Structural methods: –Control-flow based –Data-flow based

Drawbacks Recent trends in software development may limit their applicability: Component-based software development and use of commercial off-the-shelf (COTS) components: –the source code is unavailable Increasing size and complexity of the software: –source-code based coverage analysis is expensive and hard to conduct for large software

Proposed solution Test Suite Reduction Method for Regression Testing of Simple Interactions between Two Software Modules : Based on interactions behaviour monitoring Uses sequences of interface function calls to analyse interactions behaviour Takes into account function parameters Does not require access or instrumentation of a source code

Definitions Simple interaction - interaction via interface of functions which can take only scalar parameters Interface functions are the functions, which form a part of module’s programming interface. Interaction trace of interaction between module A and module B is a sequence of interface functions of module B, which are invoked by module A during software execution on a test case t. … FILE* fp = fopen(…); for (int i=0; i<3; i++) { fread(…); } fclose(fp); … fopen fread fclose …

Algorithm T – original test suite T’ – reduced test suite, T’  T t – next test case from the original test suite, t  T M t – model of interaction behaviour on the test case t M T’ – set of models of interaction behaviour on test cases from T’ Main steps: 1.Pick the next test case from the original test suite and execute the program 2.Build modules interactions model M t 3.If M t  M T’, then add M t to M T’ and add test case t to T’ 4.Run until T is exhausted yes no t := next(T) M t := build_model(t) M t  M T’ yes finish no ? M t  M T’ ? T ==  M T’ := M T’  M t T’ := T’  t M t empty? M t  M T’

Interaction Behaviour Model K-length sequence of interface functions is an arbitrary continuous sequence of length K, which can be located in an interaction trace. Model of interaction between module A and module B on a test t - set of all possible K-length sequences, which can be located in the interaction trace for the interaction between A and B on t. Use “sliding window” technique to build the model: fopen fwrite fwrite fopen fwrite fclose fwrite fclose fopen fwrite fwrite fopen fwrite fwrite fopen fwrite fwrite fopen fwrite fclose fopen fwrite fclose fwrite fwrite fclose

Equality relation between models Two given models are equal their sequence sets are equal Two sequences of functions are equal their elements are equal pairwise Function calls are equal when: the functions have the same names, the values of nominal parameters are the same, and the values of numerical parameters belong to the same intervals.

Equality relation between models with taking into account parameter values Function scalar parameters Nominal Numerical Nominal: definition area is a finite set no order relation and distance defined equality relation:  (x,y), where  - Kronecker symbol Numerical: definition area is R both order relation and distance are defined equality relation:

Results of the experiment: level of test suit reduction Subject programs: GNU Assembler & the C standard library

Results of the experiment: fault detection ability Subject programs: GNU Assembler & the C standard library

Summary The new test suite reduction method was presented, which: does not require access or instrumentation of a source code uses sequences of interface function calls as behaviour characteristic takes into account values of function parameters The experimental evaluation demonstrated: can successfully reduce the size of a test suite while retaining its fault detection ability the method outperforms the random test suite reduction method

Thank you

Questions ?