Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville

2 Topic Covered What is Regression Testing? Why Regression testing? Regression Testing Process Object-Oriented Regression Testing

3 What is Regression Testing? Testing activities occur after software changes  During software maintenance phase Execution of a set of test cases on a program in order to ensure that implemented changes do not produce unintended faults

Regression Testing Levels Unit Testing Integration Testing Acceptance Testing

Who Perform Regression Testing Developers: regression testing at the unit level or integration Test Engineers : regression testing at the function level Quality Engineers: regression testing at the system level

Why Regression Testing? Validate and Verify the modified software  Validate and verify the new features  Ensure existing features are still working(change impact) Small changes in one part of a program may have indirect undesired effects in other unrelated parts of the program Fixing could introduce new bugs

Why Regression Testing?

8 Regression Testing Process Input:  Given program P, its modified version P’, and a test set T that was used to previously test P Goal:  Find a way to utilize T to gain sufficient confidence in the correctness of P’

Regression Testing Process Activities:  Regression Test Selection Select T’ ⊆ T, a set of test cases to execute on P’  Test Suite Execution Test P’ with T’, establishing correctness of P’

Regression Test Selection Selecting a subset of the existing test suite based on information about:  Original Version (P)  Modified Version (P’)  Original Test Suite (T) Approaches:  Select All  Modification-Based Test Selection  Coverage-Based Test Selection Test Set Minimization Test Case Prioritization

Select All Traditional approach Select all the test cases in T to test P’ Advantage:  Safe Problem:  Too expensive Note: the new and the old programs could produce different outputs

Modification-Based Test Selection Select set of test cases T that covers only modified or affected portions of P Based on Module-Based Firewall  A changed software module and a closure of all possible affected modules and related integration links in a program based on the static and dynamic analyses (e.g.: a control-flow graph). We can come out different kinds of firewalls based on various test models such as:  Control Firewall  Data Firewall  Function (method) Calling Firewall  Class Firewall (Object-Oriented Regression Testing)

Modification-Based Test Selection Test selection concept based om Module Firewall  re-test of the changed module and its affected modules within the firewall  re-test all of related integration links in the firewall

Main M1M3M2 M7M6M5M4 M8 Modification-Based Test Selection : Control Flow Mi : Module

Main M1M3M2 M7M6M5M4 M8 Changed Module Modification-Based Test Selection

Main M1M3M2 M7M6M5M4 M8 Changed Module Modification-Based Test Selection A control based module firewall: M5, M1, M8, and Main

Main M1M3M2 M7M6M5M4 M Changed Module Modification-Based Test Selection A control based module firewall: M5, M1, M8, and Main Re-testing at the unit level: M5 Re-testing at the integration level: 1, 2, 3

Modification-Based Test Selection Problems:  Still too many test cases Need to further reduce the number of regression tests  Coverage: the risk of missing test cases that are able to reveal defect not yet discovered

Test Set Minimization The test suite is reduced to a minimal subset that maintains the same coverage as the original test suite with respect to a given coverage criterion Goal: Find a minimal subset T’of T such that each testable entity (function, statement,…) is covered by at least one test in T’ What we need ?  Coverage (Execution Flow Trace) per test case

Test Set Minimization Steps:  Step1: From old version P : determine the coverage for each test t ⊆ T, determine which of the k testable entities is covered.  Step2: Find a minimal subset T’ of T such that each testable entity (function, statement,…) is covered by at least one test in T’by using Test Case Prioritization Note: T’ is not unique

Test Case Prioritization Tests are prioritized based on some criteria. Highest priority test are executed earlier Test Prioritization Approaches:  Total statement coverage # of statements executed  Additional statement coverage((Heuristic #1) A higher priority is assigned to a test that covers the higher number of not yet executed statements  Total function coverage # of functions executed  Additional function coverage

Test Set Minimization Example1: Test prioritization based on total function coverage  Suppose P contains two functions: main and foo suppose that P is tested using test cases t1 and t2 During testing it was observed that :  t1 causes the execution of main but not of foo  t2 causes the execution of both main and foo  Now suppose that P’ is obtained from P by making some modification  Which of the two test cases should be included in the regression test suite? regression test suite consists of only t2 (minimization)

Test Set Minimization Example2: Test prioritization based on total statement coverage  Assume P contains two functions main and f1, where the flow charts are as follow

Test Set Minimization Example2: Assume the following test cases Test case Coverage T1 main: {1, 2, 3} f1: {1, 3} T2 main: {1, 3} f1: {1, 3} T3 main: {1, 3} f1: {1, 2, 3} A minimal test set for regression testing is {t1, t3}

Test Case Prioritization Additional statement coverage((Heuristic #1)  A higher priority is assigned to a test that covers the higher number of not yet executed statements

Additional Statement Coverage Executed statements for each test t 1 : S 1, S 2, S 3 t 2 : S 1, S 5, S 8, S 9 t 3 : S 1, S 5, S 7 t 4 : S 1, S 5, S 3, S 4 t 5 : S 1, S 2, S 7 t 6 : S 1, S 2 t 7 : S 1, S 2, S 4 t 8 : S 1, S 2, S 3, S 4, S 7 t 9 : S 1, S 6 t 10 : S 1, S 2

Additional Statement Coverage Executed statements for each test t 1 : S 1, S 2, S 3 t 2 : S 1, S 5, S 8, S 9 t 3 : S 1, S 5, S 7 t 4 : S 1, S 5, S 3, S 4 t 5 : S 1, S 2, S 7 t 6 : S 1, S 2 t 7 : S 1, S 2, S 4 t 8 : S 1, S 2, S 3, S 4, S 7 t 9 : S 1, S 6 t 10 : S 1, S 2

Additional Statement Coverage Executed statements for each test t 1 : S 1, S 2, S 3 t 2 : S 1, S 5, S 8, S 9 t 3 : S 1, S 5, S 7 t 4 : S 1, S 5, S 3, S 4 t 5 : S 1, S 2, S 7 t 6 : S 1, S 2 t 7 : S 1, S 2, S 4 t 8 : S 1, S 2, S 3, S 4, S 7 t 9 : S 1, S 6 t 10 : S 1, S 2 S: t 8

Additional Statement Coverage Executed statements for each test t 1 : S 1, S 2, S 3 Covered statements t 2 : S 1, S 5, S 8, S 9 S 1, S 2, S 3, S 4, S 7 t 3 : S 1, S 5, S 7 t 4 : S 1, S 5, S 3, S 4 t 5 : S 1, S 2, S 7 t 6 : S 1, S 2 t 7 : S 1, S 2, S 4 t 9 : S 1, S 6 t 10 : S 1, S 2 S: t 8

Additional Statement Coverage Executed statements for each test t 1 : S 1, S 2, S 3 Covered statements t 2 : S 1, S 5, S 8, S 9 S 1, S 2, S 3, S 4, S 7 t 3 : S 1, S 5, S 7 t 4 : S 1, S 5, S 3, S 4 t 5 : S 1, S 2, S 7 t 6 : S 1, S 2 t 7 : S 1, S 2, S 4 t 9 : S 1, S 6 t 10 : S 1, S 2 S: t 8

Additional Statement Coverage Executed statements for each test t 1 : S 1, S 2, S 3 Covered statements t 2 : S 1, S 5, S 8, S 9 S 1, S 2, S 3, S 4, S 7 t 3 : S 1, S 5, S 7 t 4 : S 1, S 5, S 3, S 4 t 5 : S 1, S 2, S 7 t 6 : S 1, S 2 t 7 : S 1, S 2, S 4 t 9 : S 1, S 6 t 10 : S 1, S 2 S: t 8, t 2

Additional Statement Coverage Executed statements for each test t 1 : S 1, S 2, S 3 Covered statements S 1, S 2, S 3, S 4, S 5, S 7, S 8, S 9 t 3 : S 1, S 5, S 7 t 4 : S 1, S 5, S 3, S 4 t 5 : S 1, S 2, S 7 t 6 : S 1, S 2 t 7 : S 1, S 2, S 4 t 9 : S 1, S 6 t 10 : S 1, S 2 S: t 8, t 2

Additional Statement Coverage Executed statements for each test t 1 : S 1, S 2, S 3 Covered statements S 1, S 2, S 3, S 4, S 5, S 7, S 8, S 9 t 3 : S 1, S 5, S 7 t 4 : S 1, S 5, S 3, S 4 t 5 : S 1, S 2, S 7 t 6 : S 1, S 2 t 7 : S 1, S 2, S 4 t 9 : S 1, S 6 t 10 : S 1, S 2 S: t 8, t 2

Additional Statement Coverage Executed statements for each test t 1 : S 1, S 2, S 3 Covered statements S 1, S 2, S 3, S 4, S 5, S 7, S 8, S 9 t 3 : S 1, S 5, S 7 t 4 : S 1, S 5, S 3, S 4 t 5 : S 1, S 2, S 7 t 6 : S 1, S 2 t 7 : S 1, S 2, S 4 t 9 : S 1, S 6 t 10 : S 1, S 2 S: t 8, t 2, t 9

Test Set Minimization/ Prioritization Problem:  Test minimization discards test cases.  There is a chance that, if P’ were executed against a discarded test case, it would reveal an error in the modification made

Object-Oriented Regression Testing Typical Cases  Inheritance: Getting a new subclass working without exercising the superclass is usually easy No need to re-test the superclass  We assume that inherited features just work  Component based development (Re-use) OO technologies offer strong support for reuse but they cannot prevent regression bugs No unit-testing We need integration testing

Object-Oriented Regression Testing  Dependency: Dependencies occur for many reasons and so regression faults can occur in many ways Assume a dependency is between B and D  D has a side effect on B: B fails because a new action of D is inconsistent with B’s requirements and assumptions with respect to D  D is a client of B: D sends a message that violates B’s invariant/precondition. B is not defensive and accepts the incoming message, leading to failure

Object-Oriented Regression Testing  Dependency (contd.):  D is a server of B: B sends a message to D. D has been changed (or contain bugs). D returns a value that causes a violation of B’s invariant. B fails or returns an invalid value to another baseline component C. C fails

Object-Oriented Regression Testing OO- Regression Testing Concepts  Each class must be tested when it is developed  Each class needs a regression test  Regression tests need to have standard interfaces  Thus, we can build the regression test when building the class and have a better, more stable product for less work

Object-Oriented Regression Testing OO- Regression Testing Strategy  Based on the Class Firewall Concept Identify changed classes Identify affected classes (using dependency relation) Apply the Class Test Order strategy to perform class re- testing at the unit level Use the Class Test Order to re-integrate classes together. Select, reuse, and define test cases

Example: A Changed Class in Class Diagram X Y D T S I U AC B M N RV L

X Y D T S I U AC B M N RV L Example: A Class Firewall in Class Diagram

X Y D T S I U AC B M N RV L Example: Class Test Order for The Class Firewall 4.1 4