Firewall for regression testing Tor Stålhane According to White, Jaber, Robinson and Rajlich

Why fire walls in regression testing Most regression tests are large and time- consuming to run. The concept of a “fire wall” is used to reduce the set of classes – or components – that need to be tested. A fire wall in regression testing separates the classes that depend on the class that is changed from the rest. There are two central concepts – dependency and encoding

Simple fire wall rules Rules for the fire wall in object-oriented systems: 1.Given two successive versions of an OO-system, identify the classes that have changed 2.If a changed class is part of an inheritance hierarchy we must also consider descendants of the changed class as changed 3.For each changed class, identify all classes that send a message to the changed class or receive messages from a changed class and include them inside the fire wall

Simple fire wall example

The extended fire wall The extended fire wall is identified by the three rules for constructing a simple fire wall plus 4.Identify all data paths to and from a modified class 5.Find all external classes in scope of the modified class and include them in the extended fire wall In order to identify an external class, we need some definitions.

Dependency The main concept to understand when we apply the fire wall idea, is the idea of dependency. The main question is: Is the component dealing with the plain values of the input or Must the component account for how those inputs were generated

Definitions – 1 Let C be a component with inputs {I i }, outputs {O j } and post-conditions {Q k } If Q  {Q k } and I  {I i } so that Q depends on I and on a value generated by a component C 1 that is connected to I through a data flow through one or more intermediary component C 2,…C m then I is an encoded input C is an external component in scope of C 2 and visa versa

Definitions – 2 No component C 1 => I is a plain input, i.e. not encoded All inputs of C are plain => C is an I/O component C is external in scope of D and there is no message sent from C to D => there is a hidden dependency between C and D. c c1c1 c c2c2 c1c1 I I I is a plain input I is an encode input cmcm …

Dependency example – 1 Component 1 generates the key k, which is converted to an integer in component 2. Component 3 converts the integer to a hash address A. Component 4 uses the hash address to do a table look-up to generate the value f(k).

Dependency example – 1 The post condition of component 4 depends on k, generated by component 1 => input A is encoded by component 2. Component 4 is an external in scope of component 2 and visa versa. r = f(k) but k is changed by component 2

Hidden dependency A’s post-condition is dependent on the value “colour” that is Generated in class I Encoded as an integer in class A Decoded in class B Thus class B is external in scope of A. The dependency between A and B is hidden since no message is sent between A and B

Global variables Here, the output v 3 from component 3 is used to modify the global variable g which is used as input to component 4. Component 4 is external and dependent on variable v 1 and is activated by a message from component 3 to component 4.

Extended fire wall – example

Simpel and Extended fire wall – 1 In order to test the concepts of Simple and Extended Fire wall, the authors tested several builds of a TelComm system. This system had already been tested by the developers using their standard test suite. Common – found using both TFW and EFW: 8 errors Found only using EFW: 4 errors. Two of these had not been found by the company’s test suite.

Simpel and Extended fire wall – 2

Simpel and Extended fire wall – 3 In the following table, total time for testing is the sum of Analysis time – identifying the fire wall Testing time is time needed for – test setup – test execution – analysis of the test results. EFW testing requires approximately 20 – 30% more tests 40 – 50% more person hours

Simpel and Extended fire wall – 4 TFW: 46 person hours, 8 errors => 5.7 person hours per error found EFW: 58 person hours, 12 errors => 4.8 person hours per error found

Simpel and Extended fire wall – 5 For TFW some testes were reused from the company’s original test suite All extra tests needed for EFW were written from scratch.

Simpel and Extended fire wall – 6 New system – 13 common errors and 3 errors found only by EFW

Simpel and Extended fire wall – 7 TFW: 93 person hours, 13 errors => 7.2 person hours per error found EFW: 120 person hours, 16 errors => 7.5 person hours per error found

Conclusion Extended Fire Wall testing (EFW) finds more errors that Simple Fire Wall (FWT) testing The cost in person hours per error found is approximately the same – 5 to 8 person hours all in all per error Both EFW and FWT – should only be used to test updates and changes – assumes that the initial test suite and the tested software are of a high quality.