Firewalls for regression testing Spring 2013

Use of the word “Firewalls” The way the authors White, Jaber, Robinson and Rajlich use the term “Firewall” in this work is special for their work on regression testing and must not be mixed with the common use of the term “firewall” which is part of networks and security.

Why firewalls in regression testing Most regression tests are large and time- consuming to run. The concept of a “firewall” is used to reduce the set of classes – or components – that need to be tested. A firewall in regression testing separates the classes that depend on the class that is changed from the rest of the classes. There are two central concepts – dependency and encoding

Dependency The main concept to understand when we apply the firewall idea is dependency. The main question is: Is the component dealing with the plain values of the input or Must the component account for how those inputs were generated

Definitions – 1 Let C be a component with inputs {I i } outputs {O j } post-conditions {Q k } If Q  {Q k } and I  {I i } so that Q depends on I and on a value generated by a component C 1 that is connected to I through a data flow through one or more intermediary component C 2,…C m then I is an encoded input C is an external component in scope of C 2 and visa versa

Definitions – 2 No such component C 1 => I is a plain input, i.e. not encoded All inputs of C are plain => C is an I/O component C is external in scope of D and there is no message sent from C to D => there is a hidden dependency between C and D. c c1c1 c c2c2 c1c1 I I I is a plain input I is an encoded input cmcm …

Dependency example – 1 Component 1 generates the key k, which is converted to an integer in component 2. Component 3 converts the integer to a hash address A. Component 4 uses the hash address to do a table look-up to generate the value f(k). If component 2 or 3 is changed, the result r may change.

Dependency example – 2 The post condition of component 4 depends on k, generated by component 1 => input A is encoded by component 2. Component 4 is external in scope of component 2 and visa versa. r = f(k) but k is changed by component 2 There is a hidden dependency between 2 and 4

Example of hidden dependency A’s post-condition is dependent on the value “colour” that is – Generated in class I – Encoded as an integer in class A (0 is red, 1 is yellow etc.) – Decoded in class B (receives colour code, decodes and paints the screen) Thus class B is external in scope of A. The dependency between A and B is hidden since no message is sent between A and B

Global variables The output v 3 from component 3 is used to modify the global variable g which is used as input to component 4. Component 4 is external and dependent on variable v 1 and is activated by a message from component 3 to component 4 which has no data associated with it Activation path: 1, 2, 3, 4 Messages path: 1, 2, 3, g, 4

Simple firewall example: a TFW TFW does not examine dependencies further away than one level from changed component

Process for determining TFW in OO systems 1.Given two successive versions of an oo-system, find the difference of the two versions and identify those classes that have changed 2.If any of the changed classes are in an inheritance hierarchy, also consider descendants of the changed class as changed 3.For each changed class, identify all the classes that send messages to the changed class or receive messages from the changed class and include them in the TFW See previous slide for a graph

Extended Testing Firewall - ETW EFW extends the TFW by taking into account the data flows, external components and hidden dependencies (see previous slides) EFW is obtained by following the process: – First apply steps 1-3 in order to obtain TFW 4.Identify all data paths from and to a modified class 5.Find all external classes in scope of the modified class (E in the example on next slide). Include all such classes in EFW

Extended firewall – example

Difference between TFW and EFW

Empirical Study 1: TFW vs. EFW In order to compare the concepts of Simple (TFW) and Extended Firewall (EFW), the authors tested several builds of a TelComm system. – 66 classes, lines of code, C++ This system had already been tested by the developers using their standard test suite. – Common – found using both TFW and EFW: 8 errors – Found only using EFW: 4 errors. Two of these had not been found by the company’s test suite.

TelComm system- results

TelComm system: Severity of faults All faults in previous slide were of severity 3 except for the EFW fault in build 1.1 which is of severity 2 No faults of severity 1 or 4

TelComm system: Extent of extra effort In the table on next slide, total time for testing is the sum of – Analysis time – identifying the firewall – Testing time is the time needed for test setup test execution analysis of test results. – EFW testing requires approximately 20 – 30% more tests 40 – 50% more person hours

TelComm system: Extent of extra effort TFW: 46 person hours, 8 errors => 5.7 person hours per error found EFW: 58 person hours,12 errors => 4.8 person hours per error found

TelComm system: Reused vs. new tests For TFW some testes were reused from the company’s original test suite All extra tests needed for EFW were written from scratch.

Empirical Study 2 TFW vs. ETF An empirical study was also performed on a very large system – more than 1 mill lines of code – thousands of classes – C++ code – Real time system – Windows environment

Real time system: TFW vs. EFW 13 common faults and 3 faults found only by EFW Version 1: Common faults: 3 at sev. level 4, 1 at level 3, 1 at level 2 (EFW) Version 2: Common faults 8 (6 at lev. 4, 2 at lev. 2), EFW faults, one at level 3 one at level 2

Real time system: number of tests and effort TFW: 93 person hours, 13 errors => 7.2 person hours per error found EFW: 120 person hours, 16 errors => 7.5 person hours per error found

Conclusion Extended Firewall testing (EFW) finds more faults than Simple Fire Wall (TFW) testing – 50% in the first study, 23 % in the second study The cost in person hours per fault found is approximately the same: person hours per fault EFW and TFW – should only be used to test updates and changes – assumes that the initial test suite exists and has a high quality

Reference Most of the information on these slides are from the paper (available from It’s learning) “Extended firewall for regression testing: an experience report” By: Lee White, Khaled Jaber, Brian Robinson and Václac Rajlich From: Journal of Software Maintenance and Evolution: Research and Practice, 2008, 20: