Regression Testing

Regression Testing Software maintenance is expensive Development may take 2 to 4 years Same may have to be maintained for 10 to 15 years

Regression Testing Reason for software change Errors during actual use Additional functionality Changes in external world Restructuring work Change in technologies 6. Obsolete capabilities to be deleted.

Regression Testing This retesting is called regression testing. Development testing When we modify software, we typically retest it. This retesting is called regression testing. Regression testing is the process of retesting the modified parts of the software and ensuring that no new errors have been introduced into previously tested code.

Regression Testing Purposes Increase confidence in the correctness of the modified program. Locate errors in the modified program. Preserve the quality & reliability of software. Ensure the software’s continued operation. We may perform regression testing during latter stage of software development.

Regression Testing Sr.No. Development Testing Regression Testing We can make use of existing test suites and test plans 1 We create test suites and test plan We retest modified components or affected by modifications 2 We test all software components 3 Budget give time for testing No time 4 Once on software product Many times Performed under the pressure of release date Performed in crisis situations, under greater time constraint. 5

Regression Testing Regression Test Selection Objective is to select an appropriate subset of the test suite to be run. Fragment F Fragment F’ S1. y=(x-1)*(x+1) S2. if (y=0) S3. return (error) S4. else S5. return (1/y) S5. return(1/(y-3))

Regression Testing Test Cases Test # Input Execution History t1 x = 1 S1, S2, S3 t2 x = -1 t3 x = 2 S1, S2, S5 t4 x = 0

Selective retest techniques Retest all Partition existing test suite Reusable Retestable Obsolete Unmodified code & unmodified specification Modified code & modified specification Specify incorrect Input Output relationships due to specification modification. No longer exercise the components & specifications they were designed.

Selective retest techniques classification Minimisation techniques Regression Testing First Specification obsolete test cases 2nd Coverage obsolete test cases Selective retest techniques classification Coverage techniques Minimisation techniques Safe techniques

Regression Testing Coverage Techniques Minimization Techniques They locate coverable program components that have been modified, and select test cases that exercise these components. Minimization Techniques They work like coverage techniques, except that they select minimal sets of test cases. Safe Techniques They select every test case that causes a modified program to produce different output than its original version.

Regression Testing Theoretical Foundations P : Program : modified program S : specifications for P : specifications for P’ P(i) : refers to the output of P on input i.

Regression Testing (i) : refers to the output of P’ on input i. S(i) : refers to specified output for P on input i (i) : refers to specified output for P’ on input i. T : set of test cases (test suite) created to test P

< identifier, input, output > Regression Testing A test case is a 3 - tuple, < identifier, input, output > < t, i, s(i) > Selective retest problem Find a way, making use of T, to gain sufficient confidence that P’ is correct. Select T, a set of test cases to re-execute on . 2. Test with , to establish the correctness of with respect to .

Regression Testing 3. If necessary, create , a set of new functional or structural test cases for . 4. Test with , to establish correctness of with respect to . 5. Create , a new test suite and test history for , from , and . Step 5 addresses the test suite maintenance problem: the problem of updating and storing test information.

Regression Testing Fault revealing test cases We want to select of T that will establish confidence that was modified correctly. Objective is to choose test cases from T that detect faults. A test case t detects a fault in if it causes to fail. Hence t is fault revealing for . There is no effective procedure which can select test cases of T that are fault revealing for . We may select a superset

Regression Testing Modification revealing test cases A test case t is modification revealing for P and if and only if it causes the outputs of P and to differ. Two assumptions 1. P-Correct-for –T : For each test case t in T, when P was tested with t, P halted and produced the correct outputs. 2. Obsolete – Test - Identification Determine for each test case in T, whether t is obsolete for .

Regression Testing Modification Traversing test cases Test case t is obsolete for if and only if t either specifies an input to that, according to , is invalid for , or t specifies an input output relation for . With two assumptions, we may find modification revealing test cases but it is not easy. Modification Traversing test cases A test case is modification traversing for P and if and only if : a. Executes new or modified code in b formerly executed code that has since been deleted from P.

Modification revealing Modification traversing Regression Testing obsolete Test suite T Fault-revealing Non-obsolete Modification revealing fault revealing Modification traversing

Regression Testing Execution Trace For a test case t = < n , i, s(i) > , the execution trace for t on P, ET (P(i)), is the sequence of statements < s1, s2, ……. Sn > in P that are executed when P is run on input i, listed in the order in which they are executed. We assume that in any trace ET(P(i)), the first statement is unique entry statement, and the final statement is a unique exit statement.

Regression Testing Regression test selection algorithm There are two varities of test case prioritization techniques General test case prioritization Version specific test case prioritization

Regression Testing General test case prioritization For a given program P and test suite T, we prioritize the test cases in T that will be useful over a succession of subsequent modified version of P without any knowledge of modified version. Version specific test case prioritization We prioritize the test cases in T, when P is modified to , with the knowledge of the changes that have been made in P. Objective (Version Test Case Prioritization) Execute the modified lines of code with minimum number of test cases. We consider a hypothetical program of 60 lines of code. There are 10 code coverage based test cases.

Regression Testing Execution history is given below Test Case Array T1 1, 2, 20, 30, 40, 50 T2 1, 3, 4, 21, 31, 41, 51 T3 5, 6, 7, 8, 22, 32, ,42, 52 T4 6, 9, 10, 23, 24, 33, 43, 54 T5 5, 9, 11, 12, 13, 14, 15, 20, 29, 37, 38, 39 T6 15, 16, 17, 18, 19, 23, 24, 25, 34, 35, 36 T7 26, 27, 28, 40, 41, 44, 45, 46 T8 46, 47, 48, 49, 50, 53, 55 T9 55, 56, 57, 58, 59 T10 3, 4, 60 Test Case Array

Regression Testing Suppose 1,2,5,15,35,45 and 55 Total : 8 are modified. Simplest is to execute all test cases that have any of the modified lines of code. Selected test cases : T1, T2, T3, T5, T6, T7, T8 and T9 Total : 8

Regression Testing Test cases Modified lines No. of matches T1 1, 2 2 5 1 T4 -- T5 5, 15 2 T6 15, 35 2 T7 45 1 T8 55 1 T9 55 1 T10 --

Regression Testing Number of matches found (NFOUND) is stored in an array and elements are stored in descending order. The test case that has the maximum number of matches is selected by making its candidate value = 1. This test case will be executed first & is shown on next slide.

Test cases in descending order Regression Testing Test cases No. of matches (nfound) Matches found Candidate T1 2 1, 2 1 T5 2 5, 15 T6 2 15, 35 T2 1 1 T3 1 5 T7 1 45 T8 1 55 T9 1 55 Test cases in descending order

Regression Testing T1 is selected for execution and it covers line nos. 1 and 2 LOCs that are still to be executed: [ 1, 2, 5, 15, 35, 45, 55 ] – [ 1, 2 ] = [5, 15, 35, 45, 55] Again, we check for the number of modified lines of code covered by each test case and sort them in descending order & select the one with maximum number of matches.

Regression Testing Test cases No. of matches (nfound) Matches found Candidate T5 2 5, 15 1 T6 2 15, 35 T3 1 5 T7 1 45 T8 1 55 T9 1 55

Regression Testing T5 is selected for execution and it covers line nos. 5 and 15. LOCs that are still to be executed: [ 5, 15, 35, 45, 55 ] – [ 5, 15 ] = [35, 45, 55] Again, we check for the number of modified lines of code covered by each test case and sort them in descending order & select the test case with maximum number of matches found.

Regression Testing Test cases No. of matches (nfound) Matches found Candidate T6 1 35 1 T7 1 45 T8 1 55 T9 1 55

LOCs still to be executed : [ 45, 55 ] Regression Testing T6 is selected and covers line no. 35. LOCs still to be executed : [ 45, 55 ] Test cases No. of matches (nfound) Matches found Candidate T7 1 45 1 T8 1 55 T9 1 55

LOCs still to be executed : [ 55 ] Regression Testing T7 is selected for execution covering line no. 45. LOCs still to be executed : [ 55 ] Test cases No. of matches (nfound) Matches found Candidate T8 1 55 1 T9 1 55 T8 is selected for execution Order : T1, T5, T6, T7, T8

Regression Testing Second Case Out of 10 test cases, we need to run only 5 test cases for 100% code coverage of modified lines of code. 50% saving of test case Second Case Lines of code have been modified and also deleted. Example has 20 lines of code. 5 Test Cases as given on next slide.

Regression Testing Execution History Test Cases 1, 5, 7, 15, 20 T2 2, 3, 4, 5, 8, 16, 20 T3 6, 8, 9,10, 11, 12, 13, 14, 17, 18 T4 1, 2, 5, 8, 17, 19 T5 1, 2, 6, 8, 9, 13 Modified lines : 6, 13, 17, 20 Deleted lines : 4, 7, 15

Regression Testing Execution History From the test history information, remove the deleted lines say : 4,7,15. Test case array will be: Test Cases Execution History T1 1, 5, 20 T2 2, 3, 5, 8, 16, 20 T3 6, 8, 9,10, 11, 12, 13, 14, 17, 18 T4 1, 2, 5, 8, 17, 19 T5 1, 2, 6, 8, 9, 13

Regression Testing Now, we want to find redundant test case. This situation has become important due to deletion of few lines., If we see execution history carefully. We come to know that T1 & T5 are redundant test cases.

Regression Testing Test cases Line numbers Found in test case Redundant Y/N T1 1 T4 Y 5 T2 Y 20 T2 Y T5 1 T4 Y 2 T2 Y 6 T3 Y 8 T3 Y 9 T3 Y 13 T3 Y

The remaining test cases are = T2, T3, T4 Regression Testing The remaining test cases are = T2, T3, T4 Test case array is: Test Cases Execution History T2 2, 3, 5, 8, 16, 20 T3 6, 8, 9, 10, 11, 12, 13, 14, 17, 18 T4 1, 2, 5, 8, 17, 19

No. of matches found (nfound) Regression Testing Number of matches for modified lines are Modified lines are : 6, 13, 17, 20 Test cases Matches found No. of matches found (nfound) T2 20 1 T3 6, 13, 17 3 T4 17 1

Regression Testing Test cases are sorted on the basis of matches found (descending order) Test cases No. of matches found Line numbers Candidate T3 3 6, 13, 17 1 T2 1 20 T4 1 17

Regression Testing T3 is selected for execution. Remaining modified line is : 20 Hence T3 and T2 are sufficient to execute modified lines. T1 & T5 should be deleted from the test suite. Saving : 60%