Nov 6, 2008 Presented by Amy Siu and EJ Park

Application Release 1 R1 Test Cases Application Release 2 R2 Test Cases R1 Test Cases 2 Regression testing is expensive! Validate modified software Often with existing test cases from previous release(s) Ensure existing features are still working

A strategy to Minimize the test suite Maximize fault detection ability Considerations and trade-offs Cost to select test cases Time to execute test suite Fault detection effectiveness 3

Regression test case selection techniques affect the cost-effectiveness of regression testing Empirical evaluation of 5 selection techniques No new technique proposed 4

Application Release 1 R1 Test Cases Application Release 2 PP' T T'T'' T''' Programs: P, P' Test suite: T Test cases: T T New test cases: T'' for P New test suite: T''' for P including selection from T Regression test selection problem 5

5 test case selection techniques Minimization Dataflow Safe Ad Hoc / Random Retest-All 6

Minimization Dataflow Safe Ad Hoc / Random Retest-All Select minimal sets of test cases T' Only cover modified or affected portions of P – '81 Fischer et. al – '90 Hartman and Robson 7

Select test cases T' that exercise data interactions that have been affected by modifications in P' – '88 Harrold and Soffa – '88 Ostrand and Weyuker – '89 Taha et. al 8 Minimization Dataflow Safe Ad Hoc / Random Retest-All

Guarantee that T' contains all test cases in T that can reveal faults in P' – '92 Laski and Szermer – '94 Chen et. al – '97 Rothermel and Harrold – '97 Vokolos and Frankl 9 Minimization Dataflow Safe Ad Hoc / Random Retest-All

Select T' based on hunches, or loose associations of test cases with functionality 10 Minimization Dataflow Safe Ad Hoc / Random Retest-All

Select all the test cases in T to test P' 11 Minimization Dataflow Safe Ad Hoc / Random Retest-All

How techniques differ? The ability to reduce regression testing cost The ability to detect faults Trade-offs between test size reduction and fault detection The Cost-effectiveness comparison Factors affect the efficiency and effectiveness of test selection techniques 12

Calculating the cost of RST (Regression Test Selection) Techniques They measure Reduction of E(T) by calculating the size reduction Average of A by simulating on several machines 13 A: The cost of analysis required to select test cases E(T): The cost of executing and validating the selected test cases

On a Per-Test-Case Basis Effectiveness = # of test cases revealing fault of P in T, but not in T On a Per-Test-Suite Basis Classify the result of test selection (1)No test case in T is fault revealing then T too, or (2)Some test cases in T and T both revealing fault, or (3)Some test cases in T is revealing fault, but not in T. Effectiveness = 1 – (% of no fault revealing test cases) 14 Their choice

Programs: All C programs The Siemens Programs: 7 C programs Space: Interpreter for an array definition language Player: Subsystem of Empire (Internet game) 15 Programs Faulty version How do the authors create test pool and suite?

Siemens Programs Constructing test pool of black-box test cases from Hutchins et al. Adding additional white-box test cases Space test cases from Vokolos and Frankl, randomly generated Adding new test cases from executing CFG Player 5 different unique version of player – named base version Creating own test cases from Empire information files 16 Programs Test Pool Design

17 P1 … P8 … … Siemens / Space Test Pool TC1TC2TC3 ……… T p (E) Test Suites for each program Random Number Generator … Player command2 TC1TC3 command1 Random Selection TC2 Programs Test Pool Design Test Suite Design Siemens: 0.06%~19.77% Space: 0.04%~94.35% Player: 0.77%~4.55%

Minimization Created simulator tool Dataflow Simulating dataflow testing tool Def-use pairs affected by modification Safe DejaVu: Rothermel and Harrolds RTS algorithm Detect dangerous edge Aristole: program analysis system Random: n % of test cases from T randomly 18 Only for Siemens

Variables Independent 9 Programs (Siemens, Space and Player) RTS technique (safe, dataflow, minimization, random(25, 50, 75), retest-all Test suite creation criteria Dependent The average reduction in test suite size Fault detection effectiveness Design Test suites: 100 coverage-based random 19

Internal Instrumentation effects can bias results They run each test selection algorithm on each test suite and each subject program External Limitation to generalize results to industrial practice Small size/simple fault pattern of test programs Only for corrective maintenance process Construct Adequate measurement Cost and effectiveness measurement is too coarse! 20

Comparison1 Test Size Reduction Fault Detection Effectiveness Comparison2 Program Analysis Based Techniques minimization, safe, and data-flow Random Technique 21

22 Random Techniques: Constant percentage of test casesMinimization: Always choose 1 test caseSafe and Dataflow: Similar behavior on SiemensSafe: Best on Space and Player

23 Random Techniques: Effectiveness increased by test suit sizeRandom Techniques: Increase rate diminished as size increased.Minimization: overall had the lowest effectivenessSafe & Dataflow: Similar median performance on Siemens

24 Random Techniques -Effective general -Selection Ratio Effectiveness Increase Rate Minimization -Reduction is very high -Various Effectiveness Safe -100% Effectiveness -Various Test Suite Size Dataflow -100% Effectiveness too Not safe Minimization vs. Random Assumption: k value = analysis time Comparison Method Start from a trial value of k Choose test suite from minimization Choose |Test suite| + k test suits from random Adjust k until the effectiveness is equal Comparison Result For coverage-based test suite: k = 2.7 For random test suite: k = 4.65 Safe vs. Random Same assumption about k Find k to make fixed 100(1-p)% of fault detect of Random techniques Comparison Results Coverage-based k =0, 96.7% k = 0.1, 99% Random k = 0, 89% k = 10, 95% k = 25, 99% Safe vs. Retest-all When Safe is desirable? Analysis cost is less than running the unselected test cases Test suite reduction depends on program

Minimization Smallest code size but least effective on the average applies to long-run behavior The number of test cases to choose depends on run-time Safe and Dataflow Nearly equivalent average behavior in cost-effective Safe is better than Dataflow, why? When dataflow is useful? Better analysis required for Safe Random Constant percentage of size reduction Size, fault detect effectiveness Retest-All No size reduction, 100% fault detect effectiveness 25

(1) Improve Cost Model with Other Factors (2) Extend analysis to Multiple Types of Faults (3) Develop Time-Series-Based Models (4) Scalability with More Complex Fault Distribution 26 Current Paper Java Software[1] Test Prioritization [2] With more factors [3],[4] Using Field Data [5],[6] 2004 Larger Software[7] 2005 Larger and complex Software[8] 2006 Improved Cost Model [9] Multiple Types of Faults [10] papers4 papers

[1] Mary Jean Harrold, James A. Jones, Tongyu Li, Donglin Liang, Alessandro Orso, Maikel Pennings, Saurabh Sinha, Steven Spoon, Regression Test Selection for Java Software, OOPSLA 2001, October [2] Jung-Min Kim, Adam Porter, A history-based test prioritization technique for regression testing in resource constrained environments, 24th International Conference on Software Engineering, May [3] A. G. Malishevsky, G. Rothermel, and S. Elbaum, Modeling the Cost-Benefits Tradeoffs for Regression Testing Techniques, Proceedings of the International Conference on Software Maintenance, October [4] S. Elbaum, P. Kallakuri, A. Malishevsky, G. Rothermel, and S. Kanduri, Understanding the Effects of Changes on the Cost-Effectiveness of Regression Testing Techniques, Technical Report , Department of Computer Science and Engineering, University of Nebraska -- Lincoln, July 2002 [5] Alessandro Orso, Taweesup Apiwattanapong, Mary Jean Harrold, Improving Impact Analysis and Regression Testing Using Field Data. RAMSS 2003, May [6] Taweesup Apiwattanapong, Alessandro Orso, Mary Jean Harrold, Leveraging Field Data for Impact Analysis and Regression Testing, ESEC9/FSE , September [7] Alessandro Orso, Nanjuan Shi, Mary Jean Harrold, Scaling Regression Testing to Large Software Systems, FSE 2004, November [8] J. M. Kim, A. Porter, and G. Rothermel, An Empirical Study of Regression Test Application Frequency, Journal of Software Testing, Verification, and Reliability, V. 15, no. 4, December 2005, pages [9] H. Do and G. Rothermel, An Empirical Study of Regression Testing Techniques Incorporating Context and Lifecycle Factors and Improved Cost-Benefit Models, FSE2006, November 2006 [10] H. Do and G. Rothermel, On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques, IEEE Transactions on Software Engineering, V. 32, No. 9, September 2006, pages