copyright A. Andrews, 2003 Regression Testing Department of Computer Science

copyright A. Andrews, 2003 Regression testing Testing changed software –Retest all –Selective regression testing Regression test suite Types of changes –Defect fixes –Enhancements –Adaptations –Perfective maintenance

copyright A. Andrews, 2003 Retest all Assumption: –Changes could have introduced errors anywhere in the code –Expensive, prohibitive for large systems –Reuse existing test suite –Add new tests as needed –Remove obsolete tests (discrepancies between expected and actual output)

copyright A. Andrews, 2003 Selective regression testing Impact analysis Only code impacted by change needs to be retested Select tests that exercise such code Add new tests if needed Remove obsolete tests

copyright A. Andrews, 2003 Regeneration strategy Often for defect fixes when unsure whether defect is really fixed Develop new tests –Probe around defect fix –Test affected functionality Comes in retest all, and selective regression testing flavors

copyright A. Andrews, 2003 Regression testing process Identify changes Determine which tests will remain valid: T’ Test modified software with T’ If T’ does not meet test criterion, generate new test cases T’’ Execute modified software with T’’

copyright A. Andrews, 2003 Regression testing process (2) T’: –test whether modifications have broken anything –Determined either by retest all, or selective strategy T’’: –test whether modified code works Use scope appropriate criteria

copyright A. Andrews, 2003 Classification of tests in T Reusable: –does not test software modification –Should produce same result –Need not be rerun Retestable: –Tests software modification –Must be rerun Obsolete: no longer applies

copyright A. Andrews, 2003 Practical regression testing ApproachQualityChangeImpactSchedule Full newLowKey CodeExtensiveNone MinimalHighIsolatedLocalizedVery tight ExpandedModerateKey CodeExtensiveModerate Full reuseLowSeries of fixes ExtensiveModerate

copyright A. Andrews, 2003 Approaches Full new test cycle –Redesign entire test plan –Rebuild whole test suite Minimum regression test –Reuse test plan as much as possible –Rerun minimum number of retestable test cases –Generate minimum number of new test cases for changes

copyright A. Andrews, 2003 Approaches (2) Expanded scope of regression testing –Reuse part of test plan –Rerun all retestable test cases –Generate new test cases on the full scope of changes Full reuse of existing test suite –Reuse test plan and test suite

copyright A. Andrews, 2003 Attributes of regression tests Safe: –All tests that could possibly exhibit different outputs on the modified software –Is 100% inclusive Precision: –Percentage of tests in a reduced test suite that cannot reveal regression faults, and –Are not selected for the reduced test suite

copyright A. Andrews, 2003 Attributes of regression tests (2) Efficiency: cost of identifying a reduced regression test suite Generality: range of application for the selection strategy

copyright A. Andrews, 2003 Regression test patterns Retest changed code –Class, cluster, subsystem scope –Identify segment coverage by test case –Use version control tool to identify changed segments –Select tests covering changed/deleted segments –Develop new tests for new segments Safe

copyright A. Andrews, 2003 Retest by profile Assumes operational profile Unsafe Include critical use cases Include other use cases by frequency in operational profile No impact analysis Better: determine impact on use case

copyright A. Andrews, 2003 Retest risky use cases Suspicious use cases –Use cases that depend on components, objects, middleware, resources that are Unstable, unproven Have not been shown to work together before Implement complex business rules Are complex Were fault-prone during development Critical use cases Unsafe

copyright A. Andrews, 2003 Retest within Firewall Firewall: set of components whose test cases will be included in a regression test External interface, contract change Internal implementation change