Department of Computer Science Regression Testing

Regression testing Testing changed software Regression test suite Retest all Selective regression testing Regression test suite Types of changes Defect fixes Enhancements Adaptations Perfective maintenance

Retest all Assumption: Changes could have introduced errors anywhere in the code Expensive, prohibitive for large systems Reuse existing test suite Add new tests as needed Remove obsolete tests (discrepancies between expected and actual output)

Selective regression testing Impact analysis Only code impacted by change needs to be retested Select tests that exercise such code Add new tests if needed Remove obsolete tests

Regeneration strategy Often for defect fixes when unsure whether defect is really fixed Develop new tests Probe around defect fix Test affected functionality Comes in retest all, and selective regression testing flavors

Regression testing process Identify changes Determine which tests will remain valid: T’ Test modified software with T’ If T’ does not meet test criterion, generate new test cases T’’ Execute modified software with T’’

Regression testing process (2) test whether modifications have broken anything Determined either by retest all, or selective strategy T’’: test whether modified code works Use scope appropriate criteria

Classification of tests in T Reusable: does not test software modification Should produce same result Need not be rerun Retestable: Tests software modification Must be rerun Obsolete: no longer applies

Practical regression testing Approach Quality Change Impact Schedule Full new Low Key Code Extensive None Minimal High Isolated Localized Very tight Expanded Moderate Full reuse Series of fixes

Approaches Full new test cycle Minimum regression test Redesign entire test plan Rebuild whole test suite Minimum regression test Reuse test plan as much as possible Rerun minimum number of retestable test cases Generate minimum number of new test cases for changes

Approaches (2) Expanded scope of regression testing Reuse part of test plan Rerun all retestable test cases Generate new test cases on the full scope of changes Full reuse of existing test suite Reuse test plan and test suite

Attributes of regression tests Safe: All tests that could possibly exhibit different outputs on the modified software Is 100% inclusive Precision: Percentage of tests in a reduced test suite that cannot reveal regression faults, and Are not selected for the reduced test suite

Attributes of regression tests (2) Efficiency: cost of identifying a reduced regression test suite Generality: range of application for the selection strategy

Regression test patterns Retest changed code Class, cluster, subsystem scope Identify segment coverage by test case Use version control tool to identify changed segments Select tests covering changed/deleted segments Develop new tests for new segments Safe

Retest by profile Assumes operational profile Unsafe Include critical use cases Include other use cases by frequency in operational profile No impact analysis Better: determine impact on use case

Retest risky use cases Suspicious use cases Critical use cases Unsafe Use cases that depend on components, objects, middleware, resources that are Unstable, unproven Have not been shown to work together before Implement complex business rules Are complex Were fault-prone during development Critical use cases Unsafe

Retest within Firewall Firewall: set of components whose test cases will be included in a regression test External interface, contract change Internal implementation change