Improving Effectiveness of Regression Testing of Telecommunications Systems Software Sami Torniainen Supervisor: Professor Raimo Kantola

Outline Goal Background How to improve cost-effectiveness? Methods Implementation Evaluation Further research

Goal Goal: improve cost-effectiveness of regression testing at the target company Three sub-objectives: –explore present regression testing practices –implement methods –evaluate empirically

Background - Target Company Finnish telecommunications systems developer Located in Espoo Main product: IP/MPLS routers Customers: global mobile and internet service providers

Background – V-model

Background – Regression testing “Regression testing is selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements.” (IEEE) Can occur on any level of the V-model This thesis focuses on regression testing on integration testing level

How to improve cost-effectiveness? Costs accumulate from time and resources consumed in executing tests Two approaches mainly utilized in literature –test selection techniques –test automation

Test selection techniques Reduce the number of tests Mainly based on code exploration –unit testing Also exists risk-based method –integration testing level

Test automation Automating manual working phases –creating executable test programs i.e. scripts Popular method Expensive to utilize

Implementation Limited scope (10 test suites) Risk-based test selection technique –evaluating test suites based on risk-analysis –executing only five test suites of ten Test automation –automating the control of Adtech AX-4000 –Tcl API –no manual intervention needed during test execution

Influence on regression test process

Evaluation – testing effort

Evaluation – detected errors

Evaluation – deploying effort

Evaluation - Summary Test automation reduced testing effort significantly –but was expensive to implement (94%) Risk-based test selection method also reduced testing effort –but will likely miss errors

Conclusions Despite the great implementation costs, test automation is recommended –break-even ”only” after 34 test rounds Benefits –reduced testing effort –safer –reduce the overall testing elapsed time –potentially earlier time to market

Further research Develop Tcl API –reloading of GUI-based test setups –would reduce scripting effort Optimizing test coverage of risk-based test selection technique –threshold –minimizing probability to miss errors Vs. reducing the number of test suites

Thank You! Questions?