1. Improving Effectiveness of Regression Testing of Telecommunications Systems Software Sami Torniainen 19.2.2008 Supervisor: Professor Raimo Kantola

2. Outline Goal Background How to improve cost-effectiveness? Methods Implementation Evaluation Further research

3. Goal Goal: improve cost-effectiveness of regression testing at the target company Three sub-objectives: –explore present regression testing practices –implement methods –evaluate empirically

4. Background - Target Company Finnish telecommunications systems developer Located in Espoo Main product: IP/MPLS routers Customers: global mobile and internet service providers

5. Background – V-model

6. Background – Regression testing “Regression testing is selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements.” (IEEE) Can occur on any level of the V-model This thesis focuses on regression testing on integration testing level

7. How to improve cost-effectiveness? Costs accumulate from time and resources consumed in executing tests Two approaches mainly utilized in literature –test selection techniques –test automation

8. Test selection techniques Reduce the number of tests Mainly based on code exploration –unit testing Also exists risk-based method –integration testing level

9. Test automation Automating manual working phases –creating executable test programs i.e. scripts Popular method Expensive to utilize

10. Implementation Limited scope (10 test suites) Risk-based test selection technique –evaluating test suites based on risk-analysis –executing only five test suites of ten Test automation –automating the control of Adtech AX-4000 –Tcl API –no manual intervention needed during test execution

11. Influence on regression test process

12. Evaluation – testing effort

13. Evaluation – detected errors

14. Evaluation – deploying effort

15. Evaluation - Summary Test automation reduced testing effort significantly –but was expensive to implement (94%) Risk-based test selection method also reduced testing effort –but will likely miss errors

16. Conclusions Despite the great implementation costs, test automation is recommended –break-even ”only” after 34 test rounds Benefits –reduced testing effort –safer –reduce the overall testing elapsed time –potentially earlier time to market

17. Further research Develop Tcl API –reloading of GUI-based test setups –would reduce scripting effort Optimizing test coverage of risk-based test selection technique –threshold –minimizing probability to miss errors Vs. reducing the number of test suites

18. Thank You! Questions?