Economics of automation in functional testing of network service platforms Supervisor: Prof. Heikki Hämmäinen Instructor: M.Sc. Reima Kaitajärvi Mikko Lehtonen, Nokia Networks

Contents Background Research Problem and Research Methods Testing, overview Test automation Cost efficiency model Flexi Platform, overview Results of the case study Conclusions and further study

Background Test automation is an intriguing concept, because it includes a promise of more comfortable testing. The business interests in automation are clear, e.g. to reduce the testing effort, enable earlier release for publication and save testing costs

Research Problems and Research methods –How test automation can be used to improve testing in FlexiServer’s subcomponent Basic IP Services? –How to analyze costs and benefits of test automation in FlexiServer’s subcomponent Basic IP Services? Methods: A literature study was performed to gain the required know-how of the problem domain. Case-study was performed to solve the exact research problems.

Testing, overview The main goal of testing is to verify the quality of the product Testing is consuming about 50% of development resources and effort in many software projects As the amount of source code increases significantly, the testing costs seem to increase even more  need to cut costs

Test Automation (TA) One way to enhance the cost efficiency of testing Problem: –Test automation is not always an appropriate solution –The cost efficiency of TA is difficult to quantify due to the intangible nature of many variables (e.g. quality, improved professionalism)

The model connects three perspectives: A procedure for choosing the most cost effectively automated test areas (cost efficiency calculations) defines ways to analyze the quality of automated testing provides guidelines for implementation of the testware Solution: the cost efficiency model

Cost efficiency calculations Formulas concentrate on tangible factors i.e. financial values Intangible values must be considered separately Two possible formulas:

Calculations (cont.) The most difficult thing is to chooce the correct cost factors for a given situation (formulas quite simple) Principles on how to calculate the additional costs of manual and automated testing

FlexiServer arhitechture The BIPS implemented mostly in Linux + some in Nokia extensions (e.g. SCTP stack) The BIPS implements protocols that enable the communication between network elements (IP, TCP, UDP…)

The results Reasons for the differences: 1)Implementation took more time than estimated 2)Differences in manual test time (actual_1>average)

Results (cont.) The average manual testing time is used (probably more realistic) The figure on right side is calculated using discounted future costs (NPV)

Summary result of BIPS The automation of BIPS was a quite small scale project (2 person months) –Obtained financial savings also small, –intangible benefits (e.g. improved professionalism) support financial savings –The results are dependent from the test environment Some of the tests of BIPS have to be still performed manually.

Main conclusions The main results: –the automation of BIPS –the information of the economy of automation in that environment –and the toolkit for analysing the cost efficiency of automation in a defined project Some abstracted results: The general environment (organization,SUT,test processes,test environment) affects to the economy of TA The suitable candidates for automation are tests or supporting activities that are executed often, have a long lifetime and take long time to perform manually. The choice what to automate first is essential, because the implementation of TA takes that much time that not all the test can be automated in the normal schedule of a test organization.

Further study The effects of automation to the quality of testing (and product!) and test thoroughness How the possible saved testing time is consumed?

Questions? Thank you for your interest!