SFO Technologies Pvt Ltd Test Estimation By Quantitative Project Management – A Probabilistic Approach Balamurali L – Director Quality & Testing Group Shajeena Shahul Hameed – Senior Lead Engineer QA SFO Technologies Pvt Ltd A NeST Group Company STC 2016

The product is delivered to the testing team The product is delivered to the testing team. Can plan the customer release in a month’s time Oh! No.. Lot of bugs… mmm… Atleast another month for external delivery! A month later… STC 2016

Have estimated the size of project using IFPUG Guidelines Have estimated the size of project using IFPUG Guidelines.. Productivity taken from organization data base.. Still!!! What went wrong??? STC 2016

The estimates were based only on size and organizational baseline values. Quality of work product and process followed in the project not considered!! Should have done that… STC 2016

The hard nut Estimation STC 2016

The Industry Practice Percentage of effort Work Break down structure Function Points Test Point Estimates Test Case Point Estimates Use case based estimates STC 2016

The Lacunae Quality of work product of each phase not considered Expertise assumed to be constant across the teams Test estimates to be refined based on the quality of product in each phase STC 2016

Process Performance Model Defining the relation between the dependent parameter and the various independent parameters Identify the parameters which influence the dependent parameter (Testing Effort) From the organizational data, ensure the co-relation Conduct Regression Analysis to get the relation between the dependent and independent parameters STC 2016

The PPM Approach Identify the parameters influencing Testing effort estimate Size, Expertise of team, Requirement changes, Code Complexity etc. Establish the relationship Use Historical data E.g.. Effort for Testing= 27.1 + 5.68 * Size of the product – 4.01* Expertise Index –9.4 * Requirement Stability Index + 3.14*Code Complexity STC 2016

Estimate and Refine the Test Estimate Fine-tune the data At defined frequencies As and when parameters change STC 2016

Estimate - Conventional Method Testing Productivity = Size/Testing Effort The estimated effort & actual effort differ a lot due to poor estimates, change in requirements poor quality work products STC 2016

Closed Loop Estimation Model- Initial Phase PPM based estimate using Initial Size estimate Planned Expertise of the developer Estimated Requirements Stability Index (RSI) Code Complexity (CC). STC 2016

Closed Loop Estimation Model- Project Planning Phase Resources are freeze Initial estimate refined by replacing the actual expertise index instead of estimated expertise index Expertise index - effort & vice versa Expertise index Effort STC 2016

Closed Loop Estimation Model- Along with Requirement Changes The volatility of the requirements measured a minor change in requirement  low volatile a major change in testing phase  highly volatile. Requirement Stability Index is the complement of volatility RSI - effort & vice versa RSI Effort STC 2016

Closed Loop Estimation Model- After Coding Phase Cyclomatic complexity of the code measured using the tools The actual value of complexity replaced for the estimated value there by fine tuning the total estimate Increase in Code complexity  higher the effort for testing and bug fix & vice versa. Effort for testing and bug fix Code complexity STC 2016

Advantage Better estimate which is fine tuned with the real measures of the factors influencing the total testing effort The increase (or decrease) of total testing effort is brought to the notice of management well in advance Better planning is done so that the Test manager has a better control over the chaotic phases It provides a relation with the upstream parameters of software development to the total testing effort. Strategic planning can be done for controlling these parameters using statistical process control techniques. STC 2016

Case Study – Conventional Method Size (15KLOC) Total Testing Effort for Project (100PD) Testing Productivity (150LOC/PD) STC 2016

Closed Loop Estimation Model- Initial Phase Size (15KLOC) Effort (106PD) Expertise Index (15) Estimated RSI (0.95) Using the Process Performance Model Estimated Code Complexity (20) STC 2016

Fine Tuning of Estimate Effort (126 PD) After Planning put the actual Expertise index (10) Effort (106 PD) Effort (127 PD) After Requirement Changes put the actual RSI (0.85) After Coding put the actual code complexity (28) Effort (152 PD) STC 2016

Variance 52% increase in Effort STC 2016

Scope for future work Prediction Intervals can be introduced Expertise of teams in each phase can be introduced PPMs for schedule variation, productivity, defect density..... STC 2016

22 Questions STC 2016

References & Appendix Bug Prediction and Tracking - A QPM Approach by Balamurali L and Annu George, NeST, QAI’s SEPG Conference 2003 https://www.cmcrossroads.com/sites/default/files/article/file/2013/XUS373692file1_0.pdf http://www.automation.ultimatetimepass.com/index.php/home/software-testing-estimation-techniques.html Test Effort Estimation Using Use Case Points By Suresh Nageswaran, Cognizant Technology Solutions, Pune, Quality Week 2001, San Francisco, California, USA, June 2001 Test Case Point Analysis White Paper By Nirav Patel, Muthukrishnan Govindrajan, Susmita Maharana and Shoba Ramdas From Cognizant Technology Solutions STC 2016

Author Biography Balamurali L is an M.Tech from IIT Kanpur and has an industrial experience of more than 15 years. He is working as Director- Quality and Testing Group of SFO Technologies Software Division. He has core competency in the areas of process consulting, quality assurance and control. He has played a lead role in implementing the models like CMMI Dev V1.3, and ISO9001:2008, Automotive SPICE, ISO 27001, IEC61508 and ISO 13485. He has worked as Assessment Team Member for CMMI V 1.3. A CSQA & CQSPE from QAI, he is also a Certified Tester (advanced level- Test Management) by ISTQB and plays an active role in the core group of the Trivandrum chapter of Software Process Improvement Network (SPIN). He is also a certified Test Manager by Department of IT, Govt. of India. He is a certified lead auditor for ISO 9001:2015 He has conducted many public workshops and corporate trainings on Software Engineering, Quality and Testing. He has also marked his presence in various International Conferences like QAI, STC, and ISQT by presenting papers in concepts of CMMI High Maturity, Software Testing and Automation, Quality Assurance etc. Shajeena Shahul Hameed is working as Senior Engineer-QA at SFO Technologies. She did her B-Tech from Kerala University and has more than 8 years of experience in IT industry. She is an expert in software models like CMMI and TMMI. She has also played a major role in the implementation of models like CMMI Dev V1.3, ISO 9001, ISO 13485 ISO 27001, ISO 26262 etc in the organization. She is an active member of the metrics team in organization. She is certified in ISTQB Foundation Level and Six Sigma Green Belt. She has competency in the areas of process consulting, quality assurance and control. She is a certified lead auditor for ISO 9001:2015. She has also marked her presence in various International Conferences like QAI, STC, ISQT by presenting papers in concepts of CMMI High Maturity, Software Testing, Quality Assurance etc STC 2016

Shajeena Shahul Hameed Thank You!!! Balamurali L balamurali@nestgroup.net 09847180100 Shajeena Shahul Hameed shajeena.hameed@nestgroup.net 09567738938 STC 2016