1. The ROI of Testing Presented By: Shaun Bradshaw Questcon Technologies sbradshaw@questcon.com The ROI of Testing Presented By: Shaun Bradshaw Questcon Technologies sbradshaw@questcon.com

2. Slide 2 Return on Investment

3. Slide 3 Return on Investment (ROI) What is ROI? ROI is a calculation that attempts to determine the actual or perceived future value of an expense or investment By calculating ROI, an organization can assess whether the expense/investment is justified by the resulting savings/revenue

4. Slide 4 Return on Investment (ROI) How is ROI calculated? In its most basic form ROI is calculated as follows: ROI = (Benefits – Costs)/Costs That is, the financial benefit after an investment or improvement is made minus the cost of the investment or improvement, calculated as a percentage of those costs Benefits Revenue generated Cost reduction Cost avoidance Productivity improvements Costs Labor expense Time expense Tool expense

5. Slide 5 What makes testing valuable? How do we get to a positive ROI? Produces information reliably grounded in observed system behavior [1] –Functions like a credit check [1] –Exposes risk and improves decision-making Extends the life of the software Improves the development process Increases end-user confidence and satisfaction Return on Investment (ROI)

6. Slide 6 Defect Prevention vs. Defect Detection

7. Slide 7 Importance of Testing Organizations rely on testers to do QA We cannot “test” quality into an application Test needs to be the “backdoor to QA” Requires a shift in the testing paradigm TestFinding Defects Defect Prevention vs. Defect Detection Measuring Quality

8. Slide 8 Rules & Results of Defect Prevention 60% Rule [2] Percentage of defects introduced in require- ments and design 85% Rule [2] Percentage of defects removed through reviews, inspections and testing 368:1 Rule [3] Defect Prevention vs. Defect Detection

9. Slide 9 DesignCodeTestProductionRequirements 30 $0 0 30 60 0 $0 30 90 0 $0 10 100 85 $166,600 0 15 $378,700 15 $212,100 # of defects introduced Accumulated defects Defects removed Cost Accumulated Cost X5X10X50X368X+ Testing Only Assume X is $100 & 100 total defects Defect Prevention vs. Defect Detection

10. Slide 10 DesignCodeTestProductionRequirements 30 $2,600 26 30 34 29 $14,500 $17,100 30 35 30 $30,000 $47,100 10 15 13 $65,000 $112,100 0 2 $185,700 2 $73,600 Full QA Assume X is $100 & 100 total defects Defect Prevention vs. Defect Detection X5X10X50X368X+ # of defects introduced Accumulated defects Defects removed Cost Accumulated Cost

11. Slide 11 Quantitative Benefits

12. Slide 12 Reduced Defect Repair Costs Test involvement earlier in the lifecycle reduces repair costs Automation improves defect detection Good test practices increase detection likelihood Quantitative Benefits

13. Slide 13 Decreased Production Costs Defects in production can be significantly higher than 368x - the “Annuity Nightmare” –Small rounding error goes undetected at a major financial institution –Months until discovered –Code repair is cheap –Production cost – NOT CHEAP! Increased Revenue & Profits Reduced time-to-market for functioning software Increased market share Higher customer retention/goodwill Reduced maintenance costs (easier to enhance) Quantitative Benefits

14. Slide 14 Calculating Test Value

15. Slide 15 Defect Injection Rate Total number of defects introduced into an application When each defect was introduced (using Root Cause Analysis) Number of defects introduced in each phase of the SDLC Calculate the defect injection rate as: # of defects introduced total defects Key Metrics to Calculate Test Value

16. Slide 16 Defect Repair Cost Hourly rate of each resource involved in the repair (PM, BA, Developer, Tester, etc.) Defect repair time for each resource by phase of the SDLC Number of defects repaired in each phase of the SDLC Calculate the defect repair costs as shown in the example*: *Data in the example was drawn from [4]. Key Metrics to Calculate Test Value

17. Slide 17 “Test” Effectiveness Number of defects found during “testing” activities during any phase of the SDLC Total number of defects found including in production (use a “warranty period”) Calculate test effectiveness as: # of defects found phase total # of defects found Key Metrics to Calculate Test Value

18. Slide 18 ROI Example

19. Slide 19 ROI Example

20. Slide 20 Established a test team made up of: 1 test lead 1 test analyst 1 test consultant Total 1 st year investment: $237,500 ROI Case Study

21. Slide 21 Conducted 2 projects in first 8 months Realized $173,000 savings in project-over-project defect repair costs Calculated savings do not include other benefits such as: Reduced production downtime Reduced maintenance costs Increased customer satisfaction & trust Anticipated 46% ROI at the end of year 1 ROI Case Study

22. Slide 22 Questions sbradshaw@questcon.com 336-273-2428

23. Slide 23 References 1.Bullock, James. “Calculating the Value of Testing.” Software Testing & Quality Engineering: May/June, 2000. 2.Jones, Capers. "Software Cost Estimating Methods for Large Projects." CrossTalk: April, 2005. 3.Dabney, JB. "Return on Investment of Independent Verification and Validation Study Preliminary Phase 2B Report." Fairmont, WV: NASA IV&V Facility, 2003. 4.US Dept. of Commerce, National Institute of Standards & Technology (NIST). “Planning Report 02-3: The Economic Impacts of Inadequate Infrastructure for Software Testing.” Technology Program Office, Strategic Planning & Economic Analysis Group. May, 2002.