Can we calculate the profit of testing? Jurian van de Laar Philips TASS 10th Dutch Testing Day October 8th, Leiden Version 2.0

Philips TASS, 20/10/20152 Agenda Introduction Why talk about ‘profit’ of testing ? Consequences of insufficient testing Profit of testing: calculation example Supporting practices Profit calculation example refined Conclusions

Philips TASS, 20/10/20153 Introduction Philips TASS International –Independent subsidiary of Philips Electronics –Technical and embedded software Philips TASS Netherlands –Founded in 1978, Location Eindhoven –160 software specialists –HW/SW Interfacing, Remote Services, Integration & Testing (I&T) TASS I&T task force: –Increase I&T knowledge (ISEB) –Sharing I&T best practices –Enable customers to *compose reliable embedded software products *reduce the risk of product failure.

Philips TASS, 20/10/20154 Software testing in our line of business Large quantities, complex or safety critical Changing software in operation is difficult Driven by TTM (market-windows) Commitment, metrics, change control Rigorous testing vs strategy / techniques

Philips TASS, 20/10/20155 Why talk about ‘profit’ of testing ? To quantify the benefits of testing *Emphasis mostly on cost ; benefits hard to explain *Failures after production: taken into consideration? *Early fault detection contributes to total profit To make test process more efficient *‘Calculated risks’ sometimes not really calculated *Finding severe faults in testing prevents high costs

Philips TASS, 20/10/20156 The ‘profit’ of testing … Cost of testingCost of failure Profit: prevented cost of failure - cost of testing So, testing is important …but expensive. Rigorous … but exhaustive is impossible. ‘Bottleneck’ for TTM … a ‘calculated’ risk. How much to invest in ‘sufficient’ testing?

Philips TASS, 20/10/20157 Source: Rijkswaterstaat, Source: Eindhovens Dagblad Tuesday July 27, 2004 Source: Eindhovens Dagblad Tuesday August 8, 2004 “Murphy’s Law..” “Two failures at the same time..” “.. the other failure was in the control software.” “.. expects that the supplier will install faultless software this week.”

Philips TASS, 20/10/20158 Consequences of insufficient testing Dissatisfied customers (claims) Bad publicity (image/brand damage) Service (helpdesk, logistics) Fixing the faults (repairs)

Philips TASS, 20/10/20159 Costs of fixing a fault relative to phase From: E. van Veenendaal, The Testing Practitioner, Source: B. Boehm, Software Engineering Economics, 1981 Consequence worse if fault is found late Statistics show exponential increase Operation phase: 1000 x Cost Reqt. phase

Philips TASS, 20/10/ Can we calculate the profit of testing? A simplified hypothetical example: *Costs of testing effort: 180,000 Euro. *Serious problem found in acceptance testing. *Fix in acceptance test phase costs 1,400 Euro. *If not found until in operation: costs 20,000 Euro. *‘Profit’: n x (20, ,400) - 180,000 !! (where n is amount of serious problems found)

Philips TASS, 20/10/ Supporting practices Risk based testing Using intake criteria and acceptance criteria Measuring severity Measuring field call rate

Philips TASS, 20/10/ Risk management in unit testing Classify modules using criteria for Impact and Likelihood E.g. HIGH: give higher priority and more thorough testing Result: important defects found sooner: lower costs

Philips TASS, 20/10/ Using intake and acceptance criteria Example 1: Mobile Phone Project –Intake: Every module tested and reviewed –Acceptance: customer acceptance test Example 2: TV Project –Intake procedure: paper check release notes –Acceptance: team required on customer site

Philips TASS, 20/10/ Severity Severity A: Product can not be produced, block acceptance Severity B: Serious Problem Severity C: Cosmetic / Minor issue Severity D: Acceptable

Philips TASS, 20/10/ After the project: FCR Indication to what extent the product doesn’t meet the expectations of the customer, resulting in complaints. Meaning: –Simplified example: 4% FCR: Of every 100 sold products, 4 products are returned by the customer to the service organization (warranty period) –Used for root cause analysis, not a financial driver

Philips TASS, 20/10/ Supporting practices resumed *Risk based testing –Prioritize & Differentiate => Test Strategy ! –More effective testing, Lower cost of failure. *Using intake criteria and acceptance criteria –More explicit transitions between phases –Reduce costs of fixing faults from earlier phases *Measuring severity –Monitor seriousness of (potential) failure –Input for calculating cost/benefits of testing *Measuring field call rate –Analyze failures after product release –Input for calculating costs of product failure

Philips TASS, 20/10/ Can we calculate the profit of testing? *Found n serious problems (B) in unit testing. *2% FCR would cost 20,000 x 50 = 1,000,000 Euro. *5*52 B-problems NOT found would consume all profit! Taking cost of testing into consideration, e.g. (n x 800) + 180,000: extra profit is 819,200 Euro. 1,000,000 products sold Repair cost: 50 Euro per product Selling price: 500 Euro per product *2% FCR reduction = extra profit of 1,000,000 Euro. Expectations of supplier: 10% profit (50,000,000 Euro) 4% FCR (2,000,000 Euro) Hypothetical example refined: *Root cause analysis learned: type B yields 2% FCR.

Philips TASS, 20/10/ Conclusions Testing is not cheap … … insufficient testing is expensive. Risk based testing and criteria can increase effectiveness. Measuring during projects is important … … and if we continue to measure after release, costs and profit of testing can be calculated. Your contribution in refining and improving these calculations is more than welcome !!