1. Getting Business Value out of (Test) Metrics Richard Terry (UK) richard.terry@sogeti.com Rob Baarda (NL)

2. What we will do / Structure Subjects: >Test Metric goals / stakeholders >Which metrics; some demo >Process around metrics >How to apply metrics / pittfalls >Wrap-up For each subject: intro, discussion Discussion >hand raise >hand raise with comments >open

3. Metric Goals / Stakeholders Hand raise only 1.Who is using project metrics? 2.Who is using test metrics? 3.Who would like to use test metrics? Some opinions please! Goals / Stakeholders for test metrics? OPEN! Your votes please

4. Which test metrics? Test basis Test object Test Execution Defects Repair Production Specifying test cases / scripts Test cases/ scripts Test Process Size test basis Size test object # defects in test object # defects in production For each process: # hours effort lead time # test cases # = number of # defects in test basis # repair rounds

5. Deductible metrics Effort = Size /# hours Productivity = # test cases / # hours Efficiency = # defects / (# hours or # test cases) >Specification >Test execution >Retest of repaired defects DDP Defect Detection Percentage (Europe) DRE Defect Removal Efficiency (USA) Defect injection rate for rework Damage prevented? More metrics: >Availability of test environments >OPEN

6. Dutch test metrics experiences Dutch initiative to gather test metrics Parties involved >NESMA Netherlands Software Metrics Association >Testnet, including Sogeti Dutch Testing community >LaQuSO Laboratory for Software Quality Universities Eindhoven & Nijmegen Some slides with data from the database will follow in the presentation, not in the hand-out, due to a lack of statistical evidence

7. Processes around metrics Collection in a project >Embedded in daily work >Weekly summarisation >Sanity checks >Cost: about 2% project budget Distribution For a benchmark on the level of: >Project releases >Organisation >Country >International: ISBSG International Software Benchmarking Standards Group

8. To what extent? OPEN Discussion about >Accuracy of definitions >Number of types of defects >Is a batch test case the same as an online test case? >Only testing of functionality or also security, performance, usability >How to include regression testing? >Predictive value average (mean), median, standard deviation, correlations with? Prediction model needed?

9. To apply test metrics Project >Estimating Testing New development >Estimating Testing Maintenance release >Controlling the test project Benchmark >Measure the effectiveness of the testing, Improve inside / outside the test process? PITTFALLS?

10. Make a judgement based with metrics? Of person production/ quality of work >OPEN Discussion >Some Dutch effects Of Service Level Agreement >What to measure and how to judge

11. Wrap-up Business value out of metrics Hand raise >Who would like to use test metrics? Changed opinions >Comments please (-  +,+  -) Reading: TMap Next >Chapter 13 Goal-Question-Metrics for test >Chapter 11 for data End of 2008: www.isbsg.org Participate? Leave your business card