Using Metrics to Reduce Cost of Re-work Dwight Lamppert Senior Test Manager Franklin Templeton

2 Opening Question What is the job of testing?

3 Potential Response The job of testing is to find defects.

4 Myth Exposed Myth: The Job of Testing Is to Find Defects 1 1 The Seven Principles of Lean Software Development by Mary Poppendieck

5 Quality is everyones responsibility Myth: The Job of Testing Is to Find Defects The job of tests, and the people that develop and run tests, is to prevent defects, not find them. A quality assurance organization should champion processes that build quality into the code from the start rather than test quality in later. 1 1 The Seven Principles of Lean Software Development by Mary Poppendieck

6 At a recent software seminar The common attributes of successful software projects are Heroes Chaos Re-work Untracked OT

7 Todays Discussion Case Study – 4 projects 2 Definitions Testing Metrics (what to collect & analyze) Metrics from first 2 projects Process Improvement Steps Success Story – Projects 3 & 4 Some trends & conclusions

8 Project Background Legacy mature system (mainframe) Industry regulatory enhancements Time-boxed: 3 to 4 months Waterfall SDLC First 2 Projects Project 1 – Dec09 – 1X Project 2 – June10 – 2.3X

9 Definition: Re-work Re-work – AD time to fix and testing time to report and re-test System Testing and Production defects

10 Definition: Static Testing Static Testing - Testing of a component or system at specification or implementation level without execution of that software, e.g., reviews or static code analysis.

11 Static vs. Dynamic Testing From System Testing with an Attitude, p.14, by Nathan Petschenik

12 Testing Defect Analysis When was the defect introduced (SDLC phase) When was the defect discovered (testing phase)

13 Points of Origin for Defects 2

14 Defect Discovery Phases 1. Static Testing Document walk-throughs & reviews Pre-testing Test Plan review All before Build is complete 2. System Testing 3. Production

15 Tracking the Cost of Re-work 1. Defect Fix Time 2. Re-test Time COR (Cost of Re-work) – calculated by multiplying time x rate – cost per defect

16 Testing Rework Workflow Re-work time during Test Execution is tracked as 2 Components 1. Time to Fix a Defect 2. Time to Re-test Defect Fixes

17 Project 1 - Defect Fix Cost

18 Issues from first 2 projects 1. High % of project = Cost of Re-work

19 COR = 23 to 29% Project

20 Issues from first 2 projects 1. High % of project = Cost of Re-work 2. High % of Requirements & Design (R&D) defects found later

% R&D Defects found later

22 Issues from first 2 projects 1. High % of project = Cost of Re-work 2. High % of Requirements & Design (R&D) defects found later 3. DDP (Defect Discovery %) – slightly better than average

23 Defect Discovery Percentage (DDP) DDP = % of Total Defects discovered before implementation Provides a metric to evaluate the overall effectiveness of testing. 90% - Good 95% - Very Good

24 DDP = average +

25 Process Improvement Steps 1. Developed framework in Quality Center to track COR 2. Presentations to business partners & executives 3. Lively discussion during presentations 4. AD appointed a new AD Lead 5. AD improved their internal code review 6. More emphasis on and participation during reviews, walk-throughs – Static Testing

26 Process Improvement Steps 7. More collaboration between AD and testing teams 8. Increased emphasis on unit testing 9. Risk-based testing approach 10. Improved regression work 11. Staff increased in domain knowledge

27 Projects 3 & 4 – Process Improved 1. Project 1 – Dec09 – 1x 2. Project 2 – June10 – 2.3x 3. Project 3 – May11 – 3.4x 4. Project 4 – June11 – 2.3x

28 Static Testing emphasized

29 System Testing Defects

30 % R&D Defects found later

31 DDP (Defect Discovery %)

32 COR as % Project Cost

33 Defect Discovery Rate DDR = % of Total Defects discovered during an SDLC or Test Phase Static Testing System Testing Transition Goal - Trying to find out if we are finding defects earlier.

34 DDR Trends Coding As DDR for Static Testing increases, DDR for System Testing and Production Defects decreases

35 When is Unit Testing complete? A necessary criterion for completion of developer testing is that the developers believe that the software is ready to be delivered to the users on the day it is delivered to the system-test team. From System Testing with an Attitude, p. 33, by Nathan Petschenik

36 Todays Discussion Defect metrics – powerful story Static Testing works It is a collaborative effort The positive effect ripples into Unit Testing, System Testing, UAT, and implementation No shortcuts here – influence your project to do it right

37 Some Final Thoughts Training needed for our staff (testing, AD, business, PM community) - how to be more effective in document reviews and walk-throughs. Perspectives Tester – static testing > a paradise (a new maturity level)

38 Questions & Answers ???