R&D Metrics Todd Little

What Makes a Good Metric? What is it measuring? Why do we care? How easy is it to gather & compute? Could you do it well and still suck? Could you do it poorly and still be great? How do you cheat? How do you balance the cheating?

Low Hanging Fruit Estimation Quality Factor (EQF) Defect Detection Efficiency (DDE) Defect Removal Efficiency (DRE) Cumulative Swamp Report Customer Satisfaction

Estimation Quality Factor (EQF) Elapsed Time Value to be Estimated Actual Value Initial Estimate Actual End Date Link to article by Tim Lister Blue Area Red Area EQF =

EQF from Lister/DeMarco An EQF of 5 is pretty good (i.e. averaging about 1/5 or 20 percent off.) The median for schedule estimating is about a 4, with the highest sustained scores at 8 to 9. Lister and DeMarco have never known anybody to sustain a 10 (just 10 percent off). Typical disaster project is 1.8

EQF Distribution Curve (LGC) EQF for duration has a theoretical minimum of 2.0

We slip one day at a time, EQF=2 Elapsed Time Value to be Estimated Actual Value Initial Estimate Actual End Date Blue Area Red Area EQF =

(EQF-2) Distribution Curve (LGC data)

LGC Estimation Quality LGC’s EQF measurement is pretty good. Our p(50) is 4.8, versus an industry average around 4 and a best sustained in the ~8-10. Our p(10) is 2.8, which is not bad. Calendar projects should have EQF = ~∞ EQF can be used with scope on Y-axis

Defect Detection Efficiency (DDE) & Defect Removal Efficiency (DRE)

Comments about DDE and DRE Can be for a release or a product Can be all defects or just P1/P0 Can be all defects or just customer defects

DDE and DRE SCOPUS Queries Ignore Closed, Duplicate, and Enhancements Per product is easy Per release requires release date info and is subject to version reporting

Defect Detection and Defect Removal Efficiency Product Overall P0/P1 DDE Rate Overall P0/P1 DRE Rate P0/P1 Customer DRE A 88%77%47% B 83%84%62% C 92%91%83% D 34%68%60% E 71%94%88% F 94%100% G 77%95%85% H 75%95%96% I 81%91%95%

Defect Injection vs. Defect Removal

Cumulative Swamp over time

Swamp projections

Customer Product Satisfaction

Summary EQF calculated from time tracking DDE and DRE computed from defect tracking Cumulative Swamp over time from time tracking and from defect tracking Customer Product satisfaction from Customer Support Survey

Summary MetricDescriptionHow EasyDataHow to Cheat How to balance EQF Estimation Quality Factor.Easy Project Tracking Sandbagging, Cut scope, Ship crap Throughput, DDE, Customer Satisfaction DDE Defect Detection Efficiency.Easy Defect Tracking Find lots of meaningless defects. Mark meaningless defects as P1 or P0. Defects / LOC?, Customer Satisfaction DRE Defect Removal Efficiency.Easy Defect Tracking Re-class customer defects as P2 and only consider P0 Defects / LOC?, Customer Satisfaction Customer Satisfaction Customer support product satisfactionEasy Customer Support Ask the right questionsRevenue