R&D Metrics Todd Little. What Makes a Good Metric? What is it measuring? Why do we care? How easy is it to gather & compute? Could you do it well and.

Slides:



Advertisements
Similar presentations
Metrics to improve software process
Advertisements

© 2012 Pearson Prentice Hall. All rights reserved. Measuring and Managing Customer Relationships Chapter 6.
1 Defect Removal Effectiveness Kan ch 6 Steve Chenoweth, RHIT Left – Some defects are easier to remove than others. This is the cruise ship Costa Concordia,
1 These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 5/e and are provided with permission by.
1 These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 5/e and are provided with permission by.
1 These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 5/e and are provided with permission by.
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by.
1 These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 5/e and are provided with permission by.
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by.
Metrics for Process and Projects
Computer Engineering 203 R Smith Project Tracking 12/ Project Tracking Why do we want to track a project? What is the projects MOV? – Why is tracking.
1 In-Process Metrics for Software Testing Kan Ch 10 Steve Chenoweth, RHIT Left – In materials testing, the goal always is to break it! That’s how you know.
Uncertainty surrounding the Cone of Uncertainty Todd Little “It’s tough to make predictions, especially about the future.” – Yogi Berra.
Metrics Project and Process Metrics. Why do we measure? Assessing project status Allows us to track risks Before they go critical Adjust workflow See.
Personal Software Process
Software Measurement and Process Improvement
Applied Software Project Management Andrew Stellman & Jennifer Greene Applied Software Project Management Applied Software.
SE 450 Software Processes & Product Metrics Software Metrics Overview.
Aplicaciones de Ingeniería de Software
EXAMPLES OF METRICS PROGRAMS
1 Software Quality Metrics Ch 4 in Kan Steve Chenoweth, RHIT What do you measure?
Risk Management. What is risk? You have some expected outcome –Of some event in the future Risk is the deviation of the actual future outcome from the.
C o n f i d e n c e p e r f o r m a n c e d i s t i n c t i o n q u a l i t y Nursing Facility Family Satisfaction Survey Report Prepared for: Wiley Mission.
Measures of Central Tendency
Getting Business Value out of (Test) Metrics Richard Terry (UK) Rob Baarda (NL)
CHAPTER 39 Cumulative Frequency. Cumulative Frequency Tables The cumulative frequency is the running total of the frequency up to the end of each class.
14. Introduction to inference
Cmpe 589 Spring Software Quality Metrics Product  product attributes –Size, complexity, design features, performance, quality level Process  Used.
Understanding Earned Value Analysis
CS 350, slide set 6 M. Overstreet Old Dominion University Spring 2005.
KICK-OFF MEETING AB-SD MOCK PROJECT Presenter:
Team Assignment 15 Team 04 Class K15T2. Agenda 1. Introduction 2. Measurement process 3. GQM 4. Strength Weakness of metrics.
Normal Distribution Section 2.2. Objectives  Introduce the Normal Distribution  Properties of the Standard Normal Distribution  Use Normal Distribution.
© 1998 Carnegie Mellon UniversityTutorial The Personal Software Process (PSP) The overview of the PSP that follows has been built from material made.
1 Chapter 4 Software Process and Project Metrics.
Top Down View of Estimation Test Managers Forum 25 th April 2007.
Software Metrics – part 2 Mehran Rezaei. Software Metrics Objectives – Provide State-of-art measurement of software products, processes and projects Why.
Lecture 4 Software Metrics
INFO 636 Software Engineering Process I Prof. Glenn Booker Week 9 – Quality Management 1INFO636 Week 9.
Microsoft Reseach, CambridgeBrendan Murphy. Measuring System Behaviour in the field Brendan Murphy Microsoft Research Cambridge.
SE is not like other projects. l The project is intangible. l There is no standardized solution process. l New projects may have little or no relationship.
© 2008 Sterling Commerce. Confidential and Proprietary. How to Get Along with Project Using Microsoft Project so that it actually works for you, not against.
This material is approved for public release. Distribution is limited by the Software Engineering Institute to attendees. Sponsored by the U.S. Department.
Mistakes, Errors and Defects. 12/7/2015Mistakes, Errors, Defects, Copyright M. Smith, ECE, University of Calgary, Canada 2 Basic Concepts  You are building.
Project Control Farrokh Alemi, Ph.D.. Course on Project Management Purpose Monitor progress Monitor progress Corrective action Corrective action.
Agile Metrics It’s Not All That Complicated. © 2011 VersionOne 2 Welcome – About your Trainer, Katia Sullivan VersionOne Product Trainer and Agile Coach.
Test status report Test status report is important to track the important project issues, accomplishments of the projects, pending work and milestone analysis(
Normal Distribution Practice with z-scores. Probabilities are depicted by areas under the curve Total area under the curve is 1 Only have a probability.
Inventory Management The goal of effective inventory management is to “meet or exceed your customers’ expectations of product availability with the amount.
1 These slides are designed to accompany Software Engineering: A Practitioner’s Approach, 7/e (McGraw-Hill 2009). Slides copyright 2009 by Roger Pressman.
The Population of Near-Earth Asteroids and Current Survey Completion Alan W. Harris MoreData! : The Golden Age of Solar System Exploration Rome,
Swami NatarajanOctober 1, 2016 RIT Software Engineering Software Metrics Overview.
Peter Varhol Solutions Evangelist
Chapter 4 Software Process and Project Metrics
Understanding Earned Value Analysis
Understanding Earned Value Analysis
Why Do We Measure? assess the status of an ongoing project
HORIZONT IWS/WebAdmin Many thanks for your attention
Software Quality Engineering
For University Use Only
Chapter 25 Process and Project Metrics
Process and Project Metrics
Measuring Success How to use simple bug database queries and reports
Why Do We Measure? assess the status of an ongoing project
Chapter 32 Process and Project Metrics
Mistakes, Errors and Defects
Integrated Math 2 - Santowski
Understanding Earned Value Analysis
Critical Path Simulation
Presentation transcript:

R&D Metrics Todd Little

What Makes a Good Metric? What is it measuring? Why do we care? How easy is it to gather & compute? Could you do it well and still suck? Could you do it poorly and still be great? How do you cheat? How do you balance the cheating?

Low Hanging Fruit Estimation Quality Factor (EQF) Defect Detection Efficiency (DDE) Defect Removal Efficiency (DRE) Cumulative Swamp Report Customer Satisfaction

Estimation Quality Factor (EQF) Elapsed Time Value to be Estimated Actual Value Initial Estimate Actual End Date Link to article by Tim Lister Blue Area Red Area EQF =

EQF from Lister/DeMarco An EQF of 5 is pretty good (i.e. averaging about 1/5 or 20 percent off.) The median for schedule estimating is about a 4, with the highest sustained scores at 8 to 9. Lister and DeMarco have never known anybody to sustain a 10 (just 10 percent off). Typical disaster project is 1.8

EQF Distribution Curve (LGC) EQF for duration has a theoretical minimum of 2.0

We slip one day at a time, EQF=2 Elapsed Time Value to be Estimated Actual Value Initial Estimate Actual End Date Blue Area Red Area EQF =

(EQF-2) Distribution Curve (LGC data)

LGC Estimation Quality LGC’s EQF measurement is pretty good. Our p(50) is 4.8, versus an industry average around 4 and a best sustained in the ~8-10. Our p(10) is 2.8, which is not bad. Calendar projects should have EQF = ~∞ EQF can be used with scope on Y-axis

Defect Detection Efficiency (DDE) & Defect Removal Efficiency (DRE)

Comments about DDE and DRE Can be for a release or a product Can be all defects or just P1/P0 Can be all defects or just customer defects

DDE and DRE SCOPUS Queries Ignore Closed, Duplicate, and Enhancements Per product is easy Per release requires release date info and is subject to version reporting

Defect Detection and Defect Removal Efficiency Product Overall P0/P1 DDE Rate Overall P0/P1 DRE Rate P0/P1 Customer DRE A 88%77%47% B 83%84%62% C 92%91%83% D 34%68%60% E 71%94%88% F 94%100% G 77%95%85% H 75%95%96% I 81%91%95%

Defect Injection vs. Defect Removal

Cumulative Swamp over time

Swamp projections

Customer Product Satisfaction

Summary EQF calculated from time tracking DDE and DRE computed from defect tracking Cumulative Swamp over time from time tracking and from defect tracking Customer Product satisfaction from Customer Support Survey

Summary MetricDescriptionHow EasyDataHow to Cheat How to balance EQF Estimation Quality Factor.Easy Project Tracking Sandbagging, Cut scope, Ship crap Throughput, DDE, Customer Satisfaction DDE Defect Detection Efficiency.Easy Defect Tracking Find lots of meaningless defects. Mark meaningless defects as P1 or P0. Defects / LOC?, Customer Satisfaction DRE Defect Removal Efficiency.Easy Defect Tracking Re-class customer defects as P2 and only consider P0 Defects / LOC?, Customer Satisfaction Customer Satisfaction Customer support product satisfactionEasy Customer Support Ask the right questionsRevenue