Metrics That Matter Real Measures to Improve Software Development Steven Borg, Principal ALM Consultant Northwest Cadence Steven.Borg@nwcadence.com

Every ‘best in class’ company measures software quality Every ‘best in class’ company measures software quality. There are no exceptions. If your company does not do this it is not an industry leader and there is a good chance that your software quality levels are marginal at best. Capers Jones, Applied Software Measurement

You get what you measure. If you don’t measure it, you can’t manage it You get what you measure. If you don’t measure it, you can’t manage it. You cannot improve what you can’t measure. Garbage in, garbage out. If you don’t measure it, it’s just a hobby. "You can't manage what you can't control, and you can't control what you don't measure. " Tom DeMarco

Metrics Matter Without metrics you can’t predict you can’t judge quality you can’t accurately estimate you can’t measure impacts you can’t improve consistently

Characteristics of Effective Metrics Comparable Honest Simple Actionable

True Metrics vs Proxy Metrics True Metric What you SHOULD measure Proxy Metric What you CAN measure Tip #1 Focus on the true metric, not proxy metrics

“The companies that wish to improve but do not measure are at the mercy of fads and chance. Progress may not be impossible, but it is certainly unlikely.” Capers Jones Tip #2 Only introduce one or two new metrics at a time

Deming Cycle 1st 2nd 4th 3rd

ITIL Seven Step Improvement Process Define what you should measure. Define what you can measure. Gather the data. Who? How? When? Integrity of the data. Process the data. Frequency, format, system, accuracy Analyze the data. Relationships, trends, according to plan, targets met, corrective action Present and use the information. Implement the corrective action. ITIL: Information Technology Infrastructure Library

Tip #3 Standardize by setting triggers to alert you to slips in prior metrics A few effective metrics: Quality Metrics Sizing Metrics Productivity Metrics User Metrics Business Metrics

Quality Metrics Defect Removal Efficiency Warning: Not always “simple” Code Coverage Warning: Not always “honest” Performance Profiling Test Cases per feature Bugs per feature (bug density metrics)

Sizing Metrics Lines of Code Warning: Not always “honest” Function Points Warning: Not “simple” Effort (in hours, days, weeks, etc) Warning: Not always “honest” or “comparable” Story Points Warning: Rarely “comparable”

Productivity Metrics Velocity Warning: Not always “simple” Throughput Lines of Code / Developer / Day (???) Quality* Note: Defect Removal Efficiency is highly correlated with Productivity So highly correlated it can often act as proxy metric for productivity

User Metrics User satisfaction Warning: Not always “comparable” Post-release bug count Number of Help Desk calls Warning: Not always “honest”

Business Metrics Schedule Estimation Accuracy Warning: Not always “honest” Cost Estimation Accuracy Scope Estimation Accuracy Warning: Rarely “honest” ROI Estimation Accuracy Warning: Rarely “simple”, Not always “honest”

Tip #4 First identify your problem, THEN identify the metric Tip #5 Balance introduction of new metrics across different categories The problem He broke the Build!!!

Balance introduction of new metrics across different categories Tip #5 Balance introduction of new metrics across different categories The problem today is not a deficiency in software measurement technology itself; rather, it is cultural resistance on the part of software management and staff. The resistance is due to the natural human belief that measures might be used against them. This feeling is the greatest barrier to applied software measurement. Capers Jones, Applied Software Measurement

Tip #6 NEVER use metrics to evaluate individuals Tip #7 Watch your metrics closely; adjust when necessary

Estimation and Planning “There is a perfect correlation between measurement accuracy and estimation accuracy: Companies that measure well can estimate well; companies that do not measure accurately cannot estimate accurately either.” Capers Jones "Optimism is an occupational hazard of programming: feedback is the treatment." Kent Beck “Planning and estimation are the mirror images of measurement. ”

ALM (Application Lifecycle Management Tooling) Next Slide

Why ALM (Application Lifecycle Management Tooling)? Tip #8 Count first, Calculate next, use Judgment last Why ALM (Application Lifecycle Management Tooling)? "Tools are very important element of defining a path of least resistance. If I can set up a tool so that it’s easier for a developer to do something the way that I want the developer to do it, and harder for the developer to do it some other way, then I think it’s very likely the developer is going to do it the way I want them to, because it’s easier. It’s the path of least resistance." Steve McConnell

Application Lifecycle Management Tooling Tip #9 When possible, use an integrated ALM Tool to gather metrics Tip #10 Start using the built-in reports right away!

Cultural issues with Metrics The problem today is not a deficiency in software measurement technology itself; rather, it is cultural resistance on the part of software management and staff. The resistance is due to the natural human belief that measures might be used against them. This feeling is the greatest barrier to applied software measurement. Capers Jones, Applied Software Measurement

"The problem of software process change are often complicated by the fact that no one is responsible to make it happen. If software process improvement isn't anybody's job, it is not surprising that is doesn't get done! If it is important enough to do, however, someone must be assigned the responsibility and given the necessary resources. Until this is done, software process development will remain a nice thing to do someday, but never today." Watts Humpthey

Tip #11 Put someone in charge of process improvement… use metrics to show change "If it ain't broke, don't fix it," the saying goes. Common software development practices are seriously broken, and the cost of not fixing them has become extreme. Traditional thinking would have it that the change represents the greatest risk. In software's case, the greatest risk lies with not changing - staying mired in unhealthy, profligate development practices instead of switching to practices that were proven more effective many years ago. Steve McConnell

Tip #12 Start collecting metrics now Tip #12 Start collecting metrics now. Next year, you shouldn’t be saying “I wish I had some metrics” Steven Borg