Modeling and Managing Software Productivity & Quality … balancing Efficiency and Effectiveness Softec 2011 Kuala Lumpur, Malaysia Gary A. Gack MBA, Six Sigma Black Belt, ASQ Certified Software Quality Engineer Owner, Process-Fusion.net GGack@Process-Fusion.net My intended audience includes C, V, and D level leaders, both inside and outside the IT organization – if you have a stake in the outcome of software and IT projects this presentation is intended for you – it’s about MANAGEMENT, not technology. Unfortunately software and IT projects are extremely risky – for example the recent IRS fiasco, the Denver airport baggage system, and many other failures, large and small. Often, a lot of money and time goes into this “black hole”, and even light may not come out the other end! Sadly, many of these failures and much of the waste are in part traceable to General Managers who, due to lack of understanding of the consequences, act in ways that are counter-productive. My purpose today is to acquaint you with the true magnitude of the problem, the most prevalent root causes, and some actions you can take to improve outcomes – this problem IS NOT entirely attributable to the geeks! I’ve spent more than 40 years in this industry and have been engaged on a number of occasions to manage runaway project turnarounds. In every instance it is clear in retrospect that each of these was foreseeable and preventable. Software is everywhere – in your cell phone, ATM, anti-lock brakes, avionics, and in all of the business systems you use every day. In many businesses and industries it is the primary source of differentiation and competitive advantage – hard to imagine how we could live without it, in spite of the many frustrations it causes. In many industries software and IT costs are the largest item in G&A expense – perhaps 30% of total costs in financial institutions, for example. Software acquisition, development, maintenance, and support is usually 30-50% of the total spend, and the source of 80-90% of the waste and grief. Very few executives are aware of the amazing level of waste – if the rest of your business was as leaky as software, you’d be out of business! © 2011 Process-Fusion.net

Agenda Measuring Efficiency (Productivity) the Cost of Quality Framework Measuring Effectiveness (Quality) Defect Containment Modeling & Managing Efficiency and Effectiveness Why Modeling? Scenarios Considered Effectiveness Results Efficiency Results “An Apple a Day … “ Speaking of performance, here’s some sobering industry data – less than 1/3 of software projects are fully successful (on-time, on-budget, complete). These data have been collected for about 15 years now, and as we see in this subset, the success rate has changed very little The shift from failed to challenged is very possibly a form of grade inflation – there are lots of de facto incentives to avoid calling a project a failure © 2011 Process-Fusion.net

What is “Efficient” (Productive)? How can it be measured? A Lean Perspective The Cost of Quality Framework © 2011 Process-Fusion.net

“Productive”? What does that mean? A software process is “productive” (efficient) if, relative to an alternative … It produces an equivalent or better result at lower cost. For example, if defect-finding strategy “A” finds the same number of defects as does strategy “B” (i.e., the two strategies are equally effective), but does so at lower cost, strategy “A” is more efficient than strategy “B”. A is more “productive” than B. © 2011 Process-Fusion.net

Feigenbaum’s Cost of (Poor) Quality Framework “pre-release” (finding defects) “pre-release” (fixing defects) “post-release” (fixing defects) © 2011 Process-Fusion.net

“Lean” Meets Software Development Taiichi Ohno’s 7 Wastes Defects Overproduction Inventory Extra processing Unnecessary motion Transportation Waiting Software/IT Translation Rework - missing, wrong, extra, (avoidable) Low value “features”, unused “hooks” Unassigned Backlog – Requirements, Designs Unused Documentation Task switching, concurrent assignments Delays for approvals, decisions, resources Handoffs © 2011 Process-Fusion.net

What % of Spend is “Value Added”. (i. e What % of Spend is “Value Added”? (i.e., creating new features & functions) Total Cost = Value Added: new features & functions + Finding & fixing defects - “internal” (pre-delivery) and “external” (post-delivery) + Prevention: training, process improvement efforts (% Non-Value Added) This chart carries a very simple message – small increases quality management investments lead to large decreases in the cost of defects and to an overall reduction in total costs. In software this leverage is found primarily in finding defects early by using formal Inspections – the “we don’t have time for that” argument is FALSE – evidence to the contrary is overwhelming! Effective quality management WILL reduce total software cost and reduce cycle time. Incidentally, I often hear software groups claim they already do formal inspections, but when I look at their practices I find they are not at all consistent with industry standards, particularly as most don’t collect the data that Inspections require. If your software group cannot tell you how many defects per hour are discovered by Inspection compared to test, they are certainly not doing Inspections. ?? (Prevention + Appraisal) (Rework) © 2011 Process-Fusion.net

Software Industry Cost of Quality “B”=20% Source: Capers Jones Effort Devoted to “de-scoped” features “C”=10% ?? A + B + C >= 70% © 2011 Process-Fusion.net

Key “Take-away”: To improve Efficiency (productivity), reduce NVA NVA ~= Appraisal + Rework (Optimization = what, when, how) © 2011 Process-Fusion.net

What is “Effectiveness”? How can it be measured? A Quality Perspective Defect Containment © 2011 Process-Fusion.net

“Effective”? What does that mean? Delivered software is “effective” if: it serves a valid organizational purpose - efforts are made to quantify this aspect of effectiveness with return on investment estimates, yet it is essentially a subjective evaluation. (2) it is acceptably defect free. The term “defect” in this context is intended to be broadly construed e.g., a missed or incorrect requirement is a defect; a user-unfriendly design is a defect. Hence, once a project has been initiated the effectiveness of the software process used to execute the project is appropriately measured by defect containment – i.e., the percentage of defects removed before the software product is delivered to the customer. © 2011 Process-Fusion.net

Defect Containment Defined “Total Containment Effectiveness” (TCE) = % of defects found before release e.g., 80 defects found in test, 20 found by customers = 80% TCE Measure customer defects over agreed time frames (3/6/12 months) Defect Containment “Efficiency” considers cost Phase/Iteration (“step”) Appraisal Containment = % of defects present found by a specific appraisal event e.g., of 50 defects present in requirements, 40 found by inspection = 80% “step” containment Defects present can be estimated and/or evaluated in retrospect by identifying “origin” © 2011 Process-Fusion.net

Defect Insertion and Removal Benchmarks Capers Jones, Applied Software Measurement, 3rd Ed. Do You Know Your Numbers? Phase Introduced Defects per Function Point (MIS) Requirements .75 (.84) Design 1.50 (1.69) Code 1.75 (1.97) Documents .50 ( - ) Bad Fixes .50 Appraisal Method % Removed (MIS) Unit Test 25% Function Test 30% Integration Test System Test 35% Acceptance Test Inspections 60-90% © 2011 Process-Fusion.net

Modeling & Managing Software Process Efficiency and Effectiveness Leading Indicators Provide CONTROL © 2011 Process-Fusion.net

Why Modeling? In many software groups finding and fixing defects consumes 50-70% of total cost Best practice groups reduce that by at least 50% Models allow you to think through the consequences of alternative strategies … quickly, at very low cost Models allow you to forecast both quality and financial consequences of alternatives Creating a business case in the process Creating a basis for “quality adjusted” status evaluation Modeling motivates measurement and “management by fact” © 2011 Process-Fusion.net

Model Objectives Predict (1) delivered quality and (2) total non-value-added effort (cost) Predict defect “insertion” Focus attention on defects, which account for the largest share of total development cost. Enable early monitoring of the relationship between defects likely to be present and those actually found – provide early awareness. Estimate effort needed to execute the volume of appraisal necessary to find the number of defects we forecast to remove. a ‘sanity check’ on the planned level of appraisal effort – i.e., is it actually plausible to remove an acceptable volume of defects with the level of effort planned? Forecast both “pre-release” (before delivery) and “post-release” (after delivery) NVA effort. When delivered quality is poor, post-release defect repair costs can be 50% of the original project budget. © 2011 Process-Fusion.net

IMPORTANT Caveats Don’t focus on the parameter values I have used The thought process is the important part Actual values vary considerably from place to place Where available I have used industry benchmarks All benchmarks conceal large variation Where benchmarks are not available I’ve used experience as a guide Your local values may well be quite different Use models such as these to do “what if” analysis “simulate” a range of assumptions Best/worst/most likely values © 2011 Process-Fusion.net

Scenarios Evaluated © 2011 Process-Fusion.net

Simulation Results - Containment I will be happy to provide a copy of the model and related articles for your use ggack@process-fusion.net © 2011 Process-Fusion.net

Simulation Results: Non-Value-Added “fix” “find” © 2011 Process-Fusion.net

© 2011 Process-Fusion.net

Cost of Quality Revisited WHEN you invest matters more than how much © 2011 Process-Fusion.net

“An apple a day …” Formal inspections, conducted in accordance with IEEE Std. 1028-2008, are always efficient & effective … better than any form of testing Maximum benefits come when applied to requirements, architecture, and design YOU can both reduce cost (improve productivity) and deliver better quality © 2011 Process-Fusion.net

Thank You! terima kasih 謝謝 © 2011 Process-Fusion.net