1. Modeling and Managing Software Productivity & Quality … balancing Efficiency and Effectiveness
Softec 2011
Kuala Lumpur, Malaysia
Gary A. Gack
MBA, Six Sigma Black Belt, ASQ Certified Software Quality Engineer
Owner, Process-Fusion.net
My intended audience includes C, V, and D level leaders, both inside and outside the IT organization – if you have a stake in the outcome of software and IT projects this presentation is intended for you – it’s about MANAGEMENT, not technology.
Unfortunately software and IT projects are extremely risky – for example the recent IRS fiasco, the Denver airport baggage system, and many other failures, large and small. Often, a lot of money and time goes into this “black hole”, and even light may not come out the other end!
Sadly, many of these failures and much of the waste are in part traceable to General Managers who, due to lack of understanding of the consequences, act in ways that are counter-productive. My purpose today is to acquaint you with the true magnitude of the problem, the most prevalent root causes, and some actions you can take to improve outcomes – this problem IS NOT entirely attributable to the geeks!
I’ve spent more than 40 years in this industry and have been engaged on a number of occasions to manage runaway project turnarounds. In every instance it is clear in retrospect that each of these was foreseeable and preventable.
Software is everywhere – in your cell phone, ATM, anti-lock brakes, avionics, and in all of the business systems you use every day. In many businesses and industries it is the primary source of differentiation and competitive advantage – hard to imagine how we could live without it, in spite of the many frustrations it causes.
In many industries software and IT costs are the largest item in G&A expense – perhaps 30% of total costs in financial institutions, for example. Software acquisition, development, maintenance, and support is usually 30-50% of the total spend, and the source of 80-90% of the waste and grief. Very few executives are aware of the amazing level of waste – if the rest of your business was as leaky as software, you’d be out of business!
© 2011 Process-Fusion.net

2. Agenda Measuring Efficiency (Productivity)
the Cost of Quality Framework
Measuring Effectiveness (Quality)
Defect Containment
Modeling & Managing Efficiency and Effectiveness
Why Modeling?
Scenarios Considered
Effectiveness Results
Efficiency Results
“An Apple a Day … “
Speaking of performance, here’s some sobering industry data – less than 1/3 of software projects are fully successful (on-time, on-budget, complete). These data have been collected for about 15 years now, and as we see in this subset, the success rate has changed very little
The shift from failed to challenged is very possibly a form of grade inflation – there are lots of de facto incentives to avoid calling a project a failure
© 2011 Process-Fusion.net

3. What is “Efficient” (Productive)? How can it be measured?
A Lean Perspective
The Cost of Quality Framework
© 2011 Process-Fusion.net

4. “Productive”? What does that mean?
A software process is “productive” (efficient) if, relative to an alternative …
It produces an equivalent or better result at lower cost.
For example, if defect-finding strategy “A”
finds the same number of defects as does strategy “B” (i.e., the two strategies are equally effective),
but does so at lower cost,
strategy “A” is more efficient than strategy “B”. A is more “productive” than B.
© 2011 Process-Fusion.net

5. Feigenbaum’s Cost of (Poor) Quality Framework
“pre-release”
(finding defects)
“pre-release”
(fixing defects)
“post-release”
(fixing defects)
© 2011 Process-Fusion.net

6. “Lean” Meets Software Development
Taiichi Ohno’s 7 Wastes
Defects
Overproduction
Inventory
Extra processing
Unnecessary motion
Transportation
Waiting
Software/IT Translation
Rework - missing, wrong, extra, (avoidable)
Low value “features”, unused “hooks”
Unassigned Backlog
– Requirements, Designs
Unused Documentation
Task switching, concurrent assignments
Delays for approvals,
decisions, resources
Handoffs
© 2011 Process-Fusion.net

7. What % of Spend is “Value Added”. (i. e
What % of Spend is “Value Added”? (i.e., creating new features & functions)
Total Cost = Value Added: new features & functions + Finding & fixing defects - “internal” (pre-delivery) and “external” (post-delivery) + Prevention: training, process improvement efforts
(% Non-Value Added)
This chart carries a very simple message – small increases quality management investments lead to large decreases in the cost of defects and to an overall reduction in total costs.
In software this leverage is found primarily in finding defects early by using formal Inspections – the “we don’t have time for that” argument is FALSE – evidence to the contrary is overwhelming! Effective quality management WILL reduce total software cost and reduce cycle time.
Incidentally, I often hear software groups claim they already do formal inspections, but when I look at their practices I find they are not at all consistent with industry standards, particularly as most don’t collect the data that Inspections require. If your software group cannot tell you how many defects per hour are discovered by Inspection compared to test, they are certainly not doing Inspections. ??
(Prevention + Appraisal)
(Rework)
© 2011 Process-Fusion.net

8. Software Industry Cost of Quality
“B”=20%
Source: Capers Jones
Effort Devoted to “de-scoped” features
“C”=10% ??
A + B + C >= 70%
© 2011 Process-Fusion.net

9. Key “Take-away”: To improve Efficiency (productivity), reduce NVA
NVA ~= Appraisal + Rework
(Optimization = what, when, how)
© 2011 Process-Fusion.net

10. What is “Effectiveness”? How can it be measured?
A Quality Perspective
Defect Containment
© 2011 Process-Fusion.net

11. “Effective”? What does that mean?
Delivered software is “effective” if:
it serves a valid organizational purpose
- efforts are made to quantify this aspect of effectiveness with return on investment estimates, yet it is essentially a subjective evaluation.
(2) it is acceptably defect free.
The term “defect” in this context is intended to be broadly construed
e.g., a missed or incorrect requirement is a defect; a user-unfriendly design is a defect.
Hence, once a project has been initiated the effectiveness of the software process used to execute the project is appropriately measured by defect containment – i.e., the percentage of defects removed before the software product is delivered to the customer.
© 2011 Process-Fusion.net

12. Defect Containment Defined
“Total Containment Effectiveness” (TCE)
= % of defects found before release
e.g., 80 defects found in test, 20 found by customers = 80% TCE
Measure customer defects over agreed time frames (3/6/12 months)
Defect Containment “Efficiency” considers cost
Phase/Iteration (“step”) Appraisal Containment
= % of defects present found by a specific appraisal event
e.g., of 50 defects present in requirements, 40 found by inspection = 80% “step” containment
Defects present can be estimated and/or evaluated in retrospect by identifying “origin”
© 2011 Process-Fusion.net

13. Defect Insertion and Removal Benchmarks
Capers Jones, Applied Software Measurement, 3rd Ed. Do You Know Your Numbers?
Phase Introduced
Defects per Function Point (MIS)
Requirements
.75 (.84)
Design
1.50 (1.69)
Code
1.75 (1.97)
Documents
.50 ( - )
Bad Fixes
.50
Appraisal Method
% Removed (MIS)
Unit Test
25%
Function Test
30%
Integration Test
System Test
35%
Acceptance Test
Inspections
60-90%
© 2011 Process-Fusion.net

14. Modeling & Managing Software Process Efficiency and Effectiveness
Leading Indicators Provide CONTROL
© 2011 Process-Fusion.net

15. Why Modeling?
In many software groups finding and fixing defects consumes 50-70% of total cost
Best practice groups reduce that by at least 50%
Models allow you to think through the consequences of alternative strategies … quickly, at very low cost
Models allow you to forecast both quality and financial consequences of alternatives
Creating a business case in the process
Creating a basis for “quality adjusted” status evaluation
Modeling motivates measurement and “management by fact”
© 2011 Process-Fusion.net

16. Model Objectives
Predict (1) delivered quality and (2) total non-value-added effort (cost)
Predict defect “insertion”
Focus attention on defects, which account for the largest share of total development cost.
Enable early monitoring of the relationship between defects likely to be present and those actually found – provide early awareness.
Estimate effort needed to execute the volume of appraisal necessary to find the number of defects we forecast to remove.
a ‘sanity check’ on the planned level of appraisal effort – i.e., is it actually plausible to remove an acceptable volume of defects with the level of effort planned?
Forecast both “pre-release” (before delivery) and “post-release” (after delivery) NVA effort.
When delivered quality is poor, post-release defect repair costs can be 50% of the original project budget.
© 2011 Process-Fusion.net

17. IMPORTANT Caveats Don’t focus on the parameter values I have used
The thought process is the important part
Actual values vary considerably from place to place
Where available I have used industry benchmarks
All benchmarks conceal large variation
Where benchmarks are not available I’ve used experience as a guide
Your local values may well be quite different
Use models such as these to do “what if” analysis
“simulate” a range of assumptions
Best/worst/most likely values
© 2011 Process-Fusion.net

18. Scenarios Evaluated
© 2011 Process-Fusion.net

19. Simulation Results - Containment
I will be happy to provide a copy of the model and related articles for your use
© 2011 Process-Fusion.net

20. Simulation Results: Non-Value-Added
“fix”
“find”
© 2011 Process-Fusion.net

21. © 2011 Process-Fusion.net

22. Cost of Quality Revisited
WHEN you invest matters more than how much
© 2011 Process-Fusion.net

23. “An apple a day …”
Formal inspections, conducted in accordance with IEEE Std , are always efficient & effective … better than any form of testing
Maximum benefits come when applied to requirements, architecture, and design
YOU can both reduce cost (improve productivity) and deliver better quality
© 2011 Process-Fusion.net

24. Thank You! terima kasih 謝謝
© 2011 Process-Fusion.net