Measuring the Benefits of Mature Processes 20th International Forum on COCOMO and Software Cost Modeling 24 October 2005 Many organizations have implemented process improvement efforts (e.g., CMM, CMMI, ISO, Six Sigma), in the hopes of becoming better, cheaper, and faster. Published results on these efforts often contain quantifiable data regarding these benefits. These measures are useful in showing that process improvements will return enough money to justify the investment. Other organizations have achieved their maturity level goals, but appear to see little or no financial benefits. Are the published ROI results untrue, or is it a question of properly measuring the results? This presentation will discuss issues surrounding the measurement of process improvement results. Data and lessons learned on achieving measurable ROI will be presented. A framework for measuring the benefits will be outlined based on Cost of Quality concepts and existing industry models for ROI. Based on these principles, practical tips and techniques for realizing the qualitative and quantitative benefits offered by process improvement will be provided. Insights are based on the Northrop Grumman Mission Systems process improvement program, which has successfully implemented CMM, CMMI, ISO, and Six Sigma based improvements for the past 10 years. Seventeen Northrop Grumman organizations have achieved CMMI Level 5.   Discussion Several authors have discussed the challenges in measuring the benefits of mature processes: In 1994, the Software Engineering Institute (Herbsleb et al. 1994) released a report called “Benefits of CMM-Based Process Improvement”. This report includes information from 13 organizations regarding the benefits of process improvement using the CMM for Software. The paper shows ROI ranging from 4:1 to 8.8:1. In 2003, the SEI (Goldenson and Gibson) presented preliminary ROI data from the adoption of CMMI. They summarized results from 12 organizations, categorized by cost, schedule, quality, and customer satisfaction. Sarah Sheard and Christopher Miller of the Software Productivity Consortium, in “The Shangri-La of ROI”, discuss the difficulty of accurately measuring ROI, and question its value in convincing process skeptics that process improvement is beneficial. The “Cost of Quality” accounting technique introduced in 1951 by Juran to provide ROI justification for process improvements. It was expanded in 1979 by Crosby to “get management’s attention and to provide a measurement base for seeing how quality improvement is doing.” Although used in manufacturing and services industries for controlling costs of quality initiatives and for identifying opportunities to reduce quality costs, it has seen limited adoption in the software and systems engineering community. Six Sigma, as described in numerous texts, explicitly focuses on the financial benefits of improvement. These and related work form the basis for the discussion. Rick Hefner, Ph.D. Northrop Grumman Corporation

Background Many organizations have implemented process improvement efforts (CMM, CMMI, ISO, Six Sigma) to become better, cheaper and faster Some organizations have not realized the quantitative or return- on-investment (ROI) benefits reported in the literature Are the literature claims of ROI true? Are their tricks for getting better ROI? What are the timelines for realizing these benefits? CMM® and CMMI® are registered trademarks of Carnegie Mellon University

Agenda Principles of process improvement Industry data on ROI Issues surrounding the measurement of benefits Strategic actions needed to achieve maximum ROI Northrop Grumman lessons learned

Software Projects Have Historically Suffered from Mistakes People-Related Mistakes 1. Undermined motivation 2. Weak personnel 3. Uncontrolled problem employees 4. Heroics 5. Adding people to a late project 6. Noisy, crowded offices 7. Friction between developers and customers 8. Unrealistic expectations 9. Lack of effective project sponsorship 10. Lack of stakeholder buy-in 11. Lack of user input 12. Politics placed over substance 13. Wishful thinking Process-Related Mistakes 14. Overly optimistic schedules 15. Insufficient Risk Management 16. Contractor failure Insufficient planning 17. Abandonment of planning under pressure 18. Wasted time during the fuzzy front end 19. Shortchanged upstream activities 20. Inadequate design 21. Shortchanged quality assurance 22. Insufficient management controls 23. Premature or too frequent convergence 25. Omitting necessary tasks from estimates 26. Planning to catch up later 27. Code-like-hell programming Product-Related Mistakes 28. Requirements gold-plating 29. Feature creep 30. Developer gold-plating 31. Push me, pull me negotiation 32. Research-oriented development Technology-Related Mistakes 33. Silver-bullet syndrome 34. Overestimated savings from new tools or methods 35. Switching tools in the middle of a project 36. Lack of automated source-code control Standish Group, 2003 survey of 13,000 projects 34% successes 15% failures 51% overruns Reference: Steve McConnell, Rapid Development

Many Approaches to Solving the Problem Which weaknesses are causing my problems? Which strengths may mitigate my problems? Which improvement investments offer the best return? People Business Environment Management Structure Tools Product Methods One solution! Process Technology

Approaches to Process Improvement Data-Driven (e.g., Six Sigma, Lean) Clarify what your customer wants (Voice of Customer) Critical to Quality (CTQs) Determine what your processes can do (Voice of Process) Statistical Process Control Identify and prioritize improvement opportunities Causal analysis of data Determine where your customers/competitors are going (Voice of Business) Design for Six Sigma Model-Driven (e.g., CMM, CMMI) Determine the industry best practice Benchmarking, models Compare your current practices to the model Appraisal, education Identify and prioritize improvement opportunities Implementation Institutionalization Look for ways to optimize the processes

Typical Data-Driven Results Cited in Literature Year Revenue ($B) Invested ($B) Savings ($1B) % Revenue Motorola 1986-2001 356.9(e) ND 16 4.5 Allied Signal 1998 15.1 0.5 3.3 GE 1996 79.2 0.2 1997 90.8 0.4 1 1.1 100.5 1.3 1.2 1999 111.6 0.6 2 1.8 1996-1999 382.1 1.6 4.4 Honeywell 23.6 2.2 23.7 2.5 2000 25.0 0.7 2.6 1998-2000 72.3 2.4 Ford 2000-2002 43.9 2.3 http://www.isixsigma.com/library/content/c020729a.asp

Typical Model-Driven Results Cited in Literature Category Range Yearly cost of process improvement activities $49K - $1,202K Years engaged in SPI 1 – 9 Cost of SPI per engineer $490 - $2,004 Productivity gain per year 9% - 67% Early detection gain per year (defects discovered pre-test) 6% - 25% Yearly reduction in time to market 15% - 23% Yearly reduction in post-release defects 10% - 94% Business value (ROI) 4.0 - 8.8 10:1 Defect Reduction 50% Productivity Increase Rework Drops From 54% to 4% More Accurate Estimates “Benefits of CMM-Based Process Improvement”, Herbsleb et al., Software Engineering Institute, 1994

Typical CMMI Benefits Cited in Literature Reduced costs 33% decrease in the average cost to fix a defect (Boeing) 20% reduction in unit software costs (Lockheed Martin) Reduced cost of poor quality from over 45 percent to under 30 percent over a three year period (Siemens) Faster Schedules 50% reduction in release turnaround time (Boeing) 60% reduction in re-work following test (Boeing) Increase from 50% to 95% the number of milestones met (General Motors) Greater Productivity 25-30% increase in productivity within 3 years (Lockheed Martin, Harris, Siemens) Higher Quality 50% reduction of software defects (Lockheed Martin) Customer Satisfaction 55% increase in award fees (Lockheed Martin) “Demonstrating the Impact and Benefits of CMMI: An Update and Preliminary Results,” Software Engineering Institute, CMU/SEI-2003-SR-009, Oct 2003

The Knox Cost of Quality Model Extension of the Cost of Quality model used in manufacturing Cost Category Definition Typical Costs for Software Conformance Appraisal Discovering the condition of the product Testing and associated activities, product quality audits Prevention Efforts to ensure product quality SQA administration, inspections, process improvements, metrics collection and analysis Non-conformance Internal failures Quality failures detected prior to product shipment Defect management, rework, retesting External failures Quality failures detected after product shipment Technical support, complaint investigation, defect notification Knox’s Theoretical Model for Cost of Software Quality (Digital Technical Journal, vol.5, No. 4., Fall 1993, Stephen T. Knox.)

Knox Model – Theoretical Benefits COCOMO also predicts ~10% increase in productivity for each increase in CMMI Level

Why Measuring ROI is Hard Savings ----------------- Investment ROI = What do you count as “Investment”? Training? QA? Data gathering? New practices? What would we have done instead? What do you count as “Savings? Increased predictability – what’s the value? Increased productivity – who gets the benefit? Better competitive position – how measured? How do you measure the change? Multiple causes – awareness, knowledge, infrastructure Short-term vs. long-term – Hawthorne effect Over what time-frame? See also The Shangri-La of ROI”, Sarah Sheard and Christopher Miller, Software Productivity Consortium

Northrop Grumman Mission Systems Approach Mission Success Requires Multiple Approaches Risk Management Systems Engineering Independent Reviews Training, Tools, & Templates Dashboards for Enterprise- Wide Measurement Communications & Best-Practice Sharing Robust Governance Model (Policies, Processes, Procedures) Program Effectiveness Mission Assurance Process Operations Effectiveness Effectiveness CMMI Level 5 for Software, Systems, and Services ISO 9001 and AS-9100 Certification Six Sigma

Process Effectiveness Audits & Appraisals Staff Competence & Training 5 CMMI & Six Sigma courses Policies & processes course Standard Training Modules for each job function: engineering, project management, QA, CM, etc. 13 Northrop Grumman sites externally appraised at CMMI Level 5 Process Asset Library Communications & Collaboration Assuring mission success by making the people and processes more informed and effective

Program Effectiveness Six Sigma connects process improvement and business value Six Sigma projects can help focus and measure CMMI-driven process improvements Identify the customer’s needs, maximize the value/cost Tools for management by variation (CMMI Levels 4 and 5) Results to date 4000 Green Belts, 200 Black Belts, 12 Master Black Belts 500 completed Six Sigma projects, 250 in progress Significant benefit to our customer – lower costs, better performance DEFINE MEASURE ANALYZE IMPROVE CONTROL Charter team, map process & specify CTQs Measure process performance Identify & quantify root causes Select, design & implement solution Institutionalize improvement, ongoing control Assuring mission success by identifying the customer’s needs and reducing defects

Operational Effectiveness Tying it all together People Product Process Capability Data Repository Software, Hardware, Accounting Productivity, Defects, Maintenance Phase Relationships,Systems Engineering Functions Lessons Learned Parametric Modeling Expertise DOD Software Industry Expertise Risk and Predictive Modeling Analysis Certified Function Point Specialists Six Sigma Black Belts Presence Risk Analysis Cost Estimates Cost Estimation Relationships Program Benchmarking Life Cycle Productivity Analysis Software Sizing and Modeling Predictive Modeling Quantitative Management Professional Society Board Members Active on Government Working groups Key participants on Milestone reviews) Tools Commercial Modeling Tools Northrop Grumman Developed Tools Programs Process Structured Project Reporting, Training Standardization of Data, Metrics Manual, Approval CMMI Measurement, Monitor, Manage, Report, Update and Calibrate Assuring mission success by providing independent cost, schedule and risk realism

Lessons Learned Model-driven and data-driven process improvements compliment each other Model-driven improvements are difficult to measure precisely Long improvement cycles and broad focus make it difficult to isolate cause and effect Substantial anecdotal evidence of significant ROI Data-drive improvements are more easily measured Short improvement cycles, narrow focus Efforts concentrate data, measurement systems, tie improvements to business goals