1. Process Improvement for Software Projects CMMI and Lean Six Sigma USC CSC1 510 16 Nov 2015 Rick Hefner, Ph.D. California Institute of Technology rhefner@Caltech.edu

2. Agenda Current Challenges Facing the Software Industry Capability Maturity Model Integrated Lean Six Sigma Corporate Support 2

3. What are some of the challenges facing software companies? 3

4. Software Projects Have Historically Suffered from Mistakes Reference: Steve McConnell, Rapid Development People-Related Mistakes 1. Undermined motivation 2. Weak personnel 3. Uncontrolled problem employees 4. Heroics 5. Adding people to a late project 6. Noisy, crowded offices 7. Friction between developers and customers 8. Unrealistic expectations 9. Lack of effective project sponsorship 10. Lack of stakeholder buy-in 11. Lack of user input 12. Politics placed over substance 13. Wishful thinking Process-Related Mistakes 14. Overly optimistic schedules 15. Insufficient Risk Management 16. Contractor failure Insufficient planning 17. Abandonment of planning under pressure 18. Wasted time during the fuzzy front end 19. Shortchanged upstream activities 20. Inadequate design 21. Shortchanged quality assurance 22. Insufficient management controls 23. Premature or too frequent convergence 25. Omitting necessary tasks from estimates 26. Planning to catch up later 27. Code-like-hell programming Product-Related Mistakes 28. Requirements gold-plating 29. Feature creep 30. Developer gold-plating 31. Push me, pull me negotiation 32. Research-oriented development Technology-Related Mistakes 33. Silver-bullet syndrome 34. Overestimated savings from new tools or methods 35. Switching tools in the middle of a project 36. Lack of automated source-code control Standish Group survey of 13,000 projects (2003) 34% successes 15% failures 51% overruns Standish Group survey of 13,000 projects (2003) 34% successes 15% failures 51% overruns 4

5. Top Software Engineering Issues in Defense Industry 1.The impact of requirements upon software is not consistently quantified and managed in development or sustainment 2.Fundamental system engineering decisions are made without full participation of software engineering 3.Software life-cycle planning and management by acquirers and suppliers is ineffective 4.The quantity and quality of software engineering expertise is insufficient to meet the demands of government and the defense industry 5.Traditional software verification techniques are costly and ineffective for dealing with the scale and complexity of modern systems 6.There is a failure to assure correct, predictable, safe, secure execution of complex software in distributed environments 7.Inadequate attention is given to total lifecycle issues for COTS/NDI impacts on lifecycle cost and risk 5 “Top Software Engineering Issues In Defense Industry”, NDIA Systems Engineering Division and Software Committee, Sep 2006

6. Many Approaches to Solving the Problem Which weaknesses are causing my problems? Which strengths may mitigate my problems? Which improvement investments offer the best return? People Product Technology Tools Management Structure Business Environment Process Methods 6

7. Heritage of Standards for Systems Engineering 7 EIA / IS 632 ISO/IEC 15288 Mil-Std- 499B Mil-Std- 499A 1994 1998 2002 1974 (Not Released) Mil-Std- 499 1969 (Trial Use) IEEE 1220 1998 (Full Std) Legend Supersedes Source for EIA 632 1998 EIA/IS 731 SE CM IEEE 1220 (Full Std) (Interim Standard) 2002 CMMI - SE/SW/IPPD (Interim Standard) (FDIS) ISO/IEC 19760 2002 (PDTR) 2002 ISO/IEC 15504 (FDIS) Standards for Systems Engineering, Jerry Lake, 2002

8. The Frameworks Quagmire Sarah A. Sheard, Software Productivity Consortium 8 Evolution of the Frameworks Quagmire, Software Magazine

9. Two Complimentary Approaches to Process Improvement Data-Driven (e.g., Lean Six Sigma) Clarify what your customer wants (Voice of Customer) –Critical to Quality (CTQs) Determine what your processes can do (Voice of Process) –Statistical Process Control Identify and prioritize improvement opportunities –Causal analysis of data Anticipate your customers/ competitors (Voice of Business) –Design for Six Sigma Data-Driven (e.g., Lean Six Sigma) Clarify what your customer wants (Voice of Customer) –Critical to Quality (CTQs) Determine what your processes can do (Voice of Process) –Statistical Process Control Identify and prioritize improvement opportunities –Causal analysis of data Anticipate your customers/ competitors (Voice of Business) –Design for Six Sigma Model-Driven (e.g., CMMI) Determine the industry best practice –Benchmarking, models Compare your current practices to the model –Appraisal, education Identify and prioritize improvement opportunities –Implementation –Institutionalization Look for ways to optimize the processes Model-Driven (e.g., CMMI) Determine the industry best practice –Benchmarking, models Compare your current practices to the model –Appraisal, education Identify and prioritize improvement opportunities –Implementation –Institutionalization Look for ways to optimize the processes 9

10. Capability Maturity Model Integrated 10

11. How would you ensure best practices are implemented consistently in your software company?

12. What is the Capability Maturity Model Integrated? The CMMI is a collection of industry best-practices for engineering, services, acquisition, project management, support, and process management –Developed under the sponsorship of DoD –Consistent with DoD and commercial standards Three Constellations sharing common components and structure CMMI for Development - used by engineering organizations CMMI for Acquisition - used by buyers (e.g., govt. agencies) CMMI for Services - used by service providers (e.g., help desk) 12

13. Basic Building Blocks – 22 Process Areas Project Management Project Planning Project Monitoring and Control Supplier Agreement Management Integrated Project Management) Risk Management Quantitative Project Management 13 Engineering Requirements Development Requirements Management Technical Solution Product Integration Verification Validation Support Configuration Management Process and Product Quality Assurance Measurement and Analysis Decision Analysis and Resolution Causal Analysis and Resolution Process Management Organizational Process Focus Organizational Process Definition Organizational Training Organizational Process Performance Organizational Performance Management Implemented by each project Implemented by the organization

14. Expected Practices Provide Guidance for Implementation and Institutionalization 14 SG 1 Establish Estimates SP 1.1Estimate the Scope of the Project SP 1.2Establish Estimates of Work Product and Task Attributes SP 1.3Define Project Lifecycle Phases SP 1.4Estimate Effort and Cost SG 2 Develop a Project Plan SP 2.1Establish the Budget and Schedule SP 2.2Identify Project Risks SP 2.3Plan Data Management SP 2.4Plan the Project’s Resources SP 2.5Plan Needed Knowledge and Skills SP 2.6Plan Stakeholder Involvement SP 2.7Establish the Project Plan SG 3 Obtain Commitment to the Plan SP 3.1Review Plans That Affect the Project SP 3.2Reconcile Work and Resource Levels SP 3.3Obtain Plan Commitment GG 2 Institutionalize a Managed Process GP 2.1Establish an Organizational Policy GP 2.2Plan the Process GP 2.3Provide Resources GP 2.4Assign Responsibility GP 2.5Train People GP 2.6Manage Configurations GP 2.7Identify and Involve Relevant Stakeholders GP 2.8Monitor and Control the Process GP 2.9Objectively Evaluate Adherence GP 2.10Review Status with Higher Level Management GG 3 Institutionalize a Defined Process GP 3.1 Establish a Defined Process GP 3.2 Collect Improvement Information Project Planning – Implementation Project Planning - Institutionalization

15. Practice Ratings for the Organization/Projects 15

16. How is the CMMI Used for Process Improvement? 16 www.sei.cmu.edu/ideal/ IDEAL Model

17. Typical CMMI Benefits Cited in Literature Reduced costs –33% decrease in the average cost to fix a defect (Boeing) –20% reduction in unit software costs (Lockheed Martin) Faster Schedules –50% reduction in release turnaround time (Boeing) –60% reduction in re-work following test (Boeing) Greater Productivity –25-30% increase in productivity within 3 years (Lockheed Martin, Harris, Siemens) Higher Quality –50% reduction of software defects (Lockheed Martin) Customer Satisfaction –55% increase in award fees (Lockheed Martin) 17

18. Lean Six Sigma 18

19. What is Lean Six Sigma (LSS)? Lean Six Sigma is a powerful approach to improving business LSS improvement projects are performed by teams Teams use a set of tools and techniques to understand problems and find solutions Lean Six Sigma integrates tools and techniques from two proven process improvement methods 19 +

20. DMAIC: Define-Measure-Analyze-Improve-Control A proven, structured problem-solving approach for improving existing processes Define: Identify the issue causing the perceived problem or improvement opportunity Measure: Collect data from the process Analysis: Study the data for cause and effect Improve: Eliminate cause(s) and determine resulting effects Control: Sustain the gains by monitoring the system 20

21. DMAIC Toolkit 21 Charter Kano Model Voice of the Customer Voice of the Business Quality Function Deployment Voice of the Process Data Collection Methods Measurement System Evaluation Statistical Controls Control Charts Time Series methods Non-Statistical Controls Procedural adherence Performance Management Preventive measures Design of Experiments Modeling ANOVA Tolerancing Robust Design Systems Thinking Decision & Risk Analysis Cause & Effect Diagrams/ Matrix Failure Modes & Effects Analysis Statistical Inference Reliability Analysis Root Cause Analysis, including 5 Whys Hypothesis Test DefineMeasureAnalyzeImproveControl

22. Lean - Definition Lean is the philosophy of improving performance by reducing waste and eliminating bottlenecks Used together, the two methodologies become a powerful set of tools for improving business processes "The 8 Wastes - DOWNTIME" by Source (WP:NFCC#4). Licensed under Fair use via Wikipedia. 22

23. Organizational Adoption: Roles & Responsibilities Champions – Facilitate the leadership, implementation, and deployment Sponsors – Provide resources Process Owners – Responsible for the processes being improved Master Black Belts – Serve as mentors for Black Belts Black Belts – Lead major Six Sigma projects –Typically requires 4 weeks of training Green Belts – Lead minor Six Sigma teams, or serve on improvement teams under a Black Belt –Typically requires 2 weeks of training 23

24. A Typical Lean Six Sigma Project in Aerospace The organization notes that systems integration has been problematic on past projects (budget/schedule overruns) A Six Sigma team is formed to scope the problem, collect data from past projects, and determine the root cause(s) The team’s analysis of the historical data indicates that ineffective peer reviews are leaving significant errors to be found in test Procedures and criteria for better peer reviews are written, using best practices from past projects A pilot project uses the new peer review procedures and criteria, and collects data to verify they solve the problem The organization’s standard process and training is modified to incorporate the procedures and criteria, to prevent similar problems on future projects 24

25. Corporate Support 25

26. Organizational Infrastructure Required 26 Policies, Processes, Templates & Tools Best-Practice Libraries Process Group Audits & AppraisalsMeasurement Repositories Predictive Modeling Developing and maintaining mature processes requires significant time and investment in infrastructure Process ImprovementTraining Program Communications

27. Northrop Grumman Approach: Institutionalizing Our Improvements 27 Internal Best Practices ISO/AS9100 Findings CMMI Appraisal Findings Policy Procedures Process e Toolkit PAL WorkbenchStartIt!My MS Portal Tools Information Checklists and Guides Templates and Examples Disposition Independent Audits Systems/ Software Engineering Process Group Working Groups Program Management Advisory Board Lessons Learned & Metrics Analysis Customer Comments Configuration Control Board External Best Practices Industry Standards Six Sigma Projects We systematically analyze quality and process data and trends to determine how to improve our processes We improve our process assets based on internal and external best practices Deployed to programs Increasing program efficiency msCAS PCDB

28. Lessons Learned Multiple improvement initiatives helps encourage a change in behavior as opposed to “achieving a goal” –Reinforces that change (improvement) is a way of life Benefits result from institutionalizing local improvements across the wider organization –CMMI establishes the needed mechanisms Training over half the staff as Lean Six Sigma Green Belts resulted in a change of language and culture –Voice of Customer, data-driven decisions, causal analysis, etc. –Better to understand/use tools in everyday work than to adopt the “religion” CMMI and Lean Six Sigma compliment each other –CMMI can yield behaviors without benefits –Lean Six Sigma improvements based solely on data may miss innovative improvements (assumes a local optimum) 28