1. Sponsored by the U.S. Department of Defense © 2003 by Carnegie Mellon University page 1 Pittsburgh, PA 15213-3890 CMMI ® CMMI ® : Success and Opportunities Ahead 2 nd Annual QAAM Conference, 22-23 October 2003 Mike Konrad SM SCAMPI, SCAMPI Lead Appraiser, and SEI are service marks of Carnegie Mellon University. ®CMMI, Capability Maturity Model, and CMM are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.

2. © 2003 by Carnegie Mellon University page 2 CMMI ® This Presentation Overview of CMMI The Past—where we’ve been and what has influenced us The Present—where we are The Future—where we are going and how you can help Results to Date Adoption –- who is using CMMI and where they are located Impact –- the results achieved by early adopters

3. © 2003 by Carnegie Mellon University page 3 CMMI ® The Past era of “manufacturing in quality” The premise of “manufacturing in quality” implies a focus on processes as well as on products is a long-established premise in manufacturing is based on Total Quality Management principles as taught by Shewhart, Juran, Deming, and Humphrey “It costs a lot of money to build bad products.” Augustine’s 12 th Law “The quality of a system is highly influenced by the quality of the process used to acquire, develop, and maintain it.” Phillip Crosby “Quality is Free: The Art of Making Quality Certain”

4. © 2003 by Carnegie Mellon University page 4 CMMI ® The Present era of “engineering in quality” CMMs ® focus on process and product quality business results engineering in quality CMMs continue to be adopted and used Companies in key markets are adopting CMMs defense aerospace IT, computer automotive entertainment telecommunications finance

5. © 2003 by Carnegie Mellon University page 5 CMMI ® Multiple Process Models Success of the Software CMM ® caused development of other CMMs, but they Have different structures, formats, terms, ways of measuring maturity Cause confusion, especially when more than one are used Are difficult to integrate into a combined improvement program Are difficult to use in supplier selection Software CMM Software CMM Systems Security Engr. CMM Systems Security Engr. CMM Systems Engr. CMM Systems Engr. CMM People CMM People CMM IPD CMM IPD CMM Software Acq. CMM Software Acq. CMM EIA 731

6. © 2003 by Carnegie Mellon University page 6 CMMI ® Sunsetting of SW-CMM Introduction to SW-CMM training course The last public offering from the SEI SM is December 2003. This course will continue to be available from SEI transition partners. CBA-IPI assessments and SCE evaluations The last Lead Assessor (LA) training is in December 2003. The last Lead Evaluator (LE) training is in October 2003. LA and LE authorizations expire on December 31, 2005; LAs and LEs must upgrade to SCAMPI SM to continue providing SEI-authorized appraisal services. For more information, see the Sunset FAQ at.www.sei.cmu.edu/cmmi/adoption/sunset-faq.html

7. © 2003 by Carnegie Mellon University page 7 CMMI ® The Future era of “innovating in quality” We face unprecedented engineering challenges. Customers demand quality products faster and cheaper. Management expects higher productivity. Engineering fields continually evolve and merge. Organizations are dynamic; there is much more partnering. Our knowledge and experience must be shared. The future is now! CMMI is our knowledge infrastructure

8. © 2003 by Carnegie Mellon University page 8 CMMI ® CMMI Is Integration and Improvement CMMI supports process integration and product improvement. CMMI integrates multiple disciplines into one process- improvement framework that eliminates inconsistencies and reduces duplication. CMMI provides a framework for introducing new disciplines as needs arise and therefore reduces the cost of implementing model-based improvement. CMMI is designed to minimize the impact on legacy process improvement efforts and investment.

9. © 2003 by Carnegie Mellon University page 9 CMMI ® CMMI Models CMMI-SW Staged Representation CMMI-SW Continuous Representation CMMI –meets the needs of software organizations –is an upgrade of SW-CMM –benefits from best practices contributed from all three source models Source Models Capability Maturity Model ® for Software V2, draft C (SW-CMM V2C) EIA 731, System Engineering Capability Model (SECM) Integrated Product Development Capability Maturity Model, draft V0.98 (IPD-CMM)

10. © 2003 by Carnegie Mellon University page 10 CMMI ® Project Management Requirements Management Requirements Development Technical Solution Product Integration Verification Validation Engineering Configuration Management Process and Product Quality Assurance Measurement and Analysis Decision Analysis and Resolution Organizational Environment for Integration Causal Analysis and Resolution Support Project Planning Project Monitoring and Control Supplier Agreement Management Integrated Project Management for IPPD Risk Management Integrated Teaming Integrated Supplier Management Quantitative Project Management Organizational Process Focus Organizational Process Definition Organizational Training Organizational Process Performance Organizational Innovation and Deployment Process Management Process Areas

11. © 2003 by Carnegie Mellon University page 11 CMMI ® GG2: Institutionalize a Managed Process Generic Practices Generic Goals GP 1.1: Perform Base Practices GG1: Achieve Specific Goals GG3: Institutionalize a Defined Process GP 3.1: Establish a Defined Process GP 3.2: Collect Improvement Information GG4: Institutionalize a Quantitatively Managed Process GP 4.1: Establish Quantitative Objectives for the Process GP 4.2: Stabilize Subprocess Performance GG5: Institutionalize an Optimizing Process GP 5.1: Ensure Continuous Process Improvement GP 5.2: Correct Root Causes of Problems GP 2.1: Establish an Organizational Policy GP 2.2: Plan the Process GP 2.3: Provide Resources GP 2.4: Assign Responsibility GP 2.5: Train People GP 2.6: Manage Configurations GP 2.7: Identify and Involve Relevant Stakeholders GP 2.8: Monitor and Control the Process GP 2.9: Objectively Evaluate Adherence GP 2.10: Review Status with Higher Level Management Generic Goals and Associated GPs

12. © 2003 by Carnegie Mellon University page 12 CMMI ® Improving on the Software CMM CMMI models improve on SW-CMM Version 2.0 Draft C: incorporate additional years of learning more explicitly link best practices to business objectives expand the scope of and visibility into the product life cycle and engineering activities add more best practices, (e.g., measurement, risk management, product integration, decision analysis and resolution, and supplier management) capture more robust high-maturity practices address additional generic practices needed for institutionalization more fully comply with relevant ISO standards

13. © 2003 by Carnegie Mellon University page 13 CMMI ® One Model, Two Representations Maturity Level 5 OID, CAR Maturity Level 4 OPP, QPM Maturity Level 3 REQD, TS, PI, VER, VAL, OPF, OPD, OT, IPM, RSKM, DAR Overview Introduction Structure of the Model Model Terminology Maturity Levels, Common Features, and Generic Practices Understanding the Model Using the Model Maturity Level 2 REQM, PP, PMC, SAM, MA, PPQA, CM Appendixes Engineering REQM, REQD, TS, PI, VER, VAL Project Management PP, PMC, SAM IPM, RSKM, QPM Process Management OPF, OPD, OT, OPP, OID Process Management PAs - Goals - Practices Support CM, PPQA, MA, CAR, DAR Appendixes CMMI-SE/SW Staged Overview Introduction Structure of the Model Model Terminology Capability Levels and Generic Model Components Understanding the Model Using the Model CMMI-SE/SW Continuous

14. © 2003 by Carnegie Mellon University page 14 CMMI ® Advantages of Each Representation Continuous RepresentationStaged Representation Provides maximum flexibility for order of process improvement Predefined and proven path with case study and ROI data High visibility of improvement within process areas Focuses on organizational improvement Easy upgrade from EIA 731Easy upgrade from SW-CMM Easy comparison to ISO 15504Provides familiar benchmarking capability Improvement of process areas can occur at different rates Overall results summarized in a maturity level

15. © 2003 by Carnegie Mellon University page 15 CMMI ® CMMI in a Nutshell A CMMI model provides a structured view of process improvement across an organization. CMMI can help set process improvement goals and priorities provide guidance for quality processes provide a yardstick for appraising current practices

16. © 2003 by Carnegie Mellon University page 16 CMMI ® CMMI Today Version 1.1 of the CMMI Product Suite was released January 2002. CMMI models will not change until 2005 at earliest. Many defense, aerospace, and commercial organizations are upgrading to CMMI. One appraisal method, SCAMPI, covers internal process improvement supplier source selection contract process monitoring

17. © 2003 by Carnegie Mellon University page 17 CMMI ® Discoveries in Use Ease of upgrade to CMMI has been reported by Multiple SW-CMM level 5 organizations that upgraded and maintained their maturity level (accomplished in twelve months) Multiple organizations that upgraded from EIA 731 systems engineering assessments Numerous European companies in group discussion of CMMI adoption at recent SEI-Europe quarterly meeting Mappings and gap analyses confirm evolutionary expansion from predecessor models Government and contractors agree on CMMI’s improved engineering coverage in contract monitoring ISO/CMMI compatibility appears favorable

18. © 2003 by Carnegie Mellon University page 18 CMMI ® Where We’re Going Adoption activities Transition Partner data workshops, technical notes, and book publication Interpretive Guidance project Appraisal enhancement activities SCAMPI appraisal data CMMI appraisals conducted worldwide SCAMPI enhancements Training activities CMMI training data training course upgrades

19. © 2003 by Carnegie Mellon University page 19 CMMI ® Adoption—What’s Happening Now Events related to CMMI adoption quarterly transition workshops annual NDIA/SEI CMMI User Workshop Interpretive Guidance project Technical notes and special reports: CMMI and Product Line Practices CMMI and Earned Value Management Interpreting CMMI for Operational Organizations Interpreting CMMI for Service Organizations (in progress) CMMI Mappings Specific interests (e.g., safety, security) Publication of SEI Series Book with Addison-Wesley

20. © 2003 by Carnegie Mellon University page 20 CMMI ® Interpretive Guidance Project The Interpretive Guidance project is investigating whether there is information in CMMI models that may be more easily used by software, IT, and IS organizations if special guidance or interpretation is provided. This project has collected information in the following ways: birds-of-a-feather sessions at conferences workshops at SPIN meetings expert group meetings detailed interviews with select software organizations feedback from SCAMPI appraisals a Web-based questionnaire For more information about the Interpretive Guidance Project, see.

21. © 2003 by Carnegie Mellon University page 21 CMMI ® Expert Group Members Received 31 nominations and selected 12 members Members of the group include: 1.Joseph Billi, Automatic Data Processing 2.Dr. Bill Curtis, TeraQuest 3.Doug Ebert, Vice McKesson Corporation 4.Christian Hertneck, Siemens, Corporate Technology 5.Pat O'Toole, Process Assessment, Consulting & Training 6.M. Lynn Penn, Lockheed Martin Management & Data Systems 7.Bill Peterson, Software Engineering Institute 8.Gowri S. Ramani, Hewlett Packard 9.Terry Rout, Software Quality Institute, School of Computing and Information Technology, Griffith University 10. Mark Servello, ChangeBridge, Inc. 11. Rosalind Singh, CAE 12. Gary Wolf, Manager, Raytheon

22. © 2003 by Carnegie Mellon University page 22 CMMI ® Preliminary Report Describes the data-collection activities from both BoF sessions and Web-based questionnaire efforts Includes summaries of the data collected through August 2003

23. © 2003 by Carnegie Mellon University page 23 CMMI ® Web-Based Questionnaire Notified ~7,000 people 4,000 people with direct internet access 3,000 people that the questionnaire was available placed an announcement on the SEI Web site The numbers of responses received for the sections of the questionnaire were: 668 Background and Context (required section) 587 Global Issues 339 Generic Goals and Generic Practices 182 Specific Process Areas

24. © 2003 by Carnegie Mellon University page 24 CMMI ® How would you best describe your familiarity with CMMI? 1% 19% 25% 54% 1% 0%10%20%30%40%50%60%70%80% Never heard of it Heard of it Use it occasionally Use it regularly Didn't respond Percent of Respondents Total Respondents = 668

25. © 2003 by Carnegie Mellon University page 25 CMMI ® Has your organization made a decision about adopting CMMI? 23% 48% 15% 10% 4% 0%10%20%30%40%50%60%70%80% Decision not made yet Adoption in progress Well institutionalized in organization Chosen not to adopt CMMI Didn't respond Percent of Respondents Total Respondents = 668

26. © 2003 by Carnegie Mellon University page 26 CMMI ® In your opinion, is CMMI adequate for guiding process improvement? 35% 42% 12% 1% 10% 0% 10%20%30%40%50% Almost always More often than not Sometimes Rarely if ever Don't know Didn't Respond Total Respondents = 587 Percent of Respondents

27. © 2003 by Carnegie Mellon University page 27 CMMI ® Adopting CMMI will help us to leverage our earlier investments in process improvement. 29% 47% 6% 1% 13% 4% 0%10%20%30%40%50% Strongly Agree Agree Disagree Strongly Disagree Don't Know Didn't Respond Total Respondents = 587 Percent of Respondents

28. © 2003 by Carnegie Mellon University page 28 CMMI ® The cost of adopting CMMI is impeding the adoption of CMMI in our organization. 16% 27% 32% 8% 11% 6% 0%10%20%30%40%50% Strongly Agree Agree Disagree Strongly Disagree Don't Know Didn't Respond Total Respondents = 587 Percent of Respondents

29. © 2003 by Carnegie Mellon University page 29 CMMI ® Including both systems engineering and software in a single model has been a help for us. 33% 31% 10% 5% 15% 6% 0%10%20%30%40%50% Strongly Agree Agree Disagree Strongly Disagree Don't Know Didn't Respond Total Respondents = 587 Percent of Respondents

30. © 2003 by Carnegie Mellon University page 30 CMMI ® Having a choice between the two model representations (staged or continuous) and variations (SW, SE, IPPD, SS) has been helpful for us. 14% 35% 17% 6% 20% 8% 0%10%20%30%40%50% Strongly Agree Agree Disagree Strongly Disagree Don't Know Didn't Respond Total Respondents = 587 Percent of Respondents

31. © 2003 by Carnegie Mellon University page 31 CMMI ® Preliminary Observations-1 The responses were overwhelming positive. Much of the data collected is not unique to commercial software, IT, and IS organizations. Similar data was reported by organizations in disciplines such as systems engineering and acquisition. Most organizations believe that CMMI is adequate for guiding process improvement activities and that their prior investments in process improvement have helped them to adopt CMMI. Although the majority of respondents believe that existing CMMI training courses, guidance documents, and appraisal methods are suitable for their organization’s needs, we received specific comments that identify areas for improvement.

32. © 2003 by Carnegie Mellon University page 32 CMMI ® Preliminary Observations-2 Having two model representations appears to cause concern and confusion for some organizations; however, others liked the flexibility provided by two representations. There was no obvious preference of one representation over the other. We received the most negative input in the area of costs. For many respondents, high costs inhibited their adoption of CMMI. However, others felt that cultural issues inhibited adoption. Responses were mixed as to whether return on investment (ROI) information is needed to help organizations make a business case for adopting CMMI. Many organizations felt that ROI data is necessary while others felt that ROI data would not have made a difference in making their decision to adopt CMMI.

33. © 2003 by Carnegie Mellon University page 33 CMMI ® How Well is CMMI Meeting Your Process Improvement Needs? To ensure the continuing quality and usability of the CMMI Product Suite, the SEI and CMMI Steering Group are now beginning to explore whether updates to the the CMMI Product Suite should be considered. Your input into this exploration process during the next 90 days (until December 12, 2003) will ensure that your comments are taken into account. See more information about how to participate in the 90-day comment period at.http://www.sei.cmu.edu/cmmi/continuing-improvement.html

34. © 2003 by Carnegie Mellon University page 34 CMMI ® CMMI Training—What’s Happening? CMMI training course materials have been and will continue to be improved based on change requests submitted by students and instructors: Introduction to CMMI, Staged Representation Introduction to CMMI, Continuous Representation Intermediate Concepts of CMMI CMMI Instructor Training

35. © 2003 by Carnegie Mellon University page 35 CMMI ® CMMI Appraisals—What’s Happening? SCAMPI Class B & Class C appraisal methods These methods are now being developed and piloted. Public release is planned for 2004. SCAMPI for SW-CMM and other models has been announced.

36. © 2003 by Carnegie Mellon University page 36 CMMI ® CMMI Transition Status (30 Sept. 2003) Training Introduction to CMMI—9,354 trained Intermediate CMMI—684 trained Introduction to CMMI Instructors—190 trained SCAMPI Lead Appraisers SM —299 trained Authorized Introduction to CMMI V1.1 Instructors—152 SCAMPI V1.1 Lead Appraisers—225 Transition Partners For CMMI introduction training—55 For appraisal services—128

37. © 2003 by Carnegie Mellon University page 37 CMMI ® CMMI Process Maturity Profile The following slides excerpt the report: “CMMI v1.1 Appraisal Results: A First Look, September 2003” available on the SEI Web at: www.sei.cmu.edu/sema/profile.html

38. © 2003 by Carnegie Mellon University page 38 CMMI ® Current Status SCAMPI v1.1 appraisals conducted since April 2002 release through June 2003 and reported to the SEI by July 2003: 100 appraisals 93 organizations 52 participating companies 6 reappraised organizations 357 projects 54%offshore organizations

39. © 2003 by Carnegie Mellon University page 39 CMMI ® Reporting Organization Types Based on 91 organizations

40. © 2003 by Carnegie Mellon University page 40 CMMI ® Types of Organizations Based on 45 organizations reporting Standard Industrial Classification (SIC) Code *Draft profile report

41. © 2003 by Carnegie Mellon University page 41 CMMI ® Organization Size Based on the total number of employees within the area of the organization that was appraised (92 orgs reporting) *Draft profile report

42. © 2003 by Carnegie Mellon University page 42 CMMI ® Disciplines Selected for Appraisals SW= Software Engineering SE= System Engineering IPPD= Integrated Product and Process Development SS= Supplier Sourcing Based on 99 appraisals reporting disciplines

43. © 2003 by Carnegie Mellon University page 43 CMMI ® Maturity Profile by Organization Type Based on most recent appraisal of 65 organizations reporting organization type and a maturity level rating

44. © 2003 by Carnegie Mellon University page 44 CMMI ® Summary Organizational Maturity Profile Based on most recent appraisal of 66 organizations reporting a maturity level rating

45. © 2003 by Carnegie Mellon University page 45 CMMI ® USA and Offshore Summary Organizational Maturity Profiles Based on 25 U.S. organizations and 41 offshore organizations reporting their maturity level rating

46. © 2003 by Carnegie Mellon University page 46 CMMI ® Countries Where Appraisals have been Performed and Reported to the SEI Australia Canada China France India JapanKorea, Republic ofRussia Switzerland Taiwan United Kingdom United States

47. © 2003 by Carnegie Mellon University page 47 CMMI ® Cost of quality / ROI Benefits / Savings / ImprovementsCosts Process Capability or Maturity Investment in Improvement Operational Costs Costs of Improvement Rework Effort Predictability Productivity Employee morale Enhanced functionality “ilities” Process compliance Quality Schedule / cycle time Customer satisfaction Product cost + REVENUE / SAVINGS - COSTS / EXPENSES CMMI ® Results Study Framework

48. © 2003 by Carnegie Mellon University page 48 CMMI ® Boeing, Australia Quality Schedule / cycle time Product cost In Processes is there a Pay In Processes is there a Pay-Off? Off?. Terry Stevenson, Boeing Australia, Software Engineering Australia 2003 conference. Making transition to CMMI from SW-CMM and EIA 731; early CMMI pilot in Australia RESULTS on One Project 33% decrease in the average cost to fix a defect Turnaround time for releases cut in half 60% reduction in work from Pre-Test and Post-Test Audits; passed with few outstanding actions Increased focus on product quality Increased focus on eliminating defects Developers seeking improvement opportunities

49. © 2003 by Carnegie Mellon University page 49 CMMI ® Lockheed Martin M&DS SW CMM ML2 (1993) to ML 3 (1996) to CMMI ML5 (2002) Results Award Fees during 2002 are 45% percent of unrealized award fees at ML2 1996 - 2002 Increased software productivity by 30% 16% reduction in Dollars/KLOC Decreased defect find and fix costs by 15% Internal data shared through Collaboration; August 2003. Productivity Product cost Quality Customer satisfaction

50. © 2003 by Carnegie Mellon University page 50 CMMI ® Northrop Grumman IT 1 Defect prevention using PSP and CAR at CMMI ML5 Integrating PSP sm and CMMI ® Level 5. Gabriel Hoffman, Northrop Grumman IT. May 1, 2003. Quality 2.1 3.9 3.5 6.1 6.6 0 1 2 3 4 5 6 7 12345 Build Defect Density (Defects/KLOC) DP 1 CAR DP 3 CAR DP 2 CAR

51. © 2003 by Carnegie Mellon University page 51 CMMI ® Northrop Grumman IT 2 Appraised at CMMI ML 5 in December 2002 Results met 25+ milestones in a row earned a rating of “Exceptional” in every applicable category on a formal Contractor Performance Evaluation Survey Hours Invested: 124 in Defect Prevention (CAR) Hours saved: 1650 hours (15 hours per defect) ROI:13:1 Integrating PSP sm and CMMI ® Level 5. Gabriel Hoffman, Northrop Grumman IT. May 1, 2003 Quality Schedule / cycle time Customer satisfaction Cost of quality / ROI

52. © 2003 by Carnegie Mellon University page 52 CMMI ® Accenture Transition SW-CMM to CMMI May 2001 to May 2002 Transition Time: 1149 person hours Key Content Measurement and Analysis DAR TS, RM, Change Control IPPD visions, OEI Generic Goals Results ROI: 5:1 (for quality activities) Innovation Delivered. CMMI® Level 3 in a Large Multi-Disciplinary Services Organization. Bengzon, 2003 SEPG. Cost of quality / ROI Investment in Improvement

53. © 2003 by Carnegie Mellon University page 53 CMMI ® General Motors Corporation CMMI focus 2001 Goal is Integration of Supplier Work & GM Project Execution Results: Improved schedule – projects met milestones and were fewer days late Camping on a Seesaw: GM’s IS&S Process Improvement Approach. Hoffman, Moore & Schatz, SEPG 2003. Schedule / cycle time

54. © 2003 by Carnegie Mellon University page 54 CMMI ® In Summary In today’s fast-paced, competitive business environment, approaches used in the past such as “manufacturing in quality” and the present such as “engineering in quality” are not enough. The future is innovation. CMMI helps organizations to … improve delivery of performance, cost, and schedule integrate stakeholders into project activities provide competitive world-class products and services implement an integrated enterprise business and engineering perspective use common, integrated, and improving processes for systems and software Upgrade to CMMI now… and lead the way to the future of process improvement.

55. © 2003 by Carnegie Mellon University page 55 CMMI ® For More Information… For more information about CMMI, see www.sei.cmu.edu/cmmi/ www.sei.cmu.edu/cmmi/ For questions about CMMI, send email to cmmi-comments@sei.cmu.edu cmmi-comments@sei.cmu.edu Or, contact SEI Customer Relations Phone: +1 412-268-5800 Email: customer-relations@sei.cmu.edu