Interpretive Guidance Project: What We Know CMMI User’s Conference November 18, 2003 SM SCAMPI, SCAMPI Lead Appraiser, and SEI are service marks of Carnegie Mellon University. ® Capability Maturity Model Integration, Capability Maturity Model, Capability Maturity Modeling, CMMI, and CMM are registered in the U.S. Patent & Trademark Office.

Topics Project Overview and Status Preliminary Report Summary Questions

Interpretive Guidance Objectives To understand and address the issues that software organizations have when using CMMI To allow current SW-CMM users to more easily upgrade to CMMI To eliminate as many perceived barriers to CMMI adoption as possible To make CMMI adoption easy

Project Status Highlights – Phase I Phase I activities have been completed. Collected comments from Birds-of-a-Feather sessions in conjunction with conferences and SPIN meetings Formed expert group Received responses from Web-based questionnaire Received very limited feedback from SCAMPISM appraisals Performed preliminary analysis of issues Released Interpretive Guidance Preliminary Report (available on CMMI Website)

Project Status Highlights – Phase II Purpose is to analyze issues to determine: if interpretive guidance is needed where interpretive guidance is appropriate what form interpretive guidance will take Phase II activities just getting started. At a minimum we will perform detailed analysis of the issues conduct detailed interviews to investigate issue areas further meet with groups at SEI to share analysis and to understand how their activities support/address identified issues present preliminary data at conferences to make sure that the data is correct and our analysis is accurate produce a final report to document our findings and conclusions interpretive guidance for software organizations summary and conclusions from this task

Preliminary Report Describes the data-collection activities from both Birds of a Feather (BoF) sessions and Web-based questionnaire efforts Includes summaries of the data collected through August 2003

BoF Sessions Attended CMMI Users Group ICSPI Conference New York City SPIN QAAM/QAI Conference on Managing Software Excellence PROFES 2002 Acquisition of SW-Intensive Systems SEPG 2003 Southern California SPIN meeting San Diego SPIN meeting bITa Europe Conference NDIA Transition Workshop STC 2003 European SEPG Conference Practical Software Measurement

Web-Based Questionnaire Notified ~7,000 people 4,000 people with direct internet access 3,000 people that the questionnaire was available placed an announcement on the SEI Web site The numbers of responses received for the sections of the questionnaire were: 668 Background and Context (required section) 587 Global Issues 339 Generic Goals and Generic Practices 182 Specific Process Areas

Background Nine questions were asked to understand the background of the respondent Some questions were specific to the person filling out the questionnaire Other questions were providing background information about the organization Highlights are presented in the following slides

How would you best describe your familiarity with CMMI? Didn't respond 1% Total Respondents = 668 Use it regularly 54% Use it occasionally 25% Heard of it 19% Never heard of it 1% 0% 10% 20% 30% 40% 50% 60% 70% 80% Percent of Respondents

What if any CMMI training have you received? (Multiple responses were permitted) SCAMPI team training 17% Total Respondents = 668 SCAMPI lead appraiser training 17% CMMI instructor training 10% Intermediate CMMI 35% Introduction to CMMI 95% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Percent of Respondents

conducted in your organization since June 2000? Approximately how many appraisals, if any, have been conducted in your organization since June 2000? (Multiple responses were permitted) Other appraisal methods or gap analyses 2695 Total Respondents = 668 SCE 335 CBA IPI 1395 SCAMPI class B or C 1077 SCAMPI class A 374 500 1000 1500 2000 2500 3000 Number of appraisals conducted

Has your organization made a decision about adopting CMMI? Didn't respond 4% Total Respondents = 668 Chosen not to adopt CMMI 10% Well institutionalized in organization 15% Adoption in progress 48% Decision not made yet 23% 0% 10% 20% 30% 40% 50% 60% 70% 80% Percent of Respondents

What is your personal role in process improvement? (Multiple responses were permitted) 26% 73% 42% 27% 35% 13% 0% 10% 20% 30% 40% 50% 60% 70% 80% Policy for adoption of new technologies Support for process improvement activities Member of the management team Member of the technical staff Consultant to organizations (CMM or CMMI) Other Percent of Respondents Total Respondents = 668

Approximately how many full-time equivalent (FTE) employees does your organization employ who are primarily engaged in the development, maintenance, or acquisition of software or software-intensive systems? 31% 29% 37% 3% 0% 10% 20% 30% 40% 50% 60% 70% 80% Less than 100 100 to 500 More than 500 Didn't respond Percent of Respondents Total Respondents = 668

How would you best describe your familiarity with the Software CMM? Didn't respond 2% Use it regularly 65% Use it occasionally 21% Heard of it 11% Total Respondents = 668 Never heard of it 1% 0% 10% 20% 30% 40% 50% 60% 70% 80% Percent of Respondents

Global Issues Thirteen questions were asked General questions that address CMMI adoption CMMI concepts or terminology model representations costs ROI Highlights are presented in the following slides

In your opinion, is CMMI adequate for guiding process improvement? 35% 42% 12% 1% 10% 0% 20% 30% 40% 50% Almost always More often than not Sometimes Rarely if ever Don't know Didn't Respond Total Respondents = 587 Percent of Respondents

Adopting CMMI will help us to leverage our earlier investments in process improvement. 29% 47% 6% 1% 13% 4% 0% 10% 20% 30% 40% 50% Strongly Agree Agree Disagree Strongly Disagree Don't Know Didn't Respond Total Respondents = 587 Percent of Respondents

Existing CMMI training courses, guidance documents, web resources, and other process assets are adequate for our purposes. 9% 48% 17% 6% 15% 5% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% Strongly Agree Agree Disagree Strongly Disagree Don't Know Didn't Respond Total Respondents = 587 Percent of Respondents

Existing CMMI appraisal methods are suitable for our organization's needs. Didn't Respond 5% Don't Know 26% Total Respondents = 587 Strongly Disagree 4% Disagree 15% Agree 39% Strongly Agree 11% 0% 10% 20% 30% 40% 50% Percent of Respondents

The cost of adopting CMMI is impeding the adoption of CMMI in our organization. Didn't Respond 6% Don't Know 11% Total Respondents = 587 Strongly Disagree 8% Disagree 32% Agree 27% Strongly Agree 16% 0% 10% 20% 30% 40% 50% Percent of Respondents

Including both systems engineering and software in a single model has been a help for us. Didn't Respond 6% Don't Know 15% Total Respondents = 587 Strongly Disagree 5% Disagree 10% Agree 31% Strongly Agree 33% 0% 10% 20% 30% 40% 50% Percent of Respondents

We have had difficulty in mapping our processes to the CMMI. 3% 15% 41% 18% 16% 7% 0% 10% 20% 30% 40% 50% Strongly Agree Agree Disagree Strongly Disagree Don't Know Didn't Respond Total Respondents = 587 Percent of Respondents

We have had difficulty tracking the changes and additions from models that we have previously used. 2% 10% 43% 11% 26% 8% 0% 20% 30% 40% 50% Strongly Agree Agree Disagree Strongly Disagree Don't Know Didn't Respond Total Respondents = 587 Percent of Respondents

Having a choice between the two model representations (staged or continuous) and variations (SW, SE, IPPD, SS) has been helpful for us. 14% 35% 17% 6% 20% 8% 0% 10% 30% 40% 50% Strongly Agree Agree Disagree Strongly Disagree Don't Know Didn't Respond Total Respondents = 587 Percent of Respondents

Does your organization need ROI or other quantitative evidence to help make the business case for adopting CMMI? 24% 44% 14% 12% 6% 0% 10% 20% 30% 40% 50% Yes, we must have it Yes, it certainly would help to have No, we've already built a good business case No, it's not a real issue for us Didn't Respond Total Respondents = 587 Percent of Respondents

The responses were overwhelmingly positive. Conclusions The responses were overwhelmingly positive. Much of the data collected is not unique to commercial software, IT, and IS organizations. Similar data was reported by organizations in disciplines such as systems engineering and acquisition. We still need to conduct our detailed analysis. Copies of the report are available on the CMMI Website at http://www.sei.cmu.edu/cmmi/.

? Questions ? ?

For More Information About CMMI Go to CMMI Web site: http://www.sei.cmu.edu/cmmi http://seir.sei.cmu.edu Contact SEI Customer Relations: Customer Relations Software Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213-3890 FAX: (412) 268-5800 customer-relations@sei.cmu.edu