An Effective and Efficient Approach to ARC C Appraisals SCAMPI ‘C’: A Foundation for Successful SCAMPI ‘A’ Northrop Grumman Mission Systems has conducted a quarter of the total SCAMPI ‘A’ appraisals across the globe in year 2002. This paper discusses how SCAMPI ‘C’ appraisals have helped NGMS divisions for conducting successful SCAMPI ‘A’ appraisals and there by achieving their organizational goals for process improvement. The presentation will focus on How SCAMPI C appraisals helped the organization to conduct SCAMPI ‘A’ in 30 to 40% less time (60 hours against the industry average of 90 hours) than the industry average and at lower cost. How this approach has made projects focus on small but steady improvements while still meeting their customer schedule & commitments How the SCAMPI ‘C’ appraisals have increased the confidence of the organization to go for SCAMPI ‘A’ appraisal How properly scheduled SCAMPI ‘C’ provided low cost and efficient assessment exercise for process improvement How the use of tools and organization wide guidance helped in institutionalizing the process improvements (using the bottom up approach – inputs from projects to organization) How the CMMI implementation was quantitatively measured and monitored across the entire organization How SCAMPI ‘C’ was used in providing the organization wide strategic direction on process improvement How the organization wide resource pool was established (with right knowledge and skills) to support various appraisals Experiences of NGMS in conducting SCAMPI ‘B’ and ‘C’ appraisals How Six Sigma was used to improve the NGMS appraisal process The other lessons learned The intended audiences for this presentation are ·Organizations implementing CMMI and are aiming at a maturity level ·Organizations planning to implement CMMI ·Organizations with different types of projects, multiple locations looking for efficient ways of enterprise wide process assessments   CMMI Technology Conference & User Group 17-20 November 2003 Madhu Parella, Satyam msraoparella@yahoo.co.in Janice Tauser, Northrop Grumman janice.tauser@ngc.com Rick Hefner, Northrop Grumman rick.hefner@ngc.com

Agenda Appraisal Classes Why ARC C? Our Approach ARC C Tools Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. Agenda Appraisal Classes Why ARC C? Our Approach ARC C Tools Measured Benefits In this presentation SCAMPI A is referred as SCAMPI. SCAMPI B and SCAMPI C are referred as ARC B and ARC C or Class B and Class C appraisal.

Appraisal Method Classes Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. Appraisal Method Classes Appraisal Requirements for CMMI (ARC) introduction Various requirements for ARC Classes This slide is to bring the audience to a common platform (considering that there may be some who are not familiar with appraisal classes). This presentation tries providing various measures taken to increase the confidence in ARC C appraisal results while maintaining the low cost. Source: CMU/SEI-2001-TR-034

Copyright 2003 Northrop Grumman Space and Mission Systems Corp Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. Why ARC C? Baseline CMMI implementation status for less resources and time than SCAMPI Establish a common internal benchmarking method Provide a Go/No Go decision for SCAMPI Formalized method minimizes the subjectivity associated with different appraisers, different projects These two slides provide information on why such a formal appraisal technique is required when some of the objectives like independent verification can be met by internal audit function. ARC C is a formal verification method with the flexibility of less formal verification methods like internal audit etc. Baseline CMMI implementation status: When the project first fills responses to SAT questions, the responses are based on project understanding which may or may not match organization/CMMI expectations. When an ARC C is performed all these issues are resolved and that becomes the true process improvement status on the project against which the monthly progress is measured. Uniform implementation/verification approach: For Large organizations like NGMS with different kinds of projects a verification method like ARC C provides common way of measurement across the organization. ARC C Provides formal, out of routine verification (audits) feeling to the projects.

Copyright 2003 Northrop Grumman Space and Mission Systems Corp Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. Six Sigma Six Sigma methodology was used to analyze the entire CMMI appraisal process – preparation, onsite Based on 20+ SCAMPIs conducted to date Goal was to reduce the overall time spent Projects preparing for the appraisal Appraisers conducting the appraisal Also used to increase the stability (predictability) and effectiveness of the appraisal process Appraisal is viewed as a process that must be measured and continuously improved Six sigma methodology is used to identify improvements in the appraisal process. Various six sigma projects like improvement of the overall appraisal process, improvement of evidence collection and presentation process have direct impact on conducting of ARC C appraisals on a project.

Our Approach Documented process – an ARC C requirement! Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. Our Approach Documented process – an ARC C requirement! Used to verify gaps at the start of the improvement, and to verify readiness for SCAMPI A Performed by 1 or 2 qualified appraisers, collaborating with key project personnel Uses the same evidence notebooks used for SCAMPI A, although may include helping to identify the evidence Uses our SCAMPI A consolidation and rating tools This slide explains NGMS approach in conducting an ARC C and how the confidence level of the ARC C appraisal findings is improved.

Using the Results for Appraisal Readiness Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. Using the Results for Appraisal Readiness Typically conducted for appraisal readiness 2 months prior to the SCAMPI A Have found that if more than 40 practices have insufficient evidence, should delay the appraisal Should conduct another ARC C appraisal prior to new appraisal date The number of practices that have insufficient evidence greatly impacts SCAMPI A duration Schedules are built expected 20 or fewer evidence requests This slide and next slide explains how the results from an ARC C are used to decide on when a project is likely to get ready for a SCAMPI A

Infrastructure to Support ARC C Appraisals Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. Infrastructure to Support ARC C Appraisals Qualified People – SCAMPI Lead Appraisers SCAMPI ATM trained Integrated set of tools for evidence gathering, reviews, and reporting Common documented procedures, standards and templates How NGMS institutionalizes and supports ARC C? Primarily through having a pool of qualified people who can support ARC C, Providing various tools that are required for implementation as well as verification Developing processes to support conducting of an ARC C Providing standards and templates for various common activities and use. The set of integrated tools support both projects and organizations in various phases of the process improvement life cycle. These integrated set of tools compliment each other in implementation and verification process (for example, QA can generate the process verification questionnaire based on the tailored process). More details are provided in the next slide. A documented procedure for ARC C helped in implementing the process uniformly across the organization. NGMS

Integrated Set of Tools Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. Integrated Set of Tools Self Assessment Tool (SAT) Projects record gaps against CMMI practices Policy Compliance and Tailoring Report Tool (PCTR) New projects document intended compliance Audit Checklists QA verifies compliance with policy & process Appraisal Findings Tool Appraisers record compliance with CMMI practices This chart shows various tools associated with process improvement life cycle phases. These tools ensures consistent implementation of various practices across the organization.

Benefits of Using Tools Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. Benefits of Using Tools Project road map (prepared and owned by project) Roll up to multiple levels in the organization (aligned with organizational goals) Common measurement system across the projects Quantitative – no intuition or judgments Provides indirect evidence Clearly defined responsibility for each activity The set of integrated tools support the overall process improvement at NGMS. These tools help projects plan and monitor their own process improvement. Also the tools are helpful in knowing the status of the organization process improvement status and rolling up of projects data to organization level. The status is measured uniformly across the organization with clearly defined responsibility for each activity. Since these tools are updated and the data is owned by projects, they also provide indirect evidence for an appraisal. Some of the tools facilitate quantitative measurement of process improvement activities.

Quantitative Measurement for CMMI Compliance Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. Quantitative Measurement for CMMI Compliance Self Assessment Tool (SAT) This shows an example of the how the various process improvement activities are measured and monitored using the quantitative data, This data is used both at project and organization level. Identify areas requiring special attention Planned Vs Actual Progress

Implementation Deployment Strategy Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. Measurement Data Flow Projects Implementation Measurement Data roll up Deployment OU Level The process improvement status is baselined and progress data is collected at project level (shows the planned and actual status every month). Project measurements are rolled up to organizational unit level and provides input to the organizational S/SEPG group for deployment of process improvements. Organization level measurements are collected and presented to Senior Management indicating areas of performance and areas of concern. At the Organization level these measurements provide inputs for process improvement strategy. Measurement Data roll up Strategy Org. Level

Copyright 2003 Northrop Grumman Space and Mission Systems Corp Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. Organization Support Defined & documented method for ARC C appraisal at organizational level Management tracked quantitative measures of CMMI compliance across all levels of the organization Made ARC C a prerequisite for SCAMPI A Provided an CMMI interpretations document to ensure common understanding Provided expert advise, where required Shared lessons learned through SSEPG meetings, etc. This slide gives more details on the NGMS support to ARC C appraisals. The organization support helps institutionalizing the process quicker. Experts like SCAMPI Lead appraisers and Six sigma black belts and subject matter experts provide advise where required. These lessons learned are used as inputs to update the CMMI guidance documents in the organization.

Feedback Input to Organizational Strategy Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. Feedback Input to Organizational Strategy Provide additional guidance Conduct ARC C appraisals Collect appraisal results at org. level Projects use the additional guidance Appraisals typically provide information about the process improvement bottlenecks. By ensuring this feedback is used as one of the inputs to the process improvement strategy, timely guidance can be provided to the projects. If an organization depends on the SCAMPI feedback, it may loose some important updates to organization strategy to be communicated on time. The number of SCAMPI A appraisals are limited in an organization compared to to ARC C. Also the ARC C is conducted in the earlier phases of the process improvement life cycle and the findings help an organization fix the problems before going to SCAMPI A appraisal. Identify common problem areas What happens if organizations depend only on SCAMPI to provide this feedback to strategy?

How ARC C Supports our SCAMPIs Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. How ARC C Supports our SCAMPIs Project Road Map First ARC C Baseline Second ARC C Monthly Progress Pre Assessment This flow chart shows the major steps involved in the process improvement life cycle of a project going for a SCAMPI A appraisal Initially the project fills the SAT tool which acts as a project road map for achieving CMMI. This is based on the project’s understanding of process improvement activities. The first ARC C is conducted to make sure that project’s understanding is inline with organization/CMMI expectations. The output of this ARC C may be a typical gap analysis report. This forms a baseline for measurement of process improvement progress. When the project reaches a certain milestone an intermediate ARC C is conducted. Based on the results the projects are advised further on their approach to achieve CMMI maturity level goal. In some cases the same ARC C may be taken as pre assessment for a final SCAMPI appraisal. If required an additional ARC C is scheduled for further verification. SCAMPI Readiness SCAMPI A Additional ARC C

ARC C: Effect on SCAMPI A Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. ARC C: Effect on SCAMPI A Activities with minor or no Impact Activities significantly Impacted Team Training Opening Briefing Evidence Review This chart shows a typical flowchart for a SCAMPI A appraisal indicating the activities that are impacted by an ARC C appraisal. The time taken for activities like evidence review and number of evidence requests will be substantially less when one or more ARC C appraisals are conducted prior to a SCAMPI A. These activities take major time share in a SCAMPI appraisal onsite period. The evidence review is much faster in verification mode and a prior ARC C helps an appraisal to continue in verification mode. In discovery mode the number of evidence requests tend to be more than verification. Mini team consolidation and full team consolidation take less time due to development of common interpretations of various CMMI practices. Interviews Evidence Requests Mini Team Consolidation Draft Briefing Full Team Consolidation Final Briefing

ARC C: Effect on SCAMPI A (continued) Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. ARC C: Effect on SCAMPI A (continued) Keeps SCAMPI A in ‘verification’ mode Where was maximum time spent in a CBA IPI assessment? Control gate for SCAMPI A – Saves resources (and agony?) Familiarized project/organization with appraisal process Majority of model interpretation issues dealt already Obvious weaknesses are minimized ARC C keeps any of the obvious weaknesses dealt before the SCAMPI appraisal and brings out many interpretation issues. The appraisal team can communicate their requirements much easily when a project/organization becomes familiar with appraisal process and requirements.

SCAMPI Effort With and Without ARC C Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. SCAMPI Effort With and Without ARC C For a Level 3 organization with 3 medium to large projects, and a 10 person SCAMPI A team SCAMPI with ARC C 4 member ARC C team typically spent 150 hours SCAMPI A appraisal team spent 450 hours SCAMPI without ARC C SCAMPI A appraisal team spent 600 hours Based on small number of appraisals, our experience shows it does not consume significantly more resources or more time to conduct an ARC C Appraisal prior to a SCAMPI A appraisal. The resources spent in both the cases shown above are almost equal. In the second case (with ARC C) the appraisal process remains in verification mode and has more probability to achieve the maturity level. In the first case, typically the teams spend more onsite time and this calculation takes an average of 10 hours per day for a 6 to 7 day appraisal In the second case an ARC C with a four member qualified team takes approximately 4 days for an ARC C and this might go up based on the organization history in process improvement. In this case the on site appraisal time is limited to 5 days and the experience shows that they tend to spend less onsite time. This onsite appraisal effort is dependent on various other factors (explained in the six sigma slide) along with a prior ARC C. Total Effort 600 Hrs

Copyright 2003 Northrop Grumman Space and Mission Systems Corp Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved. Conclusions ARC C: Provides an efficient and affordable verification approach Helps identify areas of improvement early Supports entire CMMI implementation life cycle For NGMS ARC C provided an efficient verification approach and its suitability and flexibility to use at various phases of life cycle has provided many benefits