1. Sponsored by the U.S. Department of Defense © 2003 by Carnegie Mellon University Version 1.0 Pittsburgh, PA 15213-3890 CMMI Technology Conference 2003 - page 1 ASI CMMI PILOT Status Briefing

2. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 2 Topics Pilot Highlights -Activities to Date -Planned Activities -Realized and Expected Benefits to ASI -About The ASI Pilot Team Path Forward for ASI -Expected Cost and Benefit of CMMI-based Process Improvement -Potential Post-pilot Activities

3. Sponsored by the U.S. Department of Defense © 2003 by Carnegie Mellon University Version 1.0 Pittsburgh, PA 15213-3890 CMMI Technology Conference 2003 - page 3 Pilot Highlights

4. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 4 Activities To Date (1 of 6)  CMMI Overview Tutorial that covered Process Areas of Maturity Levels 2 & 3 and focused upon the typical business impacts related to the area  Business Analysis to capture high impact/high need Process Areas for ASI  Had simple “thumbs-up” voting technique to determine level of impact/need  Also polled participates for the level of impact (H M L)  The high impact areas were obvious and it was easy to select the Process Areas to focus on for the pilot  CMMI Overview Tutorial that covered Process Areas of Maturity Levels 2 & 3 and focused upon the typical business impacts related to the area  Business Analysis to capture high impact/high need Process Areas for ASI  Had simple “thumbs-up” voting technique to determine level of impact/need  Also polled participates for the level of impact (H M L)  The high impact areas were obvious and it was easy to select the Process Areas to focus on for the pilot Gap Analysis Session Action Plan Implementation Close Interaction Between ASI And Consultants Site Kickoff Meeting

5. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 5 Activities To Date (2 of 6)  3 Process Areas were selected by the joint SED/SEI/ASI team :  Project Planning  Requirements Management  Measurement & Analysis Decided to “informally” work on Organizational Process Focus  3 Process Areas were selected by the joint SED/SEI/ASI team :  Project Planning  Requirements Management  Measurement & Analysis Decided to “informally” work on Organizational Process Focus Gap Analysis Session Action Plan Implementation Close Interaction Between ASI And Consultants Site Kickoff Meeting

6. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 6 Activities To Date (3 of 6)  Workshop to analyze the process areas selected - the concepts/methods used for the SCAMPI B/C development project were adapted for ASI:  Current ASI practices documented and mapped to relevant CMMI model components  Interpretation of model intent for ASI made and gaps documented  Developed Action Plan for ASI Team to address gaps found  Workshop to analyze the process areas selected - the concepts/methods used for the SCAMPI B/C development project were adapted for ASI:  Current ASI practices documented and mapped to relevant CMMI model components  Interpretation of model intent for ASI made and gaps documented  Developed Action Plan for ASI Team to address gaps found Site Kickoff Meeting Close Interaction Between ASI And Consultants Gap Analysis Session Action Plan Implementation

7. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 7 Activities To Date (4 of 6)  ASI Team Members are in various stages of implementing their Action Plan  ASI Team Members developed Project-Specific Process Descriptions  Project Planning  Requirements Management  Measurement and Analysis  ASI Quality Facilitator evaluating impact at the organizational level and relationship to QSM  ASI Team Members are in various stages of implementing their Action Plan  ASI Team Members developed Project-Specific Process Descriptions  Project Planning  Requirements Management  Measurement and Analysis  ASI Quality Facilitator evaluating impact at the organizational level and relationship to QSM Site Kickoff Meeting Close Interaction Between ASI And Consultants Gap Analysis Session Action Plan Implementation

8. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 8 Gap Analysis Session Activities To Date (5 of 6)  Monthly Face-to-Face sessions to status progress and get help/guidance from Consultants  Tutorial on writing process guidance documentation  Tutorial on business-oriented metrics  Tutorial on Process Capturing Techniques  Tutorial on CMMI Institutionalization Concepts  Feedback on Action Plans and Process Descriptions  Monthly Face-to-Face sessions to status progress and get help/guidance from Consultants  Tutorial on writing process guidance documentation  Tutorial on business-oriented metrics  Tutorial on Process Capturing Techniques  Tutorial on CMMI Institutionalization Concepts  Feedback on Action Plans and Process Descriptions Site Kickoff Meeting Action Plan Implementation Close Interaction Between ASI And Consultants

9. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 9 Activities to Date (6 of 6)  Weekly teleconferences to keep the project progressing  Review Action Items  Provide feedback on recently reviewed material  Provide guidance on any issues/problems encountered  Weekly teleconferences to keep the project progressing  Review Action Items  Provide feedback on recently reviewed material  Provide guidance on any issues/problems encountered Site Kickoff Meeting Gap Analysis Session Action Plan Implementation Close Interaction Between ASI And Consultants

10. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 10 Planned Activities (1 of 5)  Baseline Process Descriptions  Execute processes in Pilot Projects  Collect Metrics  Gather lessons learned (including benefits) and process improvements  Update processes to reflect process improvements  Evaluate processes for standardization at the Organizational Level  Baseline Process Descriptions  Execute processes in Pilot Projects  Collect Metrics  Gather lessons learned (including benefits) and process improvements  Update processes to reflect process improvements  Evaluate processes for standardization at the Organizational Level Execute New Processes Appraise Pilot Projects Brief SED Share Lessons Learned Publish Pilot Results

11. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 11 Planned Activities (1 of 5)  Prepare for Appraisal  Define Scope (Requirements Management, Project Planning, Measurement and Analysis, and Organizational Process Focus are candidate process areas)  Have regular preparation sessions with Lead Appraiser  Gather objective evidence to support appraisal activities  Set expectations  Conduct Appraisal  ASI to provide one person for the Appraisal Team  Team Training and Readiness Review (tentative: 04/27-04/30)  Appraisal (tentative: 05/24-05/27)  Prepare for Appraisal  Define Scope (Requirements Management, Project Planning, Measurement and Analysis, and Organizational Process Focus are candidate process areas)  Have regular preparation sessions with Lead Appraiser  Gather objective evidence to support appraisal activities  Set expectations  Conduct Appraisal  ASI to provide one person for the Appraisal Team  Team Training and Readiness Review (tentative: 04/27-04/30)  Appraisal (tentative: 05/24-05/27) Execute New Processes Appraise Pilot Projects Brief SED Share Lessons Learned Publish Pilot Results

12. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 12 Planned Activities (3 of 5)  Briefing Scheduled for 02/05/04 02/05 at ASI  Opportunity for ASI to provide  capabilities briefing  Pilot Status  Pilot Benefits  Lessons Learned  Opportunity for ASI to discuss with Mr. Craig his plans after the pilot  Briefing Scheduled for 02/05/04 02/05 at ASI  Opportunity for ASI to provide  capabilities briefing  Pilot Status  Pilot Benefits  Lessons Learned  Opportunity for ASI to discuss with Mr. Craig his plans after the pilot Execute New Processes Appraise Pilot Projects Brief SED Share Lessons Learned Publish Pilot Results

13. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 13 Planned Activities (4 of 5)  ASI invited to present at SE2 Conference  Half-day tutorial – March 29 Sharing materials and activities conducted with ASI and Cirrus with interested members of HSV small business community No ASI-specific information will be provided without ASI permission Opportunity for ASI to share what it deems appropriate  CMMI Panel – March 31 Provide Small Business Perspective for CMMI Implementation Share Lessons Learned  Possible Interviews with Huntsville Times for pre-conference articles  ASI invited to present at SE2 Conference  Half-day tutorial – March 29 Sharing materials and activities conducted with ASI and Cirrus with interested members of HSV small business community No ASI-specific information will be provided without ASI permission Opportunity for ASI to share what it deems appropriate  CMMI Panel – March 31 Provide Small Business Perspective for CMMI Implementation Share Lessons Learned  Possible Interviews with Huntsville Times for pre-conference articles Execute New Processes Appraise Pilot Projects Brief SED Share Lessons Learned Publish Pilot Results

14. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 14 Planned Activities (5 of 5)  Interpretive Guide for Using CMMI in Small Businesses  Focusing on general lessons learned/recommendations gathered during implementation of the various processes at ASI/Cirrus  Materials used (i.e., tutorials, pilot processes) to be included as jumpstart for other small businesses  Case Study: ASI  Published jointly among ASI, SED, SEI  All three organizations agree on content  Case Study: Cirrus  Same conditions as for Cirrus case study  Interpretive Guide for Using CMMI in Small Businesses  Focusing on general lessons learned/recommendations gathered during implementation of the various processes at ASI/Cirrus  Materials used (i.e., tutorials, pilot processes) to be included as jumpstart for other small businesses  Case Study: ASI  Published jointly among ASI, SED, SEI  All three organizations agree on content  Case Study: Cirrus  Same conditions as for Cirrus case study Execute New Processes Appraise Pilot Projects Brief SED Share Lessons Learned Publish Pilot Results

15. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 15 Realized and Expected Benefits to ASI (1 of 8) CMMI Implementation has significantly improved ability to communicate status of the project Use of CMMI-recommended practices has reduced “lottery- sensitivity” on the project -Much of what was only in my head is now accessible to other team members CMMI Implementation has positively affected the way I organize my project Requirements Management Process Description developed during the pilot works for my project and should tailor easily to other “service” projects Will fully execute and document entire process with recently received major requirements changes There is no turning back now. I am hooked! Patty…

16. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 16 Realized and Expected Benefits to ASI (2 of 8) Documentation created as a result of using CMMI: Fills in missing gaps in ISO documentation Is a natural follow onto ISO and meets ISO goals for continual improvement Communicates program roles to all stakeholders Provides (for the first time!) step-by-step instructions for new employees (especially PMs and TLs) Provides “bragging rights” in proposals and presentations Supports greater (and needed!) degree of granularity in organizational performance measurement Helps to identify trouble spots (and strengths) in organizational practices Provides a consistent approach for implementing ASI processes and procedures Larry…

17. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 17 Realized and Expected Benefits to ASI (3 of 8) Experience gathered in communication of practices has proven to be of great benefit in clarifying expectations among team members Through CMMI Implementation, the ASI Team has gained knowledge of existing ASI systems and practices that were not previously clear After the pilot is complete, ASI will have the knowledge and experience necessary to implement process improvement more efficiently The pilot project has widened the ASI Team’s thoughts toward process design and improvement ASI Team Members will now have insight, skills, and initiative to develop new ideas into new practices, with confidence The fear of change has been released from those involved There seems to be a renewed sense of interest and understanding towards process automation I see a future that includes many benefits currently unrealized Jimi…

18. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 18 Realized and Expected Benefits to ASI (4 of 8) CMMI has provided a best practice model to develop process and procedures needed in ASI QMS. Pilot project has provided some confidence that we are doing the right things. “Bottoms-up” encouraged by CMMI structure has made it easier to develop a difficult procedure. Expect expansion to other programs to work well. CMMI specifically addresses customer related processes, many of which are very relevant to the services we provide. Integrated view of processes, through CMMI Process Area interactions, has significantly improved our ability to define more effective processes Jack…

19. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 19 Realized and Expected Benefits to ASI (5 of 8) The Pilot provided the ASI Team with tools an techniques that will make implementation of improvements more cost-effective CMMI has helped me envision a path to “take the company to the next level” -Develop consistent, lasting capabilities that support our business objectives and go beyond having good people -Increase competitive advantage through specific capabilities, documented as ASI processes, that provide solutions for customers -Move away from being totally people dependent to a capability- based organization with proven processes that deliver desired results to customers. Jack…

20. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 20 Realized and Expected Benefits to ASI (6 of 8) The CMMI can be easily incorporated into ASI’s QMS The bottom-up approach encouraged by the CMMI, coupled with ASI’s QMS, will connect those parts of the system which can benefit from specific process improvements Incorporating CMMI practices into ASI’s QMS will give direction to our improvement goals and, in turn, increase customer satisfaction CMMI will strengthen ASI’s QMS and support company objectives as we grow If we incorporate strategic planning (both at he project and corporate levels) and measurable goals while streamlining our processes to suit our particular activities, our QMS would benefit Beth…

21. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 21 Realized and Expected Benefits to ASI (7 of 8) ASI can use CMMI implementation as risk avoidance for ISO audit findings looking for evidence of continual improvement CMMI is providing a common point of reference for working towards standardizing management approaches across different tasks Use of CMMI-referenced process descriptions has already started to reduce training effort for new staff CMMI usage has added insight to development of a standard work authorization process CMMI focus on measurement has added insight into risks related to staffing variances ASI will achieve higher visibility of relationship of processes to revenue, through quarterly reviews Your Consultants see the following as key realized or expected benefits …

22. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 22 Realized and Expected Benefits to ASI (8 of 8) ASI staff has gained understanding of how a bottom-up approach can be used to improve processes in the QMS ASI has gained understanding of what it takes to implement CMMI Process Areas and how to approach implementation ASI will get more visibility in the community as a company that pursues and invests in quality ASI has started to use best practices to respond to new requirements (Subcontractor Agreement Management is an example) ASI will gain CMMI appraisal experience and be counted as one of the few that have gone through an “official” appraisal (see next set of slides) Your Consultants see the following as key realized or expected benefits …

23. Carnegie Mellon University Software Engineering Institute CMMI v1.1 Today -– The Current State November 2003 CMMI v1.1 Today - Page 23 © 2003 by Carnegie Mellon University Highlights from “ CMMI Today: The Current State” PRESENTED BY Bob Rassa, Raytheon Industry CMMI Chair Clyde Chittister Chief Operating Officer, Software Engineering Institute Nov 18, 2003 SUMMARIZED BY SANDRA CEPEDA

24. Carnegie Mellon University Software Engineering Institute CMMI v1.1 Today -– The Current State November 2003 CMMI v1.1 Today - Page 24 © 2003 by Carnegie Mellon University Current Appraisal Synopsis Based on SCAMPI v1.1 appraisals conducted since April 2002 release through June 2003 and reported to the SEI by July 2003. appraisals organizations participating companies reappraised organizations projects offshore organizations Please refer to: Terms Used in this Report on page 20 Additional charts providing different views and break down of this information will be added to this briefing as more of these types of appraisals are reported to the SEI over time.

25. Carnegie Mellon University Software Engineering Institute CMMI v1.1 Today -– The Current State November 2003 CMMI v1.1 Today - Page 25 © 2003 by Carnegie Mellon University Based on organizations Reporting Organizational Types

26. Carnegie Mellon University Software Engineering Institute CMMI v1.1 Today -– The Current State November 2003 CMMI v1.1 Today - Page 26 © 2003 by Carnegie Mellon University Based on organizations reporting SIC code. For more information visit: http://www.osha.gov/oshstats/sicser.html Types of Organizations Based on Primary Standard Industrial Classification (SIC) Code

27. Carnegie Mellon University Software Engineering Institute CMMI v1.1 Today -– The Current State November 2003 CMMI v1.1 Today - Page 27 © 2003 by Carnegie Mellon University Based on organizations reporting size data Organizational Size Based on the total number of employees within the area of the organization that was appraised

28. Carnegie Mellon University Software Engineering Institute CMMI v1.1 Today -– The Current State November 2003 CMMI v1.1 Today - Page 28 © 2003 by Carnegie Mellon University Based on most recent appraisal of organizations reporting a maturity level rating Summary Organizational Maturity Profile % of Organizations

29. Carnegie Mellon University Software Engineering Institute CMMI v1.1 Today -– The Current State November 2003 CMMI v1.1 Today - Page 29 © 2003 by Carnegie Mellon University Summary CMMI Adoption is widespread DoD and Commercial US and off-shore CMMI Adoption is on much faster pace than SW-CMM at the same period of its life CMMI Training and Appraisals continue to increase at rapid pace Application Notes have covered the software-only users Update of content will occur under careful deliberation and consideration in a controlled and open process CMMI will always be sensitive to expressed user needs CMMI ROI Data will be collected, aggregated and disseminated CMMI Training will be updated as needed to meet user requirements

30. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 30 About The ASI Pilot Team Dedicated, smart, creative, wonderful people! Worked extra hard to meet their project commitments as well as their commitment to us Jack’s leadership and hands-on involvement has been a significant contributor to the success of the pilot Thank you for giving us top-notch people!!!

31. Sponsored by the U.S. Department of Defense © 2003 by Carnegie Mellon University Version 1.0 Pittsburgh, PA 15213-3890 CMMI Technology Conference 2003 - page 31 Path Forward for ASI Cost and Benefit of CMMI-based Process Improvement Potential Post-pilot Activities

32. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 32 Value of CMMI CMMI helps organizations… Improve delivery of promised performance, cost, and schedule Integrate stakeholders into project activities Provide competitive world-class products and services Implement an integrated, enterprise, business and engineering perspective Use common, integrated, and improving processes for systems and software Implement proactive program management techniques Enable staff members to move between projects and still use the same processes Create and improve processes that adapt to a changing business environment

33. Sponsored by the U.S. Department of Defense © 2003 by Carnegie Mellon University Version 1.0 Pittsburgh, PA 15213-3890 CMMI Technology Conference 2003 - page 33 Value Specific to Services Organizations (1 of 4) CMMI PAs in Project Management Project Planning Project Monitoring and Control Supplier Agreement Management Integrated Project Management Risk Management Quantitative Project Management Can Help Organization Determine the staff and schedule needed to support the customer Manage costs and stay on schedule Effectively manage their suppliers Be concerned about current and future risks to their organizations Integrate their functions with other project functions Know how to apply the predictability of their service support functions

34. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 34 Value Specific to Services Organizations (2 of 4) CMMI PAs in Support Configuration Management Process and Product Quality Assurance Measurement and Analysis Causal Analysis and Resolution Decision Analysis and Resolution Can Help Organization Know the measures of the cost of providing services and profitability, and the cost of quality Control technical and management work products Ensure their services meet quality objectives Be able to make informed selections of products or techniques for their customers Track service issues to root causes Provide services, including establishing work environments that mitigate the risks of engineering and integration tasks

35. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 35 Value Specific to Services Organizations (3 of 4) CMMI PAs in Engineering Requirements Management Requirements Development Technical Solution Product Integration Verification Validation Can Help Organization Develop and manage their support requirements Provide services that provide technical stability and support all aspects of product development and fielding Ensure interface compatibilities prior to integration Provide technical support to ensure that products are implemented correctly Evaluate the suitability of acquired products

36. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 36 Value Specific to Services Organizations (4 of 4) CMMI PAs in Process Management Organizational Process Focus Organizational Process Definition Organizational Training Organizational Process Performance Organizational Innovation and Deployment Can Help Organization Build and Maintain the Service Organization and its Culture Implement processes to Support the Service Staff Train staff members to perform their service functions Measure the effectiveness and performance of management and service processes Introduce new service functions to their customers

37. Notional Business Case for CMMI Reduced Development/ Maintenance/Service Costs Improved Productivity Less Rework Increased Revenue/Profitability Improved Customer Satisfaction Reduced Post-Release Defects Measurable Improvements of Reliability and Quality Repeat Business Increased Product/Service Sales Reduced Cycle Time Improved Process Performance Improved Service Time Enhanced Time-to-Market Performance Enhanced Service Delivery Time Bonuses for Early Delivery Improved Professional Staff Improved Employee Morale Increased Developer/Maintainer/ Service Provider Confidence Reduced Employee Turnover and Retraining Costs Improved Competitive Advantage Better Products and Services Out Sooner And Cheaper Copyright 2003, CSSA, Inc. Used with permission.

38. Building on the Legacy of SW-CMM Success for Product Development Improvements From Adopting SW-CMM (SEI, 1994) Savings vs. cost of software process improvement (median) 5:1 Percentage Improvement Annual Medians +35% -19% Time to Market -39% Post-Release Defect Reports Productivity 0 5 10 15 20 25 30 35 40 Current ROI Value to Programs (DACS, 1999) Application of SPI to “Example Organization With Example Projects”: Development CostsReduced 73% Rework Costs Reduced 96% Average Schedule Length Reduced 37% Post-Release Defects Reduced 80% Weighted Risk Likelihood Reduced 92% Return On Investment 21:1 Expect Even Higher ROI For CMMI

39. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 39 Expectations for Service Organizations Ensure stability, consistency, and quality of ASI services while making a fair profit Protect ASI from “lottery sensitivity” Develop ability to sell “processes that make ASI capable” instead of “the individual who can solve your problem” Percentage Improvement +N% -N% Service Times Customer Satisfaction Cost +N% -N% +N% Continuity of Services Value of technical/ management services

40. DoD Contractors Have Additional Motivation To Transition To The CMMI The Entire Supply Chain Is or Will Be Involved in CMMI DoDCustomers Associate Contractors Suppliers Competitors May Require Organizational Maturity or Process Capability in RFP, Implicitly or Explicitly May Use Organizational Maturity or Process Capability As a Discriminator Will Use Organizational Maturity or Process Capability to Advise Customers Will Use Organizational Maturity or Process Capability to Competitive Advantage Improve Integration of Products and Services Provide Insight Into Supplier Performance and Quality

41. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 41 Cost of CMMI Implementation We don’t have any aggregate data for the CMMI yet Treat CMMI Implementation Like a Project -Estimate based on previous process improvement efforts -Estimate based on pilot project cost

42. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 42 Summary--Benefits The bad news: Statistically-based cost/benefit is difficult to determine in organizations without a history of performance measurements that support the ROI case The good news: Even without the statistically-based ROI case, solving known (through the Business Analysis process) ASI problems using CMMI as the approach reduces risk exposure Anecdotal benefits seen by ASI so far are consistent with those seen in larger organizations, and costs (in terms of ASI team participation as an improvement team) are consistent/low percentage-wise in comparison to larger organizations Deployment of improved processes from the pilot is likely to be easier/faster for ASI than for a larger organization

43. © 2003 by Carnegie Mellon University Version 1.0 CMMI Technology Conference 2003 - page 43 Potential Post-Pilot Activities Manage ISO implementation and CMMI implementation as an integrated activity to support QMS Formalize Process Improvement Infrastructure to sustain and increase reach of improvement benefits Determine next set of process areas to implement -Perform Business Analysis with larger management group -Pursue Maturity Level X in incremental fashion if considered relevant to current business context Develop ASI-wide Process Improvement Plan Leverage ASI Pilot Team Expertise