Using Behavior Surveys for CMMI Process Deployment Presentation Title September 21, 2018 Using Behavior Surveys for CMMI Process Deployment Introduce speakers… This presentation is a practitioner’s view for deployment of process. It presents a practical approach to identifying process gaps and monitoring process deployment. The presentation does not discuss the CMMI model, but discusses a method for achieving Level 5. Raytheon North Texas Software Engineering Donna Freed Speaker Name

Overview: Using Behavior Surveys for CMMI Process Deployment Presentation Title September 21, 2018 Overview: Using Behavior Surveys for CMMI Process Deployment PPQA Support of CMMI Level 4 and 5 N TX Software Engineering Process Journey SW Steps to CMMI Level 5 R6 Process for Achieving Level 5 What Is a Behavior Survey? Survey Results Improvements Made: Levels of Control Incremental Planning Metrics Analysis focused on defect containment, and statistical process control What is the PPQA role? What is the history of N TX Software engineering? What was the approach for achieving Level 5? How did we monitor our progress and identify areas for improvement? What was the result? <insert pool story here> 9/21/2018 Speaker Name

PPQA Focuses on Improvement Presentation Title September 21, 2018 PPQA Focuses on Improvement Improve and Strengthen Process by Closing Gaps Step 3: Identify and Resolve Gaps Work with Process Owners R6 Specialist Projects Optimize Execution Step 2: How Well Process Is Supporting Programs Determine Process Gaps Identify Process Improvement The CMMI model says the level 2 process area PPQA: supports the delivery of high-quality products and services by providing the project staff and all levels of managers with appropriate visibility into, and feedback on, the processes and associated work products throughout the life cycle. CMMISM for Systems Engineering/Software Engineering, Version 1.1 (CMMI-SE/SW, V1.1) Staged Representation To achieve CMMI Level 5, PPQA focuses on improvement, to identify improvements and help close gaps in process maturity, with utilization of 6 sigma. This is the framework for today’s presentation on a creative approach to the PPQA process area. It is just about audits and noncompliances, but about continuous improvement. I hope I’m preaching to the choir. Step 1: How Well We Are Executing Feedback to Program and Process Causal Analysis 9/21/2018 3 Speaker Name

N TX Software Engineering Typical Program Presentation Title September 21, 2018 N TX Software Engineering Typical Program Few Large Programs Customer/End User: Military Number of Eng’s: 10 Number of SW Engr’s: 5 Program Duration: 18 Months PPQA Support: 3 Programs Per SQE How do we characterize software engineering in North Texas…What is our typical program? Over the last 3 years, North Texas had a staff of 23 software quality engineers on 45 active programs within software engineering. Many Small Programs 9/21/2018 Speaker Name

N TX Software Engineering Process Improvement Journey Presentation Title September 21, 2018 N TX Software Engineering Process Improvement Journey # minute Level 5 CMMI Level 5 Level 5 RTIS Integrated Product Development Process Level 4 IPI Level 5 Level 5 RTIS Integrated RTIS Integrated RTIS Integrated RTIS Integrated Level 4 Level 4 CMMI CMMI Software Sigma & Cycle Time Software Sigma Software Sigma Level 4 Level 4 CMMI CMMI RTIS Policies & Procedures RTIS Software Operating Instructions Product Product Software Sigma Software Sigma IPI IPI RTIS Policies & RTIS Policies & Product Product RTIS Policies & RTIS Policies & Development Development & Cycle Time & Cycle Time IPI IPI Procedures Procedures Development Development & Cycle Time & Cycle Time Procedures Procedures Process Process Process Process Level 3 Baseline Validation Level 3 Level 3 RTIS Software RTIS Software Level 4 Level 4 Level 4 RTIS Software RTIS Software 10x Fault Density Improvement Baseline Baseline Operating Operating 10x 10x Operating Operating Validation Validation Level 2 Repeatable Level 3 Defined 1989 1991 1990 1992 1993 1994 Level 4/5 Managed/ Optimizing Raytheon Acquisition & Transition 2000 1995 1996 1997 1999 - 2001 2002 2003 Baldridge Award Fault Density Fault Density Instructions Instructions Baldridge Baldridge Level 3 Level 4/5 Managed/ Optimizing Raytheon Acquisition & Transition 2000 1995 1996 1997 1999 - 2001 2002 2003 2000 1995 1996 1997 1999 - 2001 2002 2003 2003 Instructions Instructions Level 3 Level 3 2002 2003 2003 Award Award Improvement Improvement 2001 2002 2002 Level 2 2000 Raytheon Acquisition & Transition Level 4/5 Managed/ Optimizing 2001 2001 Level 2 Level 2 1997 1999 - 2000 2000 1996 - - 1999 1999 1995 1996 1996 1997 1997 1989 1991 1990 1992 1993 1994 Level 2 Repeatable Level 3 Defined 1989 1991 1990 1992 1993 1994 1994 1995 1995 1993 1994 1994 1992 1993 1993 1991 1992 1992 Level 3 Defined 1990 1991 1991 Raytheon Raytheon 1989 1990 1990 Level 2 Repeatable Level 4/5 Level 4/5 1989 1989 Acquisition Acquisition Level 3 Level 3 & Transition & Transition Managed/ Managed/ • Self - assessment Software Improvement Team Formed Level 2 Level 2 Defined Defined Optimizing Optimizing North Texas Software Engineering organization began the process improvement journey in 1989—14 years ago. We achieved increased process maturity through development, deployment and improvement of software development processes, while going through a significant transition in the organization. Successful Level 4 IPI assessment in June 2001. CMMI Leveol 5 PBA in February, 2003 Goal : achieving a successful CMMI Level 5 appraisal in September, 2003. Repeatable Repeatable • • Self Self - - assessment assessment • • Self Self - - assessment assessment • • Software Improvement Software Improvement • • Software Improvement Software Improvement Team Formed Team Formed Team Formed Team Formed IPI CMM - Based Internal Process Improvement Assessment IPI CMM - based Internal Process Improvement Assessment RTIS Raytheon/TI Systems RTIS Raytheon/TI Systems CMMI CMM Integrated CMMI CMM Integrated 9/21/2018 Speaker Name

Develop and Deploy Processes Presentation Title September 21, 2018 SW Steps to CMMI Level 5 2002 2003 Develop and Deploy Processes CMM 4 Fixes CMM-4 to CMMI-4 CMMI-5 What is the performance gap? 1. We addressed and corrected gaps and weaknesses which were identified during the 2001 Assessment, plus reestablished L4 performance.. 2. Moved from CMM L4 performance to CMMI L4 performance and Updated processes and tools as needed In order to achieve process improvement, the ES North Texas software engineering process group implemented Behavior Rollout Presentations to set clear expectations for process deployment on programs. Over 30 presentation were given on different topics during this time period. Each presentations set expectations for the listener to adopt new behaviors. We did not teach the model. We did identify gaps and used the Raytheon 6 sigma process to identify and develop process improvements. 3. Moved from CMMI L4 performance to CMMI L5 performance by completing deployment of updated processes and tools, and communication of Level 5 behaviors. 4. Completed Level 5 behaviors and prepare for SCAMPI Level-5 PBA ACHIEVE CMMI LEVEL PERFORMANCE then… GO for the Appraisal 6-12 Mo. Execution Level-5 SCAMPI 9/21/2018 Speaker Name

R6 Process – Life at Levels 4 & 5 Presentation Title September 21, 2018 R6 Process – Life at Levels 4 & 5 0 minutes Life at level 4 & 5 * We set goals and we monitor goals <click> * We use the measurements to stabilize our process. We establish a process capability baseline. (this is primarily level 4 and what we accomplished last year) * If the process capability baseline does not satisfy the goals, we identify, select, and prioritize opportunities for improvement * The next step is to pilot improvements to find out if the change is truly an improvement. * If the improvement is successful, it is then deployed. Might include training, tools, enablers, etc. * At that point, we re-measure, adjust goals, and establish a new capability baseline. The gap between your current capability and your goals should be closing Achieving CMMI Levels 4 and 5 are software’s implementation of R6s. Dan Burnham has specifically said this. 9/21/2018 Speaker Name

Identify Improvement Opportunities Presentation Title September 21, 2018 Identify Improvement Opportunities Ideas are the inputs to this process Ideas are gathered from many sources: Raytheon Six Sigma visualization and assessments Program level metrics analysis meetings Organization level software improvement metrics root cause analysis Make progress toward software improvement objectives Improvement ideas from individuals and management reviews Process appraisals Behavior surveys The improvement process starts with ideas—where do we get these ideas? Behavior surveys monitor how well the desired behaviors are adoped. The survey process provides feedback to identify, select, and prioritize opportunities for improvement Surveys are a feedback process based on the Behavior Rollouts 9/21/2018 Speaker Name

What Is a Behavior Survey? Presentation Title September 21, 2018 What Is a Behavior Survey? Behavior Verification Questions Summary Metrics Programs are chosen from the list of active programs tracked by the Software Leadership Team. True maintenance and test equipment are excluded. What is the process? PPQA performs surveys with Verification Questions. The Questions are defined as behaviors are rolled out. The questions are based on behavior roll out expectations for deployment. A standard reporting format is established and familiar Measurements are reported at the organization level to show trends. Results are reported for each question. Corrective actions are implemented (if needed). Good judgment is required to determine partial compliance in some areas. Judgment criteria are defined in the scoring matrix. Improve Deploy Consistent Behavior 9/21/2018 9 Speaker Name

Why Are Behavior Surveys Needed? Presentation Title September 21, 2018 Why Are Behavior Surveys Needed? Programs not executing to CMMI L4/L5 process Program execution compliant to CMMI L4/L5 process Lack Results in Adherence to CMMI process Lack Consistent Deployment Supports Enables Objective Feedback The Behavior survey encourages the behavior When PPQA asks the questions, it reinforces the behavior Participants are encouraged to deploy consistent process Behavior Surveys provide appropriate visibility with minimum investment Time investment is small for each program Resulting metrics will give management visibility into process “health” Lack Lack Organization Improvement Provides 10 9/21/2018 Speaker Name

This Is Not an Audit An Audit is more formal than a Behavior Survey Presentation Title September 21, 2018 This Is Not an Audit An Audit is more formal than a Behavior Survey Audits Based on approved plans and tailored requirements Identify non-compliances Attributable to programs Compliance Behavior Survey Based on communication packages Identify gaps Not attributable to programs Process deployment The behavior survey is more informal than a program audit. It asks high level questions to take the pulse of process deployment. Surveys take the pulse of process deployment 9/21/2018 Speaker Name

Developing Checklist Questions Presentation Title September 21, 2018 Developing Checklist Questions Survey Questions created by Software Quality Engineering (SQE) and stakeholders Results to be reported for each question Questions based on behavior communication packages of required processes and behaviors Shortens the effort to create questions. All questions are multiple choice. A typical question: “Has the program conducted a senior management review with the business area manager in the last quarter?” Results: T: Tailored (or variance) Y: Yes, complete and consistent behavior is evident P: Partial: most but not all behavior and activity is evident N: No behavior, or behavior is incomplete or inconsistent N/A: Behavior not tailored out, but it’s not yet time for the activity. blank: Behavior not assessed Questions are monitored over time to show improvement. 9/21/2018 Speaker Name

Survey of CMMI Level Behavior Presentation Title September 21, 2018 Survey of CMMI Level Behavior EXAMPLE ONLY PPQA regularly surveys the software programs across the organization about their behaviors. Reusing existing PPQA assets and methods reduces time needed to train PPQAs and get them started. Reporting format already established and familiar to the audience: the SEPG and the SW Leadership Team. HOW DO YOU READ THE CHART The objects is to re-establish consistent performance. And then maintain those behaviors. The point of this slide is to realize that our metrics show us that there is a need for corrective action. Anything with less than 80% compliance is targeted for improvement. We chose 80% because we recognize that not all programs will be at an appropriate point in their lifecycle to deploy a particular behavior, and they also may not be operating to the latest version of IPDS. 9/21/2018 Speaker Name

17 surveys conducted over a 3 year period Presentation Title September 21, 2018 Survey Timeline 2Q01 3Q01 4Q01 1Q02 CMM Level 4 IPI 2Q02 3Q02 4Q02 1Q03 2Q03 3Q03 4Q03 CMMI Level 5 PBA CMMI Level 5 SCAMPI Surveys conducted over a 3 year period, with a total of 17 surveys, with most occurring during the steps to develop and deploy processes after the Software CM Level 4 appraisal, and leading up to the CMMI Level 5 PBA. 1 in 2001, 13 in 2002, and 3 in 2003. 17 surveys conducted over a 3 year period 9/21/2018 Speaker Name

Behavior Survey 2001 CMM Level 4 Presentation Title September 21, 2018 Behavior Survey 2001 CMM Level 4 20 Programs Were Assessed, 14 questions Level 2 Projects have plans in place, , but they were not approved and released within the required timeframe. Planning continues to take longer than expected to complete. PPQA is putting more emphasis on auditing early planning 9/21/2018 Speaker Name

Behavior Survey April 2002 CMMI Level 4 Presentation Title September 21, 2018 Behavior Survey April 2002 CMMI Level 4 Peer Review Closure Timely Planning Quantitative Processes 46 programs, 18 questions Level 2 1 At this point, Timely planning is identified as an improvement opportunity. Level 4 2 Focus on consistency and content of Metrics analysis meetings, and objective evidence of causal analysis. Note: the metrics analysis meeting was made the focal point of Level 4/5 process deployment. 9/21/2018 Speaker Name

Behavior Survey May 2003 CMMI Level 5 Presentation Title September 21, 2018 Behavior Survey May 2003 CMMI Level 5 Artifact Levels of Control Individual submission of Org Imp Ideas Risk Training 23 programs, 18 questions Has the Software Risk Manager attended the Risk Management Training (RMPROC100/101) or do they have a waiver? Has anyone on the program submitted an improvement proposal to the organization? (This is an awareness question) Address PBA feedback: Does the program CM plan address levels of control? Is there an artifact list specifying which artifacts will be controlled and the final control level? 9/21/2018 Speaker Name

Levels of Control PBA Correction Presentation Title September 21, 2018 Levels of Control PBA Correction Work in Progress Artifact Release Created Created Engineering Engineering Development Development Formal Level Formal Level Level Level Level Level The Scope of Levels of Control Levels of control provide change and configuration control for work in progress Emphasis is on earlier control of artifacts and inclusion of more artifacts that are controlled at lower levels. Communicate Levels of Control concept to program team Update SCM plan and procedures to describe levels of control This improvement was a corrective action from the CMMI Level 5 PBA. Everyone understands that completed artifacts are controlled. Note: Updates have been submitted for: SCM Plan RevA.ppt SwCM Cost Estimating Rev A.doc Library_Control_Procedure_RevA.doc Master List of Software Work Products A small improvement with significant impact 9/21/2018 Speaker Name

Planning Improvement Proposal Presentation Title September 21, 2018 Planning Improvement Proposal Triggers Consistent Behavior Survey findings Programs have difficulty meeting the 90-day planning requirement Incremental SW Planning Define planning activities for each SW stage Review and Approval Stakeholder Involvement, and reduce cycle time Lean Planning Power Point slides Revised SWP350 Software Project Management training class A big improvement with significant impact 9/21/2018 Speaker Name

Defect Containment Improvement Proposal Presentation Title September 21, 2018 Defect Containment Improvement Proposal Defect Containment Monitor peer review performance Verify defect classification Consistent use of standard tools: Synergy 6.2b and Defect Logger 3.1 Survey on organization behavior Implement organizational improvement through Statistical Process Control for code reviews Developed and provided training to SWEC This improvement led to several other improvement projects which made significant changes to our peer review process, including defect classification, as well as to our metrics analysis process. Each improvement resulted in a Behavior roll out communication with expectations, a new process, enablers and training. A big improvement leading to multiple improvement projects. 9/21/2018 Speaker Name

Expectations from SPC Behavior Rollout Peer Reviewers Implement SPC on your design and code peer reviews Project Metrics Team Add SPC data to your Metrics Analysis Meetings and record minutes Execute Causal Analysis and Resolution as part of Metrics Analysis Meetings Project Lead Use new Senior Management Review template 9/21/2018

Behavior Survey September 2003 CMMI Level 5 Behavior Results Presentation Title September 21, 2018 Behavior Survey September 2003 CMMI Level 5 Behavior Results 50% Adoption of Planning Improvement 70% Adoption of SPC Improvement +22% increased in use of Org improvement DB +30% Improvement in Levels of Control 23 programs, 13 questions, with 5 questions from previous survey, and 8 new questions. Our culture adopts change quickly. Close to 50% of the programs surveyed have adopted the next generation process, which was deployed in the last 6 months. 70% of the programs have adopted the SPC process. There has been a 30% increase in deployment of levels of control, and a 20% increase in use of the master list of artifacts. In addition, there has been a 22% increase in submission of org improvement ideas by individuals. 9/21/2018 Speaker Name

Summary Behavior surveys have been successfully used: as a deployment tool to reinforce expectations, and to identify areas for process deployment The most significant improvements triggered by behavior surveys were identified as strengths in our SCAMPI: Levels of Control Incremental Planning Metrics Analysis Meetings as the focal point for quantitative process Started with basic behaviors which were refined with process maturity Evolved to address more sophisticated activities A tool for deployment, but with further analysis can also identify specific improvements Improvement in procedures Improvement in training Improvement in enablers 9/21/2018