1. Using Behavior Surveys for CMMI Process Deployment
Presentation Title
September 21, 2018
Using Behavior Surveys for CMMI Process Deployment
Introduce speakers…
This presentation is a practitioner’s view for deployment of process. It presents a practical approach to identifying process gaps and monitoring process deployment. The presentation does not discuss the CMMI model, but discusses a method for achieving Level 5.
Raytheon North Texas Software Engineering
Donna Freed
Speaker Name

2. Overview: Using Behavior Surveys for CMMI Process Deployment
Presentation Title
September 21, 2018
Overview: Using Behavior Surveys for CMMI Process Deployment
PPQA Support of CMMI Level 4 and 5
N TX Software Engineering Process Journey
SW Steps to CMMI Level 5
R6 Process for Achieving Level 5
What Is a Behavior Survey?
Survey Results
Improvements Made:
Levels of Control
Incremental Planning
Metrics Analysis focused on defect containment, and statistical process control
What is the PPQA role?
What is the history of N TX Software engineering?
What was the approach for achieving Level 5?
How did we monitor our progress and identify areas for improvement?
What was the result?
<insert pool story here>
9/21/2018
Speaker Name

3. PPQA Focuses on Improvement
Presentation Title
September 21, 2018
PPQA Focuses on Improvement
Improve and
Strengthen Process by
Closing Gaps
Step 3:
Identify and Resolve Gaps
Work with Process Owners
R6 Specialist Projects
Optimize Execution
Step 2:
How Well Process Is Supporting Programs
Determine Process Gaps
Identify Process Improvement
The CMMI model says the level 2 process area PPQA:
supports the delivery of high-quality products and services by providing the project staff and all levels of managers with appropriate visibility into, and feedback on, the processes and associated work products throughout the life cycle.
CMMISM for Systems Engineering/Software Engineering, Version 1.1
(CMMI-SE/SW, V1.1)
Staged Representation
To achieve CMMI Level 5, PPQA focuses on improvement,
to identify improvements and
help close gaps in process maturity, with utilization of 6 sigma.
This is the framework for today’s presentation on a creative approach to the PPQA process area. It is just about audits and noncompliances, but about continuous improvement. I hope I’m preaching to the choir.
Step 1:
How Well We Are Executing
Feedback to Program and Process
Causal Analysis
9/21/2018 3
Speaker Name

4. N TX Software Engineering Typical Program
Presentation Title
September 21, 2018
N TX Software Engineering Typical Program
Few Large Programs
Customer/End User: Military
Number of Eng’s: 10
Number of SW Engr’s: 5
Program Duration: 18 Months
PPQA Support: 3 Programs Per SQE
How do we characterize software engineering in North Texas…What is our typical program?
Over the last 3 years, North Texas had a staff of 23 software quality engineers on 45 active programs within software engineering.
Many Small Programs
9/21/2018
Speaker Name

5. N TX Software Engineering Process Improvement Journey
Presentation Title
September 21, 2018
N TX Software Engineering Process Improvement Journey
# minute
Level 5
CMMI
Level 5
Level 5
RTIS Integrated
Product
Development
Process
Level 4
IPI
Level 5
Level 5
RTIS Integrated
RTIS Integrated
RTIS Integrated
RTIS Integrated
Level 4
Level 4
CMMI
CMMI
Software Sigma
& Cycle Time
Software Sigma
Software Sigma
Level 4
Level 4
CMMI
CMMI
RTIS Policies &
Procedures
RTIS Software
Operating
Instructions
Product
Product
Software Sigma
Software Sigma
IPI
IPI
RTIS Policies &
RTIS Policies &
Product
Product
RTIS Policies &
RTIS Policies &
Development
Development
& Cycle Time
& Cycle Time
IPI
IPI
Procedures
Procedures
Development
Development
& Cycle Time
& Cycle Time
Procedures
Procedures
Process
Process
Process
Process
Level 3
Baseline
Validation
Level 3
Level 3
RTIS Software
RTIS Software
Level 4
Level 4
Level 4
RTIS Software
RTIS Software
10x
Fault Density
Improvement
Baseline
Baseline
Operating
Operating
10x
10x
Operating
Operating
Validation
Validation
Level 2
Repeatable
Level 3
Defined
1989
1991
1990
1992
1993
1994
Level 4/5
Managed/
Optimizing
Raytheon
Acquisition
& Transition
2000
1995
1996
1997
1999 -
2001
2002
2003
Baldridge
Award
Fault Density
Fault Density
Instructions
Instructions
Baldridge
Baldridge
Level 3
Level 4/5
Managed/
Optimizing
Raytheon
Acquisition
& Transition
2000
1995
1996
1997
1999 -
2001
2002
2003
2000
1995
1996
1997
1999 -
2001
2002
2003
2003
Instructions
Instructions
Level 3
Level 3
2002
2003
2003
Award
Award
Improvement
Improvement
2001
2002
2002
Level 2
2000
Raytheon
Acquisition
& Transition
Level 4/5
Managed/
Optimizing
2001
2001
Level 2
Level 2
1997
1999 -
2000
2000
1996 - -
1999
1999
1995
1996
1996
1997
1997
1989
1991
1990
1992
1993
1994
Level 2
Repeatable
Level 3
Defined
1989
1991
1990
1992
1993
1994
1994
1995
1995
1993
1994
1994
1992
1993
1993
1991
1992
1992
Level 3
Defined
1990
1991
1991
Raytheon
Raytheon
1989
1990
1990
Level 2
Repeatable
Level 4/5
Level 4/5
1989
1989
Acquisition
Acquisition
Level 3
Level 3
& Transition
& Transition
Managed/
Managed/ •
Self -
assessment
Software Improvement
Team Formed
Level 2
Level 2
Defined
Defined
Optimizing
Optimizing
North Texas Software Engineering organization began the process improvement journey in 1989—14 years ago.
We achieved increased process maturity through development, deployment and improvement of software development processes,
while going through a significant transition in the organization.
Successful Level 4 IPI assessment in June 2001.
CMMI Leveol 5 PBA in February, 2003
Goal : achieving a successful CMMI Level 5 appraisal in September, 2003.
Repeatable
Repeatable • •
Self
Self - -
assessment
assessment • •
Self
Self - -
assessment
assessment • •
Software Improvement
Software Improvement • •
Software Improvement
Software Improvement
Team Formed
Team Formed
Team Formed
Team Formed
IPI CMM -
Based Internal Process Improvement Assessment
IPI CMM -
based Internal Process Improvement Assessment
RTIS Raytheon/TI Systems
RTIS Raytheon/TI Systems
CMMI CMM Integrated
CMMI CMM Integrated
9/21/2018
Speaker Name

6. Develop and Deploy Processes
Presentation Title
September 21, 2018
SW Steps to CMMI Level 5
Develop and Deploy Processes
CMM 4 Fixes
CMM-4 to CMMI-4
CMMI-5
What is the performance gap?
1. We addressed and corrected gaps and weaknesses which were identified during the 2001 Assessment, plus reestablished L4 performance..
2. Moved from CMM L4 performance to CMMI L4 performance and Updated processes and tools as needed
In order to achieve process improvement, the ES North Texas software engineering process group implemented Behavior Rollout Presentations to set clear expectations for process deployment on programs. Over 30 presentation were given on different topics during this time period. Each presentations set expectations for the listener to adopt new behaviors. We did not teach the model. We did identify gaps and used the Raytheon 6 sigma process to identify and develop process improvements.
3. Moved from CMMI L4 performance to CMMI L5 performance by completing deployment of updated processes and tools, and communication of Level 5 behaviors.
4. Completed Level 5 behaviors and prepare for SCAMPI
Level-5
PBA
ACHIEVE CMMI LEVEL PERFORMANCE
then…
GO for the Appraisal
6-12 Mo. Execution
Level-5
SCAMPI
9/21/2018
Speaker Name

7. R6 Process – Life at Levels 4 & 5
Presentation Title
September 21, 2018
R6 Process – Life at Levels 4 & 5
0 minutes
Life at level 4 & 5
* We set goals and we monitor goals
<click>
* We use the measurements to stabilize our process. We establish a process capability baseline.
(this is primarily level 4 and what we accomplished last year)
* If the process capability baseline does not satisfy the goals, we identify, select, and prioritize opportunities for improvement
* The next step is to pilot improvements to find out if the change is truly an improvement.
* If the improvement is successful, it is then deployed. Might include training, tools, enablers, etc.
* At that point, we re-measure, adjust goals, and establish a new capability baseline. The gap between your current capability and your goals should be closing
Achieving CMMI Levels 4 and 5 are software’s implementation of R6s. Dan Burnham has specifically said this.
9/21/2018
Speaker Name

8. Identify Improvement Opportunities
Presentation Title
September 21, 2018
Identify Improvement Opportunities
Ideas are the inputs to this process
Ideas are gathered from many sources:
Raytheon Six Sigma visualization and assessments
Program level metrics analysis meetings
Organization level software improvement metrics root cause analysis
Make progress toward software improvement objectives
Improvement ideas from individuals and management reviews
Process appraisals
Behavior surveys
The improvement process starts with ideas—where do we get these ideas?
Behavior surveys monitor how well the desired behaviors are adoped. The survey process provides feedback to identify, select, and prioritize opportunities for improvement
Surveys are a feedback process based on the Behavior Rollouts
9/21/2018
Speaker Name

9. What Is a Behavior Survey?
Presentation Title
September 21, 2018
What Is a Behavior Survey?
Behavior Verification Questions
Summary
Metrics
Programs are chosen from the list of active programs tracked by the Software Leadership Team.
True maintenance and test equipment are excluded.
What is the process? PPQA performs surveys with Verification Questions. The Questions are defined as behaviors are rolled out. The questions are based on behavior roll out expectations for deployment. A standard reporting format is established and familiar
Measurements are reported at the organization level to show trends. Results are reported for each question. Corrective actions are implemented (if needed). Good judgment is required to determine partial compliance in some areas. Judgment criteria are defined in the scoring matrix.
Improve
Deploy
Consistent
Behavior
9/21/2018 9
Speaker Name

10. Why Are Behavior Surveys Needed?
Presentation Title
September 21, 2018
Why Are Behavior Surveys Needed?
Programs not executing
to CMMI L4/L5 process
Program execution compliant
to CMMI L4/L5 process
Lack
Results in
Adherence to
CMMI process
Lack
Consistent Deployment
Supports
Enables
Objective
Feedback
The Behavior survey encourages the behavior
When PPQA asks the questions, it reinforces the behavior
Participants are encouraged to deploy consistent process
Behavior Surveys provide appropriate visibility with minimum investment
Time investment is small for each program
Resulting metrics will give management visibility into process “health”
Lack
Lack
Organization Improvement
Provides 10
9/21/2018
Speaker Name

11. This Is Not an Audit An Audit is more formal than a Behavior Survey
Presentation Title
September 21, 2018
This Is Not an Audit
An Audit is more formal than a Behavior Survey
Audits
Based on approved plans and tailored requirements
Identify non-compliances
Attributable to programs
Compliance
Behavior Survey
Based on communication packages
Identify gaps
Not attributable to programs
Process deployment
The behavior survey is more informal than a program audit. It asks high level questions to take the pulse of process deployment.
Surveys take the pulse of process deployment
9/21/2018
Speaker Name

12. Developing Checklist Questions
Presentation Title
September 21, 2018
Developing Checklist Questions
Survey Questions created by Software Quality Engineering (SQE) and stakeholders
Results to be reported for each question
Questions based on behavior communication packages of required processes and behaviors
Shortens the effort to create questions. All questions are multiple choice. A typical question:
“Has the program conducted a senior management review with the business area manager in the last quarter?”
Results:
T: Tailored (or variance)
Y: Yes, complete and consistent behavior is evident
P: Partial: most but not all behavior and activity is evident
N: No behavior, or behavior is incomplete or inconsistent
N/A: Behavior not tailored out, but it’s not yet time for the activity.
blank: Behavior not assessed
Questions are monitored over time to show improvement.
9/21/2018
Speaker Name

13. Survey of CMMI Level Behavior
Presentation Title
September 21, 2018
Survey of CMMI Level Behavior
EXAMPLE
ONLY
PPQA regularly surveys the software programs across the organization about their behaviors. Reusing existing PPQA assets and methods reduces time needed to train PPQAs and get them started.
Reporting format already established and familiar to the audience: the SEPG and the SW Leadership Team.
HOW DO YOU READ THE CHART
The objects is to re-establish consistent performance. And then maintain those behaviors.
The point of this slide is to realize that our metrics show us that there is a need for corrective action. Anything with less than 80% compliance is targeted for improvement.
We chose 80% because we recognize that not all programs will be at an appropriate point in their lifecycle to deploy a particular behavior, and they also may not be operating to the latest version of IPDS.
9/21/2018
Speaker Name

14. 17 surveys conducted over a 3 year period
Presentation Title
September 21, 2018
Survey Timeline
2Q01
3Q01
4Q01
1Q02
CMM Level 4 IPI
2Q02
3Q02
4Q02
1Q03
2Q03
3Q03
4Q03
CMMI Level 5 PBA
CMMI Level 5 SCAMPI
Surveys conducted over a 3 year period, with a total of 17 surveys, with most occurring during the steps to develop and deploy processes after the Software CM Level 4 appraisal, and leading up to the CMMI Level 5 PBA.
1 in 2001, 13 in 2002, and 3 in 2003.
17 surveys conducted over a 3 year period
9/21/2018
Speaker Name

15. Behavior Survey 2001 CMM Level 4
Presentation Title
September 21, 2018
Behavior Survey CMM Level 4
20 Programs Were Assessed, 14 questions
Level 2
Projects have plans in place, , but they were not approved and released within the required timeframe. Planning continues to take longer than expected to complete. PPQA is putting more emphasis on auditing early planning
9/21/2018
Speaker Name

16. Behavior Survey April 2002 CMMI Level 4
Presentation Title
September 21, 2018
Behavior Survey April CMMI Level 4
Peer Review
Closure
Timely
Planning
Quantitative
Processes
46 programs, 18 questions
Level 2
1 At this point, Timely planning is identified as an improvement opportunity.
Level 4
2 Focus on consistency and content of Metrics analysis meetings, and objective evidence of causal analysis. Note: the metrics analysis meeting was made the focal point of Level 4/5 process deployment.
9/21/2018
Speaker Name

17. Behavior Survey May 2003 CMMI Level 5
Presentation Title
September 21, 2018
Behavior Survey May CMMI Level 5
Artifact Levels
of Control
Individual submission of
Org Imp Ideas
Risk
Training
23 programs, 18 questions
Has the Software Risk Manager attended the Risk Management Training (RMPROC100/101) or do they have a waiver?
Has anyone on the program submitted an improvement proposal to the organization? (This is an awareness question)
Address PBA feedback:
Does the program CM plan address levels of control?
Is there an artifact list specifying which artifacts will be controlled and the final control level?
9/21/2018
Speaker Name

18. Levels of Control PBA Correction
Presentation Title
September 21, 2018
Levels of Control PBA Correction
Work in Progress
Artifact Release
Created
Created
Engineering
Engineering
Development
Development
Formal Level
Formal Level
Level
Level
Level
Level
The Scope of Levels of Control
Levels of control provide change and configuration control for work in progress
Emphasis is on earlier control of artifacts and inclusion of more artifacts that are controlled at lower levels.
Communicate Levels of Control concept to program team
Update SCM plan and procedures to describe levels of control
This improvement was a corrective action from the CMMI Level 5 PBA.
Everyone understands that completed artifacts are controlled.
Note: Updates have been submitted for:
SCM Plan RevA.ppt
SwCM Cost Estimating Rev A.doc
Library_Control_Procedure_RevA.doc
Master List of Software Work Products
A small improvement with significant impact
9/21/2018
Speaker Name

19. Planning Improvement Proposal
Presentation Title
September 21, 2018
Planning Improvement Proposal
Triggers
Consistent Behavior Survey findings
Programs have difficulty meeting the 90-day planning requirement
Incremental SW Planning
Define planning activities for each SW stage
Review and Approval
Stakeholder Involvement, and reduce cycle time
Lean Planning
Power Point slides
Revised SWP350 Software Project
Management training class
A big improvement with significant impact
9/21/2018
Speaker Name

20. Defect Containment Improvement Proposal
Presentation Title
September 21, 2018
Defect Containment Improvement Proposal
Defect Containment
Monitor peer review performance
Verify defect classification
Consistent use of standard tools: Synergy 6.2b and Defect Logger 3.1
Survey on organization behavior
Implement organizational improvement through
Statistical Process Control for code reviews
Developed and provided training to SWEC
This improvement led to several other improvement projects which made significant changes to our peer review process, including defect classification, as well as to our metrics analysis process. Each improvement resulted in a Behavior roll out communication with expectations, a new process, enablers and training.
A big improvement leading to multiple improvement projects.
9/21/2018
Speaker Name

21. Expectations from SPC Behavior Rollout
Peer Reviewers
Implement SPC on your design and code peer reviews
Project Metrics Team
Add SPC data to your Metrics Analysis Meetings and record minutes
Execute Causal Analysis and Resolution as part of Metrics Analysis Meetings
Project Lead
Use new Senior Management Review template
9/21/2018

22. Behavior Survey September 2003 CMMI Level 5 Behavior Results
Presentation Title
September 21, 2018
Behavior Survey September CMMI Level 5 Behavior Results
50% Adoption of Planning
Improvement
70% Adoption of SPC
Improvement
+22% increased in use
of Org improvement DB
+30% Improvement in
Levels of Control
23 programs, 13 questions, with 5 questions from previous survey, and 8 new questions.
Our culture adopts change quickly.
Close to 50% of the programs surveyed have adopted the next generation process, which was deployed in the last 6 months.
70% of the programs have adopted the SPC process.
There has been a 30% increase in deployment of levels of control, and a 20% increase in use of the master list of artifacts.
In addition, there has been a 22% increase in submission of org improvement ideas by individuals.
9/21/2018
Speaker Name

23. Summary Behavior surveys have been successfully used:
as a deployment tool to reinforce expectations, and
to identify areas for process deployment
The most significant improvements triggered by behavior surveys were identified as strengths in our SCAMPI:
Levels of Control
Incremental Planning
Metrics Analysis Meetings as the focal point for quantitative process
Started with basic behaviors which were refined with process maturity
Evolved to address more sophisticated activities
A tool for deployment, but with further analysis can also identify specific improvements
Improvement in procedures
Improvement in training
Improvement in enablers
9/21/2018