1. 3rd Annual CMMI Technology Conference and User Group
Kent Schneider, President
Defense Enterprise Solutions
Northrop Grumman Information Technology
November 18, 2003
Speaker Notes
Audience: approx 300 attendees including; DOD OSD Sponsors for SEI, DOD acquisition personnel, SEI management and senior staff (CMMI Steward of Product Suite; NDIA Sys Eng Div (industry CMMI sponsor; SPC senior staff; most DoD corporate (large and small) IT process and quality managers and staff; selected commercial IT and SE/SW managers and staff, Process Appraisal and Improvement Vendors and community
Facility: Large banquet room; elevated podium; attendees at table after lunch; projector and computer available for large screen
Time: 40 minutes including introduction and a few questions
Themes
1.) My business unit has:
- achieved some practical business value from quantitative management (L4) and optimizing (L5) processes, methods, techniques and tools (I. e. High maturity)
Obtained customers satisfaction AOP goal
Obtained cost and schedule performance goal e. g. your Ops review includes quantitative/statistical control; your QMB calls for innovation and deploys optimizing defect prevention techniques (L5)
- Effectively applied the integrated maturity model (ie CMMI) for systems and software engineering (ie JEDMICS does both HW and SW; AIT does both Hardware and Software)
2.) Demonstrate and communicate “what management commitment look like”
Key Points
-Diverse organization initially comprised of projects using different but similar standard organizational standard processes.
-Process improvement tied to customer satisfaction and financial performance especially cost and schedule performance
-“Walk the talk”
-Commitment
- CMMI facilitated the integration of system engineering with software engineering (and beyond)

2. Organizational Perspective
Northrop Grumman
$26 B in revenue; 120,000 employees; 50 states; 25 countries
Information Technology (IT) Sector
$4.7 B in sales; 23,000 employees; 48 states; 15 countries
Defense Enterprise Solutions (DES) Business Unit
$610 M in sales; 3,045 employees, 23 states, 4 countries
DES provides enterprise-wide technology solutions to the Defense marketplace
Major Application Areas:
Logistics
Mission Support
Science & Technology
Simulation, Analysis, and Training
Communications & Infrastructure
Kent will find a slide ro replace this one (seeking “quad” chart)
Quickly summary the Organization – Use familiar slide but don’t dwell on it as the audience doesn’t like sales pitches
Northrop Grumman
Seven operating sectors
Electronic Systems
Information Technology
Mission Systems
Integrated Systems
Newport News (ship building)
Ship Systems
Space Technology
Seven business units
Commercial Information Services
Internal Information Services
Computing Systems
Defense Mission Systems
Defense Enterprise Solutions
Government Solutions
TASC
Five organizational units
Simulation, Analysis & Training Systems
Logistics Systems
Mission Support Systems
Science & Technology
Communications & Infrastructure Systems
Organizational units further broken down into divisions
10/19/03
Copyright Northrop Grumman Company 2003

3. Core Processes Common to Multiple Disciplines
CMMI
Core
Information Technology
Products & Services in
Constant Change
IT Consulting
Sys Arch, Engin & Delivery
Enterprise Integration
IT Infrastructure Management
Applications Management
Sys Eng & Tech Assistance (SETA)
Functional Process Outsourcing SW
SW CMM
Core
SE Discipline SE
SE CM
SW Discipline
Core
Processes
IPPD-CMM
IPD
This chart exhibits that “core” processes, especially those for management and support, are common to both system engineering and software engineering.
We were early adopters of the CMMI for the reasons exhibited on this chart.
We viewed system and software engineering as integrated (“we were CMMI before CMMI became popular”)
We could not practically improve against multiple maturity models
Contributed substantially to the CMMI Development (including People:Wilson, Hollenbach, Pflugrad, etc and Tools: ie CMMI Poster)
Core
IPD Discipline
Other
CMM’s
10/19/03
Copyright Northrop Grumman Company 2003

4. Organizational Process Maturity Pedigree
2001
2002
1999
2000
Logicon LISS L3
ENABLER
Logicon LAT L3
(to other units)
Logicon LTS
Northrop Grumman
Information Technology
Defense Enterprise Solutions
Logicon LIS
LIEB
CMMI L5 L5
Litton PRC
Simply summarize that DES was initially comprised of projects and programs, that while they had achieved process maturity, they each had legacy organizational processes. Particularly noteworthy was the SW CMM L5 obtainment (ie PRC).
While this presented a challenge, we were able to layer quantitative management (Level 4) and optimizing methods (level 5) on top of each inherited program.
Notes:
To become CMMI Level 5 in 2002 while…
Forming DES from a merger of 3 major legacy organizations
Contractually reconstituting all projects
Using a newly developed accounting system
Setting up a new management team
Pursuing very aggressive financial targets
DES implemented the following actions:
PA Process Owners (FT EPI staff, organizational reps & subject matter experts)
Resolved change requests from a CMMI Gap Analysis
Worked with the NGIT sector to build “umbrella” processes to which legacy OSSPs comply
Process Improvement Support and QA reps were assigned to each DES operating unit to upgrade L2-L3 processes
Project reps implemented L4-L5 processes by participating on:
Level 4 Working Group (QPM)
Level 5 Working Group (OID – Process)
Metrics Office (OPP)
Defect Prevention Working Group (CAR)
Technology Change Management SIG (OID – Technology)
(to other units) L3
Litton TASC
SPII
(to other units)
10/19/03
Copyright Northrop Grumman Company 2003

5. Business performance linked to Process and Quality Performance
2003 Goals for Performance
Business performance linked to Process and Quality Performance
This exhibit exhibits key DES business goals, as extracted from the DES 2002 Annual Operating Plan (AOP), and the corresponding quality/process performance objectives. We think that achievement of these process/quality objectives will positively impact financial performance and ensure customer satisfaction (and associated “past performance” evaluation scores). This is the link between business objectives and process/quality performance
Note that our business goals and process performance objectives are measurable. We have a quantitative indicator of customer satisfaction via customer-provided evaluations of our work such as through surveys, award fee evaluation or other contract performance measurements. Since “past performance” is the most significant evaluation factor used by our federal customers, it is a very appropriate business objective to match to process and quality performance.
Tracking Cost Performance, Schedule Performance and Defect Density on a monthly basis provides timely indicators of process performance for corrective action (special cause of variance) or improvement (reduce variation).
10/19/03
Copyright Northrop Grumman Company 2003

6. Organizational and Project Quantitative Management Process Overview
DES Business Objectives
DES management checks org and project data against DES goals (process capability baseline). 1
DES management selects quality and process goals & measurements 6
Organization
Project
Projects select related goals & measurements for each life cycle phase. 2 5
Projects check performance against project goals and business objectives.
This charts illustrates our general roadmap for quantitative management of process and quality performance.
It begins with our business objectives (see previous slide).
The organizational management team selected the process goals and measurements (also on previous page)
Projects then select related by project specific goals, (3) track their process performance (I will show you an example), (4) take improvement action as indicated by the data, (5) Check if their performance is meeting goals and business measures and
(6) The organizational management monitors the data against the process performance goals and business measures
Projects track process performance over time. 3
Projects improve performance by removing root causes for out-of-bound conditions. 4
11/18/2003
Copyright Northrop Grumman Company 2003

7. Quantitative Management
Measurable contractual commitments:
Delivering products and services on time and under budget
Delivering quality (low-defect) products
Measurable process and quality commitments; DES developed process performance and product quality goals to tract program performance:
Cost
Schedule
Quality
Demonstrable senior management commitment
DES tracks this through its senior management operations reviews (a sample follows)
This slide introduces the example of Ops Review of Project quantitative data and exhibits management commitment to contract performance and to process and quality performance
10/19/03
Copyright Northrop Grumman Company 2003

8. S&T/ITS Program and QM Indicators
Critical/Urgent Defect Density/KLOC (Peer Review)
5.3
4.9
Goal:
Critical/Urgent Defect Total Defects
 5/KLOC  10/KLOC
Action:
DP Cycle #1 countermeasures are effective. Reduced targeted “checking” defects from 2.6/KLOC to 1.2/KLOC.
DP Cycle #2 focused on defects escaping the PSP Cycle. Follow up from countermeasures show improvement in schedule and quality.
DP Cycle #3 focused on units with highest test defects. Countermeasure approved to peer review code again prior to customer dry run testing.
SCoVs – 6 complete, 2 in progress, 5 approved for Builds 4 & 5.
DDt – SPC for Defect Density by test (DDt) is not useful, investigating alternate metrics, began Rayleigh Curve.
Technical Highlights: Construction Complete
Inspection & Test - 100% of all code was peer reviewed. 100% Unit Test and Internal Integration Test complete.
Total DDt in Unit Test /KLOC (Goal  2)
Total DDt Internal Integration Test - .44/KLOC (Goal  1)
3.2
1.4
1.3
Total Defect Density/KLOC (Peer Review)
An example of quantitative management
This chart and the following illustrate monitoring process and quality data in a manner that ensures both corrective action in a timely manner as well as areas of improvement
Not intended to see the detail but note that this program set goals for, and tracked, Coding Defects (this slide), Test Defects, Documentation Defects, and Schedule and Cost Performance (Subsequent slides) achieving customer satisfaction
24.1
21.6
18.3
19.3
13.2
10/19/03
Copyright Northrop Grumman Company 2003

9. S&T/ITS Program and QM Indicators
ITS Test Defects By Test Cycle
Defect Density by KLOC
Goal:
Critical/Urgent Defect Total Defects
DDt Unit  .5/KLOC  2/KLOC
DDt  .25/KLOC  1/KLOC
Action:
None for Defect Density goals.
Technical Highlights:
UNDER Cycle Goals!
DP Cycle
Remeasure
Build
Critical + Urgent
Total
Unit Test
.33
1.4
0.1 – 0.6
0.056
0.18
0.7
0.065
0.46
0.75
0.233
0.73
0.8
0.132
0.38
0.85
0.066
0.9
DP Cycle
Completed Dry Run Test (.85) with enough time to review enhancements. Decrease in showstopper defects.
DP Cycle remeasure led to new ideas for team and enhancements analysis
10/19/03
Copyright Northrop Grumman Company 2003

10. S&T/ITS Program and QM Indicators
ITS Phase Based Estimates after Dry Run Test (8/31/03)
Goal:
DDs – Defect Discovery
Critical/Urgent Total Defects
 .25/KLOC  1/KLOC
Action:
None for Defect Discovery goals.
Technical Highlights:
Total Test Defect Density - Sept 2003 is 5.2/KLOC! There is no significant change to defect density this month. This is relative to 19/KLOC in Peer Review!
DDs
Dry Run Test Latent = 0.044/KLOC
Discovery Predictions next = 0.405/KLOC
Discovery Predictions next = 0.132/KLOC
Projected Latent
Error Discovery Efficiency
Next Phases Estimates
4.725
99.85%
43, 14
10/19/03
Copyright Northrop Grumman Company 2003

11. S&T/ITS Program and QM Indicators
Critical/Urgent Defect Density – Misc. Documentation
Goal: Total Defects  1/Page
Actual:
Total STF Defects = .688/Page
Total User Manual Defects = .709/page
Total Misc. Doc Defects = .701/page
(Critical & Urgent = .165/page)
Action: No new SCoV
Technical Highlights:
Reviewed 100% documentation
100% documentation DDr exceeds goal (Misc Documentation Mean Critical + Urgent continues to decrease)
Total Defect Density – Misc. Documentation
10/19/03
Copyright Northrop Grumman Company 2003

12. S&T/ITS Program and QM Indicators
CLIN 1 and CLIN 2 – CPIm (Monthly – Sep 03)
Goal: 1  0.1 ( )
Actual: Current CPIm Mean = .730
Action: Continue ECP plans and attempt cost impact recovery from customer as part of the DMSI schedule alignment ECP.
Technical Highlights: As expected the CPIm remains out of goal. Low number of defects allowed a positive cost variance to increase the mean by 5.7%.
CLIN 1 and CLIN 2 – SPIm (Monthly – Sep 03)
Goal: 1  0.1 ( )
Actual: Current SPIm Mean = .753
Action: Continue with ECPs and Test Activities
Technical Highlights: Released CBT. Preparing Software Test Report.
10/19/03
Copyright Northrop Grumman Company 2003

13. S&T/ITS Program and QM Indicators
As stated in the ITS Quantitative Management Plan
Category 1: Customer Satisfaction
Goal
Deliver no known open Level 1-3 severity defects
Achieve contractual expectations as defined in the ITS SRS and implied customer expectations as defined in the ITS SQASP
Actual
100% known peer review and test defects have been repaired.
Customer completed performance survey. ITS was rated “Exceptional” in all categories. Looking to receive CPARs in CLIN 2. None received to date.
Action None
Technical Highlights
Customer continues to fund contract mods to keep team momentum and buy conversion data schedule time. Customer continues to fund T&M CLIN. Requested synchronized ITS schedule to DMSI schedule.
·         .
10/19/03
Copyright Northrop Grumman Company 2003

14. Organizational Innovation
To provide higher quality and better services to our customers, DES:
Set aggressive goals based on past quantitative performance
Research innovations internally and externally to measurably improve our performance.
Report out monthly on performance…
An example follows…
L5 Maturity comes from organizational innovation (including TCM)
The innovation needs to be data driven
The following case study provided the data we needed to pilot and deploy PSP in applicable environments.
Project results impact organizational results
10/19/03
Copyright Northrop Grumman Company 2003

15. Two projects with: Project Comparisons
Same customer (different divisions)
Similar applications
Both Client-Server (PowerBuilder / Oracle)
Similar level of team expertise and training
Estimated to be similar size and effort (~$2 Million)
10/19/03
Copyright Northrop Grumman Company 2003

16. Key Process Insertion Differences
Project A:
Disciplined team design process used to create sound developer work packets
Personal Software Process used consistently by developers
SEI developed course for software developers which provides process at an individual level for producing software components and documentation (user & technical)
Peer Reviews conducted on the most critical 20% of the software
Project B:
We decided that this project was “too far along” to benefit from process insertion
10/19/03
Copyright Northrop Grumman Company 2003

17. Project Test Time / Cost
Project A
Project B
Integration / Acceptance Test Cost
$78.K
$612K
Normalized (per KLOC)
$0.95K
$4.05K
Time to Accept (months)
3.7
14.6
Normalized (months per 100 KLOC)
4.5
9.7
10/19/03
Copyright Northrop Grumman Company 2003

18. Project Quality Project A Project B Duration (months) 31.8 43.0
Size (KLOC) 82
151
Developer Defect Density
9.4
17.3
Peer Review Exit Density
4.78
Delivered Defect Density
1.55
5.27
10/19/03
Copyright Northrop Grumman Company 2003

19. Positive Effect of Process Insertion
Mature Processes & PSP led to
Project A’s Integration and Acceptance Test Costs were 24% of Project B
Project A’s Test Duration was 46% of Project B
Project A’s delivered defect density was 29% of Project B
Process saved cost, reduced schedule, and improved quality
10/19/03
Copyright Northrop Grumman Company 2003

20. 2002 SCAMPI Appraisal Results
F1=Fully Implemented
Out of Scope
Process Maturity
CMMI SE/SW Level 5 for Staged and Continuous Representations
Software CMM Level 5 also
Organizational Process Strengths
Integrated SE and SW Eng
Sharing and reusing process assets
Data driven corrective action
Technology Change Management
Global Best Practices
Requirements Elicitation
Quantitative Data Management
Cost, Schedule and Quality Performance
JEDMICS exceeds software quality goals (.0156 vs .06 defects/KSLOC)
PERMS exceeds revenue goals by 11%
AIT has 100% on-time delivery
Only 2% of total defects found in delivered SIGS product
ITS is on cost and schedule (CPIm = .965 & SPIm = .985, target = 1.0 +/- .1)
SIGS averages a customer satisfaction of 98.5% over last four years
I haven’t been talking about ratings, but through deployment of the quantitative practices with data driven innovation, we This achieved a CMMI SE/SW Level 5 in December of (led by an external authorized lead evaluator). We also concurrently received a SW CMM Level 5 and a continuous rating of our capability
The upper left box exhibits some strengths determined at the evaluation.
The lower left depicts some project performance results.
10/19/03
Copyright Northrop Grumman Company 2003

21. Knowledge Management and Process Management
We manage our Processes as a Community of Practice
We help our integrated teams share and put process into action
Processes underpin each “community of practice”
Next Steps for DES
Knowledge management is an enabler for “Communities of Practice,” groups of people with a common mission or discipline in behalf of the company. It allows people to access and share information effectively in support of the processes the community of practice uses. It facilitates collaboration, particularly when the community of practice is distributed geographically. As we go through this presentation, we will get to a more practical view of knowledge management and its importance to Northrop Grumman.
We will further support process deployment as a KM community of practice
10/19/03
Copyright Northrop Grumman Company 2003

22. CMMI Extensibility and Flexibility
Information assurance
Help desk
Integrated Enterprise Process
IT Managed services
Network management
Program Management Processes
Engineering Management Processes
Engineering Processes
Our integrated system and software engineering processes are depicted in the center of this exhibit exhibiting our management, engineering, support and process processes.
Around the edges are other important IT and service activities that we perform for our customers (that are not easily “covered” by the CMMI at this time.
We are evaluating how the CMMI can best be applied to these activities and will soon have a technical report on the match with recommended ways that the CMMI can be continuously improved.
We need the CMMI sponsor and custodian to ensure CMMI flexibility and extensibility in line with our dynamic IT change
Process Infrastructure Processes
Asset management
Desktop/Server management
11/18/2003
Copyright Northrop Grumman Company 2003

23. Conclusions
CMMI Practices have a direct and positive influence on cost, schedule, and quality performance
It helps to tie the quantitative goals and measures to customer needs and relevant project problems
CMMI high maturity enables organizational and project agility in high priority areas
Projects can improve dramatically in 6 months or less using defect prevention cycles of 1 week or less
The transition from SW-CMM to CMMI is not inherently difficult, costly, and time-consuming, but…
It pays to proactively improve against emerging models and standards
It is much easier to attain the same maturity level the second time
It is much easier to transition between models when you focus on performance
Summarize key points
Also
Later this week, Pflugrad will be presenting our strategic process improvement plan and Craig Hollenbach will be presenting our quantitatively measured process improvements
10/19/03
Copyright Northrop Grumman Company 2003