3rd Annual CMMI Technology Conference and User Group Kent Schneider, President Defense Enterprise Solutions Northrop Grumman Information Technology November 18, 2003 Speaker Notes Audience: approx 300 attendees including; DOD OSD Sponsors for SEI, DOD acquisition personnel, SEI management and senior staff (CMMI Steward of Product Suite; NDIA Sys Eng Div (industry CMMI sponsor; SPC senior staff; most DoD corporate (large and small) IT process and quality managers and staff; selected commercial IT and SE/SW managers and staff, Process Appraisal and Improvement Vendors and community Facility: Large banquet room; elevated podium; attendees at table after lunch; projector and computer available for large screen Time: 40 minutes including introduction and a few questions Themes 1.) My business unit has: - achieved some practical business value from quantitative management (L4) and optimizing (L5) processes, methods, techniques and tools (I. e. High maturity) Obtained customers satisfaction AOP goal Obtained cost and schedule performance goal e. g. your Ops review includes quantitative/statistical control; your QMB calls for innovation and deploys optimizing defect prevention techniques (L5) - Effectively applied the integrated maturity model (ie CMMI) for systems and software engineering (ie JEDMICS does both HW and SW; AIT does both Hardware and Software) 2.) Demonstrate and communicate “what management commitment look like” Key Points -Diverse organization initially comprised of projects using different but similar standard organizational standard processes. -Process improvement tied to customer satisfaction and financial performance especially cost and schedule performance -“Walk the talk” -Commitment - CMMI facilitated the integration of system engineering with software engineering (and beyond)

Organizational Perspective Northrop Grumman $26 B in revenue; 120,000 employees; 50 states; 25 countries Information Technology (IT) Sector $4.7 B in sales; 23,000 employees; 48 states; 15 countries Defense Enterprise Solutions (DES) Business Unit $610 M in sales; 3,045 employees, 23 states, 4 countries DES provides enterprise-wide technology solutions to the Defense marketplace Major Application Areas: Logistics Mission Support Science & Technology Simulation, Analysis, and Training Communications & Infrastructure Kent will find a slide ro replace this one (seeking “quad” chart) Quickly summary the Organization – Use familiar slide but don’t dwell on it as the audience doesn’t like sales pitches Northrop Grumman Seven operating sectors Electronic Systems Information Technology Mission Systems Integrated Systems Newport News (ship building) Ship Systems Space Technology Seven business units Commercial Information Services Internal Information Services Computing Systems Defense Mission Systems Defense Enterprise Solutions Government Solutions TASC Five organizational units Simulation, Analysis & Training Systems Logistics Systems Mission Support Systems Science & Technology Communications & Infrastructure Systems Organizational units further broken down into divisions 10/19/03 Copyright Northrop Grumman Company 2003

Core Processes Common to Multiple Disciplines CMMI Core Information Technology Products & Services in Constant Change IT Consulting Sys Arch, Engin & Delivery Enterprise Integration IT Infrastructure Management Applications Management Sys Eng & Tech Assistance (SETA) Functional Process Outsourcing SW SW CMM Core SE Discipline SE SE CM SW Discipline Core Processes IPPD-CMM IPD This chart exhibits that “core” processes, especially those for management and support, are common to both system engineering and software engineering. We were early adopters of the CMMI for the reasons exhibited on this chart. We viewed system and software engineering as integrated (“we were CMMI before CMMI became popular”) We could not practically improve against multiple maturity models Contributed substantially to the CMMI Development (including People:Wilson, Hollenbach, Pflugrad, etc and Tools: ie CMMI Poster) Core IPD Discipline Other CMM’s 10/19/03 Copyright Northrop Grumman Company 2003

Organizational Process Maturity Pedigree 2001 2002 1999 2000 Logicon LISS L3 ENABLER Logicon LAT L3 (to other units) Logicon LTS Northrop Grumman Information Technology Defense Enterprise Solutions Logicon LIS LIEB CMMI L5 L5 Litton PRC Simply summarize that DES was initially comprised of projects and programs, that while they had achieved process maturity, they each had legacy organizational processes. Particularly noteworthy was the SW CMM L5 obtainment (ie PRC). While this presented a challenge, we were able to layer quantitative management (Level 4) and optimizing methods (level 5) on top of each inherited program. Notes: To become CMMI Level 5 in 2002 while… Forming DES from a merger of 3 major legacy organizations Contractually reconstituting all projects Using a newly developed accounting system Setting up a new management team Pursuing very aggressive financial targets DES implemented the following actions: PA Process Owners (FT EPI staff, organizational reps & subject matter experts) Resolved change requests from a CMMI Gap Analysis Worked with the NGIT sector to build “umbrella” processes to which legacy OSSPs comply Process Improvement Support and QA reps were assigned to each DES operating unit to upgrade L2-L3 processes Project reps implemented L4-L5 processes by participating on: Level 4 Working Group (QPM) Level 5 Working Group (OID – Process) Metrics Office (OPP) Defect Prevention Working Group (CAR) Technology Change Management SIG (OID – Technology) (to other units) L3 Litton TASC SPII (to other units) 10/19/03 Copyright Northrop Grumman Company 2003

Business performance linked to Process and Quality Performance 2003 Goals for Performance Business performance linked to Process and Quality Performance This exhibit exhibits key DES business goals, as extracted from the DES 2002 Annual Operating Plan (AOP), and the corresponding quality/process performance objectives. We think that achievement of these process/quality objectives will positively impact financial performance and ensure customer satisfaction (and associated “past performance” evaluation scores). This is the link between business objectives and process/quality performance Note that our business goals and process performance objectives are measurable. We have a quantitative indicator of customer satisfaction via customer-provided evaluations of our work such as through surveys, award fee evaluation or other contract performance measurements. Since “past performance” is the most significant evaluation factor used by our federal customers, it is a very appropriate business objective to match to process and quality performance. Tracking Cost Performance, Schedule Performance and Defect Density on a monthly basis provides timely indicators of process performance for corrective action (special cause of variance) or improvement (reduce variation). 10/19/03 Copyright Northrop Grumman Company 2003

Organizational and Project Quantitative Management Process Overview DES Business Objectives DES management checks org and project data against DES goals (process capability baseline). 1 DES management selects quality and process goals & measurements 6 Organization Project Projects select related goals & measurements for each life cycle phase. 2 5 Projects check performance against project goals and business objectives. This charts illustrates our general roadmap for quantitative management of process and quality performance. It begins with our business objectives (see previous slide). The organizational management team selected the process goals and measurements (also on previous page) Projects then select related by project specific goals, (3) track their process performance (I will show you an example), (4) take improvement action as indicated by the data, (5) Check if their performance is meeting goals and business measures and (6) The organizational management monitors the data against the process performance goals and business measures Projects track process performance over time. 3 Projects improve performance by removing root causes for out-of-bound conditions. 4 11/18/2003 Copyright Northrop Grumman Company 2003

Quantitative Management Measurable contractual commitments: Delivering products and services on time and under budget Delivering quality (low-defect) products Measurable process and quality commitments; DES developed process performance and product quality goals to tract program performance: Cost Schedule Quality Demonstrable senior management commitment DES tracks this through its senior management operations reviews (a sample follows) This slide introduces the example of Ops Review of Project quantitative data and exhibits management commitment to contract performance and to process and quality performance 10/19/03 Copyright Northrop Grumman Company 2003

S&T/ITS Program and QM Indicators Critical/Urgent Defect Density/KLOC (Peer Review) 5.3 4.9 Goal: Critical/Urgent Defect Total Defects  5/KLOC  10/KLOC Action: DP Cycle #1 countermeasures are effective. Reduced targeted “checking” defects from 2.6/KLOC to 1.2/KLOC. DP Cycle #2 focused on defects escaping the PSP Cycle. Follow up from countermeasures show improvement in schedule and quality. DP Cycle #3 focused on units with highest test defects. Countermeasure approved to peer review code again prior to customer dry run testing. SCoVs – 6 complete, 2 in progress, 5 approved for Builds 4 & 5. DDt – SPC for Defect Density by test (DDt) is not useful, investigating alternate metrics, began Rayleigh Curve. Technical Highlights: Construction Complete Inspection & Test - 100% of all code was peer reviewed. 100% Unit Test and Internal Integration Test complete. Total DDt in Unit Test - 1.39/KLOC (Goal  2) Total DDt Internal Integration Test - .44/KLOC (Goal  1) 3.2 1.4 1.3 Total Defect Density/KLOC (Peer Review) An example of quantitative management This chart and the following illustrate monitoring process and quality data in a manner that ensures both corrective action in a timely manner as well as areas of improvement Not intended to see the detail but note that this program set goals for, and tracked, Coding Defects (this slide), Test Defects, Documentation Defects, and Schedule and Cost Performance (Subsequent slides) achieving customer satisfaction 24.1 21.6 18.3 19.3 13.2 10/19/03 Copyright Northrop Grumman Company 2003

S&T/ITS Program and QM Indicators ITS Test Defects By Test Cycle Defect Density by KLOC Goal: Critical/Urgent Defect Total Defects DDt Unit  .5/KLOC  2/KLOC DDt  .25/KLOC  1/KLOC Action: None for Defect Density goals. Technical Highlights: UNDER Cycle Goals! DP Cycle Remeasure Build Critical + Urgent Total Unit Test .33 1.4 0.1 – 0.6 0.056 0.18 0.7 0.065 0.46 0.75 0.233 0.73 0.8 0.132 0.38 0.85 0.066 0.9 DP Cycle Completed Dry Run Test (.85) with enough time to review enhancements. Decrease in showstopper defects. DP Cycle remeasure led to new ideas for team and enhancements analysis 10/19/03 Copyright Northrop Grumman Company 2003

S&T/ITS Program and QM Indicators ITS Phase Based Estimates after Dry Run Test (8/31/03) Goal: DDs – Defect Discovery Critical/Urgent Total Defects  .25/KLOC  1/KLOC Action: None for Defect Discovery goals. Technical Highlights: Total Test Defect Density - Sept 2003 is 5.2/KLOC! There is no significant change to defect density this month. This is relative to 19/KLOC in Peer Review! DDs Dry Run Test Latent = 0.044/KLOC Discovery Predictions next = 0.405/KLOC Discovery Predictions next = 0.132/KLOC Projected Latent Error Discovery Efficiency Next Phases Estimates 4.725 99.85% 43, 14 10/19/03 Copyright Northrop Grumman Company 2003

S&T/ITS Program and QM Indicators Critical/Urgent Defect Density – Misc. Documentation Goal: Total Defects  1/Page Actual: Total STF Defects = .688/Page Total User Manual Defects = .709/page Total Misc. Doc Defects = .701/page (Critical & Urgent = .165/page) Action: No new SCoV Technical Highlights: Reviewed 100% documentation 100% documentation DDr exceeds goal (Misc Documentation Mean Critical + Urgent continues to decrease) Total Defect Density – Misc. Documentation 10/19/03 Copyright Northrop Grumman Company 2003

S&T/ITS Program and QM Indicators CLIN 1 and CLIN 2 – CPIm (Monthly – Sep 03) Goal: 1  0.1 (0.9 - 1.10) Actual: Current CPIm Mean = .730 Action: Continue ECP plans and attempt cost impact recovery from customer as part of the DMSI schedule alignment ECP. Technical Highlights: As expected the CPIm remains out of goal. Low number of defects allowed a positive cost variance to increase the mean by 5.7%. CLIN 1 and CLIN 2 – SPIm (Monthly – Sep 03) Goal: 1  0.1 (0.9 - 1.10) Actual: Current SPIm Mean = .753 Action: Continue with ECPs and Test Activities Technical Highlights: Released CBT. Preparing Software Test Report. 10/19/03 Copyright Northrop Grumman Company 2003

S&T/ITS Program and QM Indicators As stated in the ITS Quantitative Management Plan Category 1: Customer Satisfaction Goal Deliver no known open Level 1-3 severity defects Achieve contractual expectations as defined in the ITS SRS and implied customer expectations as defined in the ITS SQASP Actual 100% known peer review and test defects have been repaired. Customer completed performance survey. ITS was rated “Exceptional” in all categories. Looking to receive CPARs in CLIN 2. None received to date. Action None Technical Highlights Customer continues to fund contract mods to keep team momentum and buy conversion data schedule time. Customer continues to fund T&M CLIN. Requested synchronized ITS schedule to DMSI schedule. ·         . 10/19/03 Copyright Northrop Grumman Company 2003

Organizational Innovation To provide higher quality and better services to our customers, DES: Set aggressive goals based on past quantitative performance Research innovations internally and externally to measurably improve our performance. Report out monthly on performance… An example follows… L5 Maturity comes from organizational innovation (including TCM) The innovation needs to be data driven The following case study provided the data we needed to pilot and deploy PSP in applicable environments. Project results impact organizational results 10/19/03 Copyright Northrop Grumman Company 2003

Two projects with: Project Comparisons Same customer (different divisions) Similar applications Both Client-Server (PowerBuilder / Oracle) Similar level of team expertise and training Estimated to be similar size and effort (~$2 Million) 10/19/03 Copyright Northrop Grumman Company 2003

Key Process Insertion Differences Project A: Disciplined team design process used to create sound developer work packets Personal Software Process used consistently by developers SEI developed course for software developers which provides process at an individual level for producing software components and documentation (user & technical) Peer Reviews conducted on the most critical 20% of the software Project B: We decided that this project was “too far along” to benefit from process insertion 10/19/03 Copyright Northrop Grumman Company 2003

Project Test Time / Cost Project A Project B Integration / Acceptance Test Cost $78.K $612K Normalized (per KLOC) $0.95K $4.05K Time to Accept (months) 3.7 14.6 Normalized (months per 100 KLOC) 4.5 9.7 10/19/03 Copyright Northrop Grumman Company 2003

Project Quality Project A Project B Duration (months) 31.8 43.0 Size (KLOC) 82 151 Developer Defect Density 9.4 17.3 Peer Review Exit Density 4.78 Delivered Defect Density 1.55 5.27 10/19/03 Copyright Northrop Grumman Company 2003

Positive Effect of Process Insertion Mature Processes & PSP led to Project A’s Integration and Acceptance Test Costs were 24% of Project B Project A’s Test Duration was 46% of Project B Project A’s delivered defect density was 29% of Project B Process saved cost, reduced schedule, and improved quality 10/19/03 Copyright Northrop Grumman Company 2003

2002 SCAMPI Appraisal Results F1=Fully Implemented Out of Scope Process Maturity CMMI SE/SW Level 5 for Staged and Continuous Representations Software CMM Level 5 also Organizational Process Strengths Integrated SE and SW Eng Sharing and reusing process assets Data driven corrective action Technology Change Management Global Best Practices Requirements Elicitation Quantitative Data Management Cost, Schedule and Quality Performance JEDMICS exceeds software quality goals (.0156 vs .06 defects/KSLOC) PERMS exceeds revenue goals by 11% AIT has 100% on-time delivery Only 2% of total defects found in delivered SIGS product ITS is on cost and schedule (CPIm = .965 & SPIm = .985, target = 1.0 +/- .1) SIGS averages a customer satisfaction of 98.5% over last four years I haven’t been talking about ratings, but through deployment of the quantitative practices with data driven innovation, we This achieved a CMMI SE/SW Level 5 in December of 2002. (led by an external authorized lead evaluator). We also concurrently received a SW CMM Level 5 and a continuous rating of our capability The upper left box exhibits some strengths determined at the evaluation. The lower left depicts some project performance results. 10/19/03 Copyright Northrop Grumman Company 2003

Knowledge Management and Process Management We manage our Processes as a Community of Practice We help our integrated teams share and put process into action Processes underpin each “community of practice” Next Steps for DES Knowledge management is an enabler for “Communities of Practice,” groups of people with a common mission or discipline in behalf of the company. It allows people to access and share information effectively in support of the processes the community of practice uses. It facilitates collaboration, particularly when the community of practice is distributed geographically. As we go through this presentation, we will get to a more practical view of knowledge management and its importance to Northrop Grumman. We will further support process deployment as a KM community of practice 10/19/03 Copyright Northrop Grumman Company 2003

CMMI Extensibility and Flexibility Information assurance Help desk Integrated Enterprise Process IT Managed services Network management Program Management Processes Engineering Management Processes Engineering Processes Our integrated system and software engineering processes are depicted in the center of this exhibit exhibiting our management, engineering, support and process processes. Around the edges are other important IT and service activities that we perform for our customers (that are not easily “covered” by the CMMI at this time. We are evaluating how the CMMI can best be applied to these activities and will soon have a technical report on the match with recommended ways that the CMMI can be continuously improved. We need the CMMI sponsor and custodian to ensure CMMI flexibility and extensibility in line with our dynamic IT change Process Infrastructure Processes Asset management Desktop/Server management 11/18/2003 Copyright Northrop Grumman Company 2003

Conclusions CMMI Practices have a direct and positive influence on cost, schedule, and quality performance It helps to tie the quantitative goals and measures to customer needs and relevant project problems CMMI high maturity enables organizational and project agility in high priority areas Projects can improve dramatically in 6 months or less using defect prevention cycles of 1 week or less The transition from SW-CMM to CMMI is not inherently difficult, costly, and time-consuming, but… It pays to proactively improve against emerging models and standards It is much easier to attain the same maturity level the second time It is much easier to transition between models when you focus on performance Summarize key points Also Later this week, Pflugrad will be presenting our strategic process improvement plan and Craig Hollenbach will be presenting our quantitatively measured process improvements 10/19/03 Copyright Northrop Grumman Company 2003