Building a Business Case for Systems Engineering Thank you for taking the time to meet with us today. We’re here to speak with you about improving acquisition program performance by leveraging the Systems Engineering activities of your suppliers. Bob Rassa, Chair, NDIA Systems Engineering Division Past President, IEEE AES

Who Pulls it All Together ? The Systems Engineer Required skills Global system-wide perspective Full life-cycle perspective Forward-looking Multidisciplinary technical knowledge Fact-based decision-making Multi-tasking Tasks Performed * Requirements Development Requirements Management Trade Studies System Architecture Development Interface Management Configuration Management Project Planning Project Monitoring and Control Risk Management Product Integration Planning and Oversight Verification Planning and Oversight Validation Planning and Oversight How likely is project success if these activities are not done well? So, who has the skills and multidisciplinary knowledge needed to pull all of these activities together? ______________________ It can only be the Systems Engineer. The Systems Engineer maintains a global system-wide perspective that encompasses the entire life cycle of the product. He possesses multidisciplinary technical knowledge that enables him to address the interdisciplinary activities. He employs fact-based decision-making techniques, and manages multiple threads of the system design simultaneously.` Consider the tasks performed by the Systems Engineering. If requirements development, trade studies, or architecture are done poorly, how likely is project success? * Some tasks are done in partnership with the Project Manager

The Importance of System Engineering GAO-09-362T - Actions Needed to Overcome Long-standing Challenges with Weapon Systems Acquisition and Service Contract Management “costs … of major defense acquisition programs increased 26 percent and development costs increased by 40 percent from first estimates” “programs … failed to deliver capabilities when promised—often forcing warfighters to spend additional funds on maintaining legacy systems” “current programs experienced, on average, a 21-month delay in delivering initial capabilities to the warfighter” Why? “… managers rely heavily on assumptions about system requirements, technology, and design maturity, which are consistently too optimistic. These gaps are largely the result of a lack of a disciplined systems engineering analysis prior to beginning system development … Systems Acquisition remains problematic for the DoD. A recent GAO report indicates that costs of acquisition programs are typically 26% over budget, and development costs are typically 40% over initial estimates. These programs routinely fail deliver the capabilities when promised, experiencing, on average, a 21 month delay ___________________________ The report finds that optimistic assumptions about system requirements, technology, and design maturity play a large part in these failures, and that these optimistic assumptions are largely the result of a lack of disciplined systems engineering analysis early in the program.

The Problem It is difficult to justify the costs of SE in terms that project managers and corporate managers can relate to. The costs of SE are evident Cost of resources Schedule time The benefits are less obvious and less tangible Cost avoidance (e.g., reduction of rework from interface mismatches) Risk avoidance (e.g., early risk identification and mitigation) Improved efficiency (e.g., clearer organizational boundaries and interfaces) Better products (e.g., better understanding and satisfaction of stakeholder needs) Here’s the problem. While there is a lot of anecdotal evidence regarding the value of Systems Engineering, there is relatively little quantitative evidence Everyone can see the costs of Systems Engineering – the costs of the labor applied, the time allocated in the schedule. The benefits of Systems Engineering may be less identifiable. They often manifest themselves as Risks that didn’t materialize Rework that didn’t need to be done Customer complaints that didn’t occur, and Product deficiencies that are circumvented These are difficult to quantify To get a true picture of the value of Systems Engineering, we need to quantitatively measure it’s impact on project performance. We need to quantify the effectiveness and value of SE by examining its effect on project performance?

Obtain quantitative evidence of the costs and associated benefits of The Solution Obtain quantitative evidence of the costs and associated benefits of Systems Engineering activities via a survey of So our solution is to obtain quantitative evidence of the costs and associated benefits of Systems Engineering activities. We can do this via a survey of development projects development projects

PROJECT PERFORMANCE vs. TOTAL SE CAPABILITY The Bottom Line 1 For the projects that did the least SE, only 15% delivered the best project performance. PROJECT PERFORMANCE vs. TOTAL SE CAPABILITY 1.00 0.75 0.50 0.25 0.00 Best Performance ( x > 3.0 ) Moderate ( 2.5  x  3.0 ) Lower ( x < 2.5 ) 15% 12% 56% 59% 46% 13% 39% 29% 31% We processed the data to calculate a project performance ‘score’ based on conformance to budget and schedule, and satisfaction of requirements. Each project was categorized as exhibiting Lower, Moderate, or Best Performance based on this score. We set our thresholds so that approximately 1/3 of the projects fit within each of these categories We processed the data to calculate a SE capability ‘score’ based on the existence and quality of SE artifacts. Each project was categorized as exhibiting Lower, Moderate, or Higher capabilities, based on this score. Again, we set our thresholds so that approximately 1/3 of the projects fit within each of these categories We then analyzed these scores to quantify the strength of the relationship between them. For those projects exhibiting Lower Systems Engineering capabilities, we found that 39% exhibited Lower Project Performance 46% exhibited Moderate Project Performance, and only 15% exhibited Best Project Performance. For those projects exhibiting Moderate Systems Engineering capabilities, we found that 29% exhibited Lower Project Performance 59% exhibited Moderate Project Performance, and only 12% exhibited Best Project Performance. But for those projects exhibiting Higher Systems Engineering capabilities, we found that 31% exhibited Lower Project Performance 13% exhibited Moderate Project Performance, and only 56% exhibited Best Project Performance. This provides a clear indication of the value of SE. We also calculate statistical measures of the relationship between SE and Project Performance. Gamma measures the strength of the relationship, where +1 indicates a strong supporting relationship, 0 indicates no relationship, and -1 indicates a strong opposing relationship. P is the probability that the indicated relationship could arise by chance alone Lower Capability ( x  2.5 ) N = 13 Moderate Capability ( 2.5 < x < 3.0 ) N = 17 Higher Capability (x  3.0 ) N = 16 For the projects that did the most SE, 56% delivered the best project performance Gamma = 0.32 p = 0.04

Product Architecture Capability vs. Project Performance Product architecture assessment examined High-level product structure documentation Including multiple views Interface Descriptions Using the same analysis techniques, we can assess the relationship between specific SE activities and Project Performance Here we examine the relationship between Product Architecture and Project Performance. In assessing Product Architecture, we examined the high-level product structure documentation and the interface descriptions developed by the project. This area of SE showed the strongest single relationship to Project performance. For projects with Lower Capabilities in this area, only 11% exhibited Best Performance. For projects with Higher Capabilities in this area, 46% exhibited Best Performance. The Gamma value of 0.4 indicates a moderately strong or strong relationship. The p-value of only 0.2% speaks to the strong validity of the data. Better Product Architecture has a “Moderately Strong / Strong” positive relationship with Better Performance

Trade Study Capability vs. Project Performance Trade Study assessment examined Documentation of Trade Study selection criteria Documentation or Trade Study results Stakeholder involvement Here we examine the relationship between Trade Study Capabilities and Project Performance. In assessing Trade Studies, we examined the documentation of Trade Study selection criteria, the documentation of results, and levels of stakeholder involvement. This area of SE also showed a significant positive relationship to Project performance. For projects with Lower Capabilities in this area, only 17% exhibited Best Performance. For projects with Higher Capabilities in this area, 49% exhibited Best Performance. The Gamma value of 0.37 indicates a moderately strong or strong relationship. The p-value of only 3% indicates the good reliability of the data. Better Trade Studies have a “Moderately Strong / Strong” positive relationship with Better Performance

Technical Solution Capability vs. Project Performance Technical Solution performance is the combination of both Product Architecture and Trade Study performance Technical Solution is the combination of both Product Architecture and Trade Study activities This combination of areas of SE showed a significant positive relationship to Project performance. For projects with Lower Capabilities in this area, only 7% exhibited Best Performance. For projects with Higher Capabilities in this area, 46% exhibited Best Performance. The Gamma value of 0.36 indicates a moderately strong relationship. The p- value of only 3% indicates good data reliability Better Technical Solution processes have a “Moderately Strong” positive relationship with Better Performance

IPT Utilization vs. Project Performance IPT (Integrated Product Team) assessment examined Effective IPT Usage on Project Supplier participation IPT for Systems Engineering SE Representation on each IPT The effective utilization of Integrated Product Teams or IPTs also shows a moderately strong positive relationship with Project Performance. In assessing IPT Utilization, we examined the deployment of IPTs on the project, supplier participation on IPTs, the existence of a Systems Engineering IPT, and Systems Engineering participation on other IPTs. This area of SE also showed a significant positive relationship to Project performance. For projects with Lower Capabilities in this area, only 13% exhibited Best Performance. For projects with Higher Capabilities in this area, 53% exhibited Best Performance. The Gamma value of 0.34 indicates a moderately strong relationship. The p- value of only 4% indicates good data reliability. Better IPT Deployment has a “Moderately Strong” positive relationship with Better Performance

Requirements Development & Management vs. Project Performance Requirements assessment examined Customer & derived requirements lists Hierarchical allocation to system elements CONOPs, scenarios, and Use cases Criteria for authorization of req’ts providers and acceptance of req’ts Change control process Traceability to Stakeholder needs Requirements Development and Management also shows a moderately strong positive relationship with Project Performance. In assessing Requirements Development capabilities, we examined the documentation of customer-defined and derived requirements, the hierarchical allocation of requirements to system elements, and the existence of CONOPs, scenarios, and use cases. In assessing Requirements Management capabilities, we examined the existence of criteria for authorization of requirements providers, criteria for acceptance of requirements, an effective change control process, and traceability of requirements to stakeholder needs. This area of SE also showed a significant positive relationship to Project performance. For projects with Lower Capabilities in this area, only 18% exhibited Best Performance. For projects with Higher Capabilities in this area, 55% exhibited Best Performance. The Gamma value of 0.33 indicates a moderately strong relationship. The p- value of only 4% indicates good data reliability Better Requirements Development and Management has a “Moderately Strong” positive relationship with Better Performance

Requirements + Technical Solution Capability vs. Project Performance When looking at the impact of COMBINED SE activities, we see even stronger relationships When we analyze COMBINED areas of Systems Engineering, we find even stronger relationships to Project Performance. For projects with Lower Capabilities in both Requirements Development and Management and Technical Solution, only 13% exhibited Best Performance. For projects with Higher Capabilities in both of these areas, 50% exhibited Best Performance. The Gamma value of 0.49 indicates a STRONG relationship. The p-value of only 0.5% indicates excellent data reliability Better Requirements Dev’t & Mg’t and Better Technical Solution processes have a “Strong” positive relationship with Better Performance

Summary of Relationships This chart shows the strength of relationship between each of the Systems Engineering Process Areas and Project Performance, sorted by Gamma value. The information on this chart provides guidance as to which Systems Engineering activities have the greatest impact. You might note the NEGATIVE relationship between Project Monitoring and Control and Project Performance. Does this mean that the more we monitor a project, the worse it performs? Remember that our analysis is only identifying relationships, not cause and effect. More likely this negative relationship means that the worse a project performs, the more monitoring and control it is subjected to. Composite Measures Moderately Strong to Strong Relationship Moderately Strong Relationship Strong Relationship Weak Relationship

Validation vs. Project Performance Validation assessment examined Validation Procedures Documented Acceptance Criteria List of items under Configuration Management Better Validation capabilities have a “Moderately Strong” positive relationship with Better Performance

Risk Management vs. Project Performance Risk Management assessment examined List of Risks Risk Mitigation Plans Monitoring and Reporting of Risks and Mitigation Plans Integration with Project Decision Making Integration with IMS Better Risk Management has a “Moderately Strong” positive relationship with Better Performance

Verification vs. Project Performance Verification assessment examined Verification Procedures Documented Acceptance Criteria Documented Technical Review Process Documented non-advocate reviews Better Verification capabilities have a “Moderately Strong” positive relationship with Better Performance

Product Integration vs. Project Performance Product Integration assessment examined Documented Integration Process Documented Integration Criteria Better Product Integration capabilities have a “Weak” positive relationship with Better Performance

Configuration Mg’t vs. Project Performance Product Integration assessment examined Change Control Board Charter Records of requested and implemented changes Configuration Baselines Better Configuration Management capabilities have a “Weak” positive relationship with Better Performance

Project Planning vs. Project Performance Project Planning assessment examined Project Planning Processes Work Breakdown Structure Technical Approach IMP and IMS Plan for technical reviews Systems Engineering Plan Better Project Planning capabilities have a “Weak” positive relationship with Better Performance

Project Monitoring vs. Control and Project Performance Project Planning assessment examined SE Costing and Tracking Cost and Schedule Baselines EVMS Data EVMS Data from Suppliers Defined Thresholds for SPI and CPI variance Better Project Monitoring and Control capabilities have a “Weak” negative relationship with Better Performance

Project Challenge vs. Project Performance Project challenge factors: # of Life cycle phases Project characteristics (e.g., size, effort, duration, volatility) Technical complexity Teaming relationships As projects become more challenging (e.g., larger, more complex, longer, more organizations involved), success declines. ‘Easy’ projects (the lower third of project challenge) exhibit 50% Best Performance. ‘Hard’ Projects (the upper third of Project Challenge) show only 25% Best Performance More Challenging Projects do not perform as well.

Relating Project Performance to Project Challenge and SE Capability