1. Building a Business Case for Systems Engineering
Thank you for taking the time to meet with us today. We’re here to speak with you about improving acquisition program performance by leveraging the Systems Engineering activities of your suppliers.
Bob Rassa, Chair,
NDIA Systems Engineering Division
Past President, IEEE AES

2. Who Pulls it All Together ?
The Systems Engineer
Required skills
Global system-wide perspective
Full life-cycle perspective
Forward-looking
Multidisciplinary technical knowledge
Fact-based decision-making
Multi-tasking
Tasks Performed *
Requirements Development
Requirements Management
Trade Studies
System Architecture Development
Interface Management
Configuration Management
Project Planning
Project Monitoring and Control
Risk Management
Product Integration Planning and Oversight
Verification Planning and Oversight
Validation Planning and Oversight
How likely is project success if these activities are not done well?
So, who has the skills and multidisciplinary knowledge needed to pull all of these activities together?
______________________
It can only be the Systems Engineer.
The Systems Engineer maintains a global system-wide perspective that encompasses the entire life cycle of the product. He possesses multidisciplinary technical knowledge that enables him to address the interdisciplinary activities. He employs fact-based decision-making techniques, and manages multiple threads of the system design simultaneously.`
Consider the tasks performed by the Systems Engineering. If requirements development, trade studies, or architecture are done poorly, how likely is project success?
* Some tasks are done in partnership with the Project Manager

3. The Importance of System Engineering
GAO T - Actions Needed to Overcome Long-standing Challenges with Weapon Systems Acquisition and Service Contract Management
“costs … of major defense acquisition programs increased 26 percent and development costs increased by 40 percent from first estimates”
“programs … failed to deliver capabilities when promised—often forcing warfighters to spend additional funds on maintaining legacy systems”
“current programs experienced, on average, a 21-month delay in delivering initial capabilities to the warfighter”
Why?
“… managers rely heavily on assumptions about system requirements, technology, and design maturity, which are consistently too optimistic. These gaps are largely the result of a lack of a disciplined systems engineering analysis prior to beginning system development …
Systems Acquisition remains problematic for the DoD. A recent GAO report indicates that costs of acquisition programs are typically 26% over budget, and development costs are typically 40% over initial estimates.
These programs routinely fail deliver the capabilities when promised, experiencing, on average, a 21 month delay
___________________________
The report finds that optimistic assumptions about system requirements, technology, and design maturity play a large part in these failures, and that these optimistic assumptions are largely the result of a lack of disciplined systems engineering analysis early in the program.

4. The Problem
It is difficult to justify the costs of SE in terms that project managers and corporate managers can relate to.
The costs of SE are evident
Cost of resources
Schedule time
The benefits are less obvious and less tangible
Cost avoidance (e.g., reduction of rework from interface mismatches)
Risk avoidance (e.g., early risk identification and mitigation)
Improved efficiency (e.g., clearer organizational boundaries and interfaces)
Better products (e.g., better understanding and satisfaction of stakeholder needs)
Here’s the problem. While there is a lot of anecdotal evidence regarding the value of Systems Engineering, there is relatively little quantitative evidence
Everyone can see the costs of Systems Engineering – the costs of the labor applied, the time allocated in the schedule.
The benefits of Systems Engineering may be less identifiable. They often manifest themselves as
Risks that didn’t materialize
Rework that didn’t need to be done
Customer complaints that didn’t occur, and
Product deficiencies that are circumvented
These are difficult to quantify
To get a true picture of the value of Systems Engineering, we need to quantitatively measure it’s impact on project performance.
We need to quantify the effectiveness and value of SE by examining its effect on project performance?

5. Obtain quantitative evidence of the costs and associated benefits of
The Solution
Obtain
quantitative
evidence
of the
costs
and
associated
benefits of
Systems Engineering
activities via a
survey of
So our solution is to obtain quantitative evidence of the costs and associated benefits of Systems Engineering activities. We can do this via a survey of development projects
development projects

6. PROJECT PERFORMANCE vs. TOTAL SE CAPABILITY
The Bottom Line 1
For the projects that did the least SE, only 15% delivered the best project performance.
PROJECT PERFORMANCE vs. TOTAL SE CAPABILITY
1.00
0.75
0.50
0.25
0.00
Best
Performance
( x > 3.0 )
Moderate
( 2.5  x  3.0 )
Lower
( x < 2.5 )
15%
12%
56%
59%
46%
13%
39%
29%
31%
We processed the data to calculate a project performance ‘score’ based on conformance to budget and schedule, and satisfaction of requirements. Each project was categorized as exhibiting Lower, Moderate, or Best Performance based on this score. We set our thresholds so that approximately 1/3 of the projects fit within each of these categories
We processed the data to calculate a SE capability ‘score’ based on the existence and quality of SE artifacts. Each project was categorized as exhibiting Lower, Moderate, or Higher capabilities, based on this score. Again, we set our thresholds so that approximately 1/3 of the projects fit within each of these categories
We then analyzed these scores to quantify the strength of the relationship between them.
For those projects exhibiting Lower Systems Engineering capabilities, we found that 39% exhibited Lower Project Performance 46% exhibited Moderate Project Performance, and only 15% exhibited Best Project Performance.
For those projects exhibiting Moderate Systems Engineering capabilities, we found that 29% exhibited Lower Project Performance 59% exhibited Moderate Project Performance, and only 12% exhibited Best Project Performance.
But for those projects exhibiting Higher Systems Engineering capabilities, we found that 31% exhibited Lower Project Performance 13% exhibited Moderate Project Performance, and only 56% exhibited Best Project Performance.
This provides a clear indication of the value of SE.
We also calculate statistical measures of the relationship between SE and Project Performance. Gamma measures the strength of the relationship, where +1 indicates a strong supporting relationship, 0 indicates no relationship, and -1 indicates a strong opposing relationship.
P is the probability that the indicated relationship could arise by chance alone
Lower Capability
( x  2.5 )
N = 13
Moderate Capability
( 2.5 < x < 3.0 )
N = 17
Higher
Capability
(x  3.0 )
N = 16
For the projects that did the most SE, 56% delivered the best project performance
Gamma = 0.32
p = 0.04

7. Product Architecture Capability vs. Project Performance
Product architecture assessment examined
High-level product structure documentation
Including multiple views
Interface Descriptions
Using the same analysis techniques, we can assess the relationship between specific SE activities and Project Performance
Here we examine the relationship between Product Architecture and Project Performance.
In assessing Product Architecture, we examined the high-level product structure documentation and the interface descriptions developed by the project.
This area of SE showed the strongest single relationship to Project performance. For projects with Lower Capabilities in this area, only 11% exhibited Best Performance. For projects with Higher Capabilities in this area, 46% exhibited Best Performance.
The Gamma value of 0.4 indicates a moderately strong or strong relationship. The p-value of only 0.2% speaks to the strong validity of the data.
Better Product Architecture has a “Moderately Strong / Strong” positive relationship with Better Performance

8. Trade Study Capability vs. Project Performance
Trade Study assessment examined
Documentation of Trade Study selection criteria
Documentation or Trade Study results
Stakeholder involvement
Here we examine the relationship between Trade Study Capabilities and Project Performance.
In assessing Trade Studies, we examined the documentation of Trade Study selection criteria, the documentation of results, and levels of stakeholder involvement.
This area of SE also showed a significant positive relationship to Project performance. For projects with Lower Capabilities in this area, only 17% exhibited Best Performance. For projects with Higher Capabilities in this area, 49% exhibited Best Performance.
The Gamma value of 0.37 indicates a moderately strong or strong relationship. The p-value of only 3% indicates the good reliability of the data.
Better Trade Studies have a “Moderately Strong / Strong” positive relationship with Better Performance

9. Technical Solution Capability vs. Project Performance
Technical Solution performance is the combination of both Product Architecture and Trade Study performance
Technical Solution is the combination of both Product Architecture and Trade Study activities
This combination of areas of SE showed a significant positive relationship to Project performance. For projects with Lower Capabilities in this area, only 7% exhibited Best Performance. For projects with Higher Capabilities in this area, 46% exhibited Best Performance.
The Gamma value of 0.36 indicates a moderately strong relationship. The p- value of only 3% indicates good data reliability
Better Technical Solution processes have a “Moderately Strong” positive relationship with Better Performance

10. IPT Utilization vs. Project Performance
IPT (Integrated Product Team) assessment examined
Effective IPT Usage on Project
Supplier participation
IPT for Systems Engineering
SE Representation on each IPT
The effective utilization of Integrated Product Teams or IPTs also shows a moderately strong positive relationship with Project Performance.
In assessing IPT Utilization, we examined the deployment of IPTs on the project, supplier participation on IPTs, the existence of a Systems Engineering IPT, and Systems Engineering participation on other IPTs.
This area of SE also showed a significant positive relationship to Project performance. For projects with Lower Capabilities in this area, only 13% exhibited Best Performance. For projects with Higher Capabilities in this area, 53% exhibited Best Performance.
The Gamma value of 0.34 indicates a moderately strong relationship. The p- value of only 4% indicates good data reliability.
Better IPT Deployment has a “Moderately Strong” positive
relationship with Better Performance

11. Requirements Development & Management vs. Project Performance
Requirements assessment examined
Customer & derived requirements lists
Hierarchical allocation to system elements
CONOPs, scenarios, and Use cases
Criteria for authorization of req’ts providers and acceptance of req’ts
Change control process
Traceability to Stakeholder needs
Requirements Development and Management also shows a moderately strong positive relationship with Project Performance.
In assessing Requirements Development capabilities, we examined the documentation of customer-defined and derived requirements, the hierarchical allocation of requirements to system elements, and the existence of CONOPs, scenarios, and use cases. In assessing Requirements Management capabilities, we examined the existence of criteria for authorization of requirements providers, criteria for acceptance of requirements, an effective change control process, and traceability of requirements to stakeholder needs.
This area of SE also showed a significant positive relationship to Project performance. For projects with Lower Capabilities in this area, only 18% exhibited Best Performance. For projects with Higher Capabilities in this area, 55% exhibited Best Performance.
The Gamma value of 0.33 indicates a moderately strong relationship. The p- value of only 4% indicates good data reliability
Better Requirements Development and Management has a
“Moderately Strong” positive relationship with Better Performance

12. Requirements + Technical Solution Capability vs. Project Performance
When looking at the impact of COMBINED SE activities, we see even stronger relationships
When we analyze COMBINED areas of Systems Engineering, we find even stronger relationships to Project Performance.
For projects with Lower Capabilities in both Requirements Development and Management and Technical Solution, only 13% exhibited Best Performance. For projects with Higher Capabilities in both of these areas, 50% exhibited Best Performance.
The Gamma value of 0.49 indicates a STRONG relationship. The p-value of only 0.5% indicates excellent data reliability
Better Requirements Dev’t & Mg’t and Better Technical Solution processes have a “Strong” positive relationship with Better Performance

13. Summary of Relationships
This chart shows the strength of relationship between each of the Systems Engineering Process Areas and Project Performance, sorted by Gamma value. The information on this chart provides guidance as to which Systems Engineering activities have the greatest impact.
You might note the NEGATIVE relationship between Project Monitoring and Control and Project Performance. Does this mean that the more we monitor a project, the worse it performs? Remember that our analysis is only identifying relationships, not cause and effect. More likely this negative relationship means that the worse a project performs, the more monitoring and control it is subjected to.
Composite Measures
Moderately Strong
to Strong Relationship
Moderately Strong
Relationship
Strong Relationship
Weak Relationship

14. Validation vs. Project Performance
Validation assessment examined
Validation Procedures
Documented Acceptance Criteria
List of items under Configuration Management
Better Validation capabilities have a “Moderately Strong” positive relationship with Better Performance

15. Risk Management vs. Project Performance
Risk Management assessment examined
List of Risks
Risk Mitigation Plans
Monitoring and Reporting of Risks and Mitigation Plans
Integration with Project Decision Making
Integration with IMS
Better Risk Management has a “Moderately Strong” positive relationship with Better Performance

16. Verification vs. Project Performance
Verification assessment examined
Verification Procedures
Documented Acceptance Criteria
Documented Technical Review Process
Documented non-advocate reviews
Better Verification capabilities have a “Moderately Strong” positive relationship with Better Performance

17. Product Integration vs. Project Performance
Product Integration assessment examined
Documented Integration Process
Documented Integration Criteria
Better Product Integration capabilities have a “Weak” positive relationship with Better Performance

18. Configuration Mg’t vs. Project Performance
Product Integration assessment examined
Change Control Board Charter
Records of requested and implemented changes
Configuration Baselines
Better Configuration Management capabilities have a “Weak” positive relationship with Better Performance

19. Project Planning vs. Project Performance
Project Planning assessment examined
Project Planning Processes
Work Breakdown Structure
Technical Approach
IMP and IMS
Plan for technical reviews
Systems Engineering Plan
Better Project Planning capabilities have a “Weak” positive relationship with Better Performance

20. Project Monitoring vs. Control and Project Performance
Project Planning assessment examined
SE Costing and Tracking
Cost and Schedule Baselines
EVMS Data
EVMS Data from Suppliers
Defined Thresholds for SPI and CPI variance
Better Project Monitoring and Control capabilities have a “Weak” negative relationship with Better Performance

21. Project Challenge vs. Project Performance
Project challenge factors:
# of Life cycle phases
Project characteristics (e.g., size, effort, duration, volatility)
Technical complexity
Teaming relationships
As projects become more challenging (e.g., larger, more complex, longer, more organizations involved), success declines. ‘Easy’ projects (the lower third of project challenge) exhibit 50% Best Performance. ‘Hard’ Projects (the upper third of Project Challenge) show only 25% Best Performance
More Challenging Projects do not perform as well.

22. Relating Project Performance to Project Challenge and SE Capability