SynGenics Corporation 72 E Granville Road Worthington OH ©2003 SynGenics Corporation. All rights reserved. The Use of Multicriterial Optimization Analysis and Sensitivity as a Measure of Risk In Aerospace Systems Development Carol Ventresca and George A. Richards, Ph.D., SynGenics Marvin C. Gridley and Gregory A. Addington, Ph.D. AFRL/VAA Presented at INFORMS Annual Meeting, Atlanta, Georgia, 19–21 October 2003 Document Number ASC Cleared for public release.

2 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Presentation Outline  Structurally Integrated Compact Inlet Technology (STRICT)  Affordability Process and Metrics  Sensitivity Analysis for Robust Design  Results  Conclusions  Status

3 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Goals of STRICT Program  Duct Configuration  Best propulsion system performance  100% LOS blockage  L/D goal = 2.5, threshold = 3.0  ACIS class total pressure recovery  Minimize steady state distortion  Minimize first 4 harmonics  Consider structural integration  Flow Control System  Minimize flow/engine impact  Minimize integration impact, i.e., ducting, structural  Provide best-value technology advancements, while identifying and managing cost and risk drivers as an integral part of the program

4 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Potential Customers for STRICT  Uninhabited Air Vehicles  Sensor Craft: high altitude, long endurance, moderate LO  UCAV: DARPA/AFRL UCAV mission, lower cost, higher performance, more LO  Future Strike  M ~ 2.5 long range, high performance, very LO  Transport  NASA BWB  SOF/ATT  Highly integrated high BPR engines  Very LO for USAF

5 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Affordability Process  Capture the Voice of the Customer  Identify Customers  Capture Each Customer’s Criteria  Synthesize Criteria into a Composite Set to Guide the Program  Define Exit Criteria  Select and Optimize Solution Concept  Identify and Select Best Solution Principle Define Alternatives Evaluate Each with Respect to Criteria Perform Value Analysis, Create Scorecard  Optimize Selected Alternative

6 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Focus on Affordability  Leads to  Better Systems Performance: Does What the Customer Wants it to Do Reliability: Meets or Exceeds Needs Cost: At a Price the Customer Will Pay Supportability: Available for Use within Resources  More Efficient Design Process  Directed by Air Force Research Laboratory (AFRL) & Department of Defense (DoD)

7 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Block Diagram of the Process Model Courtesy Air Force Research Laboratory

8 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. IPPD Process Artifacts  Quantitative Predictive Measures  Response Values Related to Criteria  Quantified Uncertainty and Risk  Documented Criteria (e.g., Exit Criteria)  Worksheets and Scorecards  Relationships: Factors and Responses  Graphics

9 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Requirements/Criteria  Criteria, a.k.a. “Requirements”  Carefully Documented  For Each Customer  For the Project (“Constructed Customer”)  Captured in Requirements Matrix or HOQ  Are Specific to Customer  Defined to Be Measurable

10 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Measures of Merit  Two Predictive Measures for Each Alternative  Desirability: a Measure of Customer Satisfaction  Risk: Probability of Failure  Calculated at Three Levels  With Respect to Each Criterion  With Respect to Each Type of Criteria  Overall  For Each “Customer”

11 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Worksheet  Presents  One Alternative  Against All Criteria of a Single Type  For One Customer  Metrics for Each Criterion  Expected Value  Measure of Variability  d  Expected Desirability    Probability of Failure to Meet Threshold

12 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. The Value Scorecard  Comprises  One Scorecard for Each Type  The Affordability Scorecard  Relates to a Single Defined Customer  The “Constructed Customer” with its Composite Set of Requirements Is Key to Project Management

13 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Value Scorecard (Low Level)  Presents  Multiple Alternatives  For Criteria of a Specified Type  For a Single Customer  Metrics for Each Criterion  Expected Value  d i  Expected Desirability   i  Probability of Failure to Meet Threshold

14 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Value Scorecard (Low Level) Continued  Aggregates Metrics across Type  D j  Composite Desirability for Type j   j  Probability of Failure to Meet  One Threshold for a Criterion of Given Type j  Called “The ‘Type’ Scorecard”  e.g., “The Performance Scorecard”  Facilitates Identification of Discriminators, Risk Drivers, and Technical Challenges

15 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Value Scorecard (High Level)  “Affordability Scorecard” Presents  Multiple Alternatives  For All Types  For a Single Customer  Metrics for Each Criterion Type  D j  Composite Desirability for Type j   j  Probability of Failure to Meet  One Threshold for Criteria of Type j

16 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Value Scorecard (High Level) Continued  Top-Level Aggregate Metrics  CSI  Customer Satisfaction Index = Composite Desirability Aggregated across Types   Aff  Pr(Failure) to Meet  One Threshold  Called “The Affordability Scorecard”  Facilitates Identification of  Cost and Risk Drivers  Technical Challenges and Tradeoffs

17 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Attributes of Criteria  ID#, Name, Description  Type (e.g., Performance, Cost)  Unit of Measure  Customer-Specific Attributes  Objective  Threshold(s)  Priority and Weighting Factor within Type  Desirability Function  Weighting of Types with Respect to Each Other

18 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. STRICT UCAV Requirements

19 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. STRICT UCAV Requirements Continued

20 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. STRICT Weighting Summary

21 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. STRICT Performance Scorecard

22 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. STRICT Producibility Scorecard

23 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. STRICT Cost Scorecard

24 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. STRICT Other Scorecard

25 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. STRICT “Affordability” Scorecard

26 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Sensitivity Analysis  Sensitivity Analysis Was Performed  To Identify Which Criteria Had Significant Influence  To Evaluate the Influence of Weighting Factors  To Highlight Areas Where Special Attention Is Warranted

27 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Sensitivity to Criterion Response Values

28 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Comparison of Sensitivities for the Three Alternative Configurations

29 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Explored Partial Derivatives  Calculated Partial Derivatives  Of Each Measure of Merit  With Respect to Each Performance, Producibility, and Cost Criterion  Modeled Sensitivity of CSI to Weighting Factors for Types

30 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. More Sensitivity Analysis Conclusions  The Results Are Most Sensitive To  #25, Number of Sensors  #32, Inlet Structural Procurement Cost  #33, Operations and Support Costs  The Results Are Least Sensitive To  #17, Throat Mach Number  #42, Inlet Structural Weight  #24, Number of Effectors

31 SynGenics Corporation © 2003 SynGenics Corporation. All rights reserved. Status of STRICT  A Good Set of Criteria Has Been Established  Further Consideration Given to Whether the Set is Complete  Configuration 20X Was Selected for Further Development  O&S, Procurement Cost, # of Sensors Were Discriminators  Identification of Criteria with Greatest Influence Supports Resource Investment Decisions (e.g., Collection of Data)  The Criteria with Highest Weightings and Lowest Desirabilities Are the Ones to Which the Results Are Most Sensitive  STRICT Is Proceeding with IPPD Affordability Objectives in Focus