1. SpaceLiner 100 Propulsion Task Force Candidate Technology Evaluation and Assessment & Prioritization Workshop Huntsville, Al. April 10-11, 2001 Russel Rhodes NASA KSC

2. Purpose of Review Provide a review of the “Technology Evaluation Process” developed by the SPST in support of the SL-100 Functional Requirements and Key Objectives. –Focus is on Criteria Development and weighting used in Workshop –Supportive of Civil, DOD, and Commercial needs for Third Generation Space Transportation Systems Challenge the affected technical community to consider the knowledge and information from this process in developing and justifying the technologies capable of satisfying SL-100 key objectives. –“Example: Two orders of magnitude decrease in costs & 10,000 times safer than today's Space Transportation Sys.”

3. The Evaluation Tool Development Process for Technical Benefit Define Customer What Does the Customer Want? Develop “Whats” Customer Weight the “Whats” Determine the “Hows” for Each “What” Determine the “Hows” for Each “What” Establish Correlation of Each “How” to Every “What” Synergy Group Synergy Group Establish Relative Merit of Each “How” Result List of Prioritized and Weighted “Hows” Which Produce the “Whats” That the Customer Wanted Evaluate Each Candidate on Each “How” Evaluate Each Candidate on Each “How” Prioritized List of Propulsion Technologies Likely to Achieve Goals Prioritized List of Propulsion Technologies Likely to Achieve Goals Synergy Group Tech. Candidates

4. The Attributes of a Space Transportation System Operating Programmatics How do we improve in all these phases?

5. The Attributes Prioritized But these are still qualitative “whats” … we required a more useful expression of “how”... 3 fold scoring used - importance, plus need to improve and where are we now.

6. Measurable Criteria - “How” Used an Iterative Structured Process Process forces the consideration of improvement in customer wants

7. SpaceLiner 100 Propulsion Task Force SPST Task Charter SPST Approach Complies with National Space Policy: Attributes are Anchored to the National Space Policy: SPST Flow Diagram Responds to National Space Policy: SPST Will Provide Recommended Functional Requirements Necessary to Develop Space Transportation Architectures, Concepts, and Systems that Surface the Technology Needs Required to Realize the Objectives of SpaceLiner 100 “3rd Generation RLV Feasible Concepts” SPST will Assist NASA in the Development of a Prioritized Portfolio of Technologies Capable of Reaching the SpaceLiner 100 3rd Generation RLV Objectives SPST SPACELINER 100 PROPULSION TASK FORCE ”FUNCTIONAL REQUIREMENTS SUB-TEAM" OBJECTIVE: Develop SL-100 functional requirements capable of achieving the 3rd generation RLV objectives Develop the Technology Evaluation Criteria and Weights for both Technical Benefit and Programmatics for use in the AHP model to be used at a Technology Evaluation Workshop while anchoring on the existing SPST data base where possible. This sub-team accepted Garry Lyles challenge to further develop and take accountability for his algorithm for “Systems Approach to Safety, Reliability, and Cost” and anchor the Evaluation Workshop Criteria to the algorithm

8. SpaceLiner 100 Propulsion Task Force In addressing this objective, SpaceLiner 100 Propulsion Technology 3rd Generation RLV Evaluation Criteria--The First Step: ---Select the correct team make-up ---Must be balanced with Designers, Operators, Managers, and Technologists ---Required to bring the needed experience and knowledge together for consensus building to provide quality and creditable process

9. SpaceLiner 100 Propulsion Task Force Functional Requirements Sub-Team Membership Russel Rhodes, NASA-KSC - Lead Uwe Hueter, NASA-MSFC Walt Dankhoff, SAIC Bryan DeHoff, Aero.Tech.Serv. Glen Law, Aerospace Corp. Mark Coleman, CPIA Robert Bruce, NASA-SSC Ray Byrd, Boeing-KSC Clyde Denison, NGC Bill Pannell, NASA-MSFC Dan Levack, Boeing/Rocketdyne Bill Escher, SAIC Pat Odom, SAIC David Christensen, LMCO Jim Bray, LM-MAF Tony Harrison, NASA-MSFC Keith Dayton/John Robinson, Boeing Co Andy Prince, MSFC Carey McCleskey, NASA-KSC Jay Penn, Aerospace Corp. John Hutt, NASA-MSFC CUSTOMER PROVIDING EVALUATION INPUT: Uwe Hueter, NASA-MSFC

10. 71% Influence (29% Other Factors) Influence Diagramming Stresses importance of the Dependability Attribute Objectives on achieving SpaceLiner Goals Safety Cost Operable Responsive Recurring Cost Spaceliner Goals Lyles Algorithm Influence Diagramming Approach 41% 33% 52% 56%44% (59% Other) (67% Other) (48% Other) (Life Cycle Cost) Non-Recurring Cost Dependable (InherentReliability) SpaceLiner-100 Propulsion Task Force SpaceLiner 100 Key Attribute Influence Relationships Management Visibility of Influence Achieved when using Weighted Design Criteria for Technology Prioritization ( The How’s to achieve the What’s Desired )

11. R&D Investment Influence on Achievement of SpaceLiner-100 Key Objectives Operations and DDT&E Integrated Key Attributes Influence Relationships 100X CHEAPER COST, $/LB LOW DDT&E ACQUISITION COST 10,000 X SAFER OPERABLE LOW RECURRING COST RESPONSIVE LOW NON- RECURRING COST INVESTORS INCENTIVE LOW LIFE CYCLE COST SAFE DEPENDABLE INHERENT RELIABILITY DUAL USE POTENTIAL LOW COST R&D BENEFIT FOCUSED SHORT SCHEDULE TECHNOLOGY OPTIONS LOW RISK DDT&E SHORT SCHEDULE R&D DDT&E OPERS COST FOCUS ATTRIBUTES KEY LOW RISK R&D R&D ATTRTIBUTES DDT&E ATTRIBUTES OPERATIONS ATTRIBUTES L I F E C Y C L E C O S T NON-RECURRING INVESTMENT GEN3 GOALS FLEET PURCHASE

12. SpaceLiner-100 Propulsion Task Force SpaceLiner-100 Assessment/Prioritization Process & Criteria Sub-Team Products for SL-100 3 rd Gen. RLV Developed Functional Requirements for SL-100 3rd Generation RLV Developed the Influence Diagram Algorithm with SPST 3rd Gen. RLV data base focusing on SL-100 Goals Established and weighted desired attributes focusing on the importance and need to improve to achieve the SL-100 performance requirements Established and weighted measurable design criteria and provided paretos Identified and selected the top 26 good discriminating design criteria with weights for use at evaluating top level technologies

13. SpaceLiner-100 Propulsion Task Force SpaceLiner-100 Assessment/Prioritization Process & Criteria Sub-Team Products for SL-100 3 rd Gen. RLV "Continued" Established programmatic factors and weighted measurable quality characteristics for both the Acquisition and R & D Phases and provided paretos Provided functional requirements, design criteria, and programmatic criteria to Technology Candidate Developers Provided Technical Benefit (Design Criteria) and Programmatics Criteria (Acquisition and R & D Phases) for use in the AHP evaluation tool Developed a definitions reference document to capture terminology used in this work to provide understanding and communications required for process usefulness

14. BENEFIT (TECHNICAL) ATTRIBUTES WEIGHTING Note: Weighting Factors are 1 to 5 with 5 being the most important

15. BENEFIT (TECHNICAL) ATTRIBUTES WEIGHTING

16. BENEFIT (TECHNICAL) ATTRIBUTES WEIGHTING PRIORITIZED

17. SpaceLiner-100 Propulsion Task Force SL-100 Propulsion Assessment/Prioritization Process & Criteria Sample Piece of Actual Matrix

18. SpaceLiner-100 Propulsion Task Force SL-100 Propulsion Assessment/Prioritization Process & Criteria Pareto of all Design Criteria down to Top 26 good discriminators (*) used in Workshop Process # of active systems required to maintain a safe vehicle (-)603 2.72% # of different propulsion systems (-)582 * 2.62% 5.34% # of systems with BIT BITE (+)542 2.45% 7.79% # of components with demonstrated high reliability (+)541 2.44% 10.% # of hands on activities req'd (-)534 2.41% 12.63% # of active components required to function including flight operations (-)527 * 2.38% 15.01% # of potential leakage / connection sources (-)527 2.37% 17.39% # of systems requiring monitoring due to hazards (-)523 2.36% 19.74% System margin (+)508 * 2.29% 22.03% # of toxic fluids (-)495 * 2.23% 24.27% % of propulsion system automated (+)488 * 2.20% 26.47% # of unique stages (flight and ground) (-)483 * 2.18% 28.64% % of propulsion subsystems monitored to change from hazard to safe (+)470 2.12% 30.76% # of in-space support sys. req'd for propulsion sys. ( - ) 465 2.10% 32.86% Design Variability (-)464 * 2.09% 34.95% # of active on-board space sys. req'd for propulsion ( - ) 454 * 2.05% 37.00% On-board Propellant Storage & Management Difficulty in Space (-) 453 * 2.04% 39.04% # of purges required (flight and ground) (-)428 1.93% 40.97% # of confined spaces on vehicles (-)427 1.92% 42.89% Technology readiness levels (+)425 * 1.92% 44.81% # of active ground systems required for servicing (-)420 1.89% 46.71% # of different fluids in system (-)404 * 1.82% 48.53% # of checkouts required (-)403 1.82% 50.34% # of propulsion sub-systems with fault tolerance (+)398 * 1.79% 52.14% # of inspection points (-)390 1.76% 53.90% Mass Fraction required (-)387 * 1.75% 55.64% Hours for turnaround (between launches or commit to new mission) (-) 374 1.69% 57.33% ISP Propellant transfer operation difficulty (resupply) (-) 371 1.68% 59.01% # of pollutive or toxic materials (-)350 1.58% 60.58% # of expendables (fluid, parts, software) (-)348 1.57% 62.15% Minimum Impulse bit (-) 332 1.50% 63.65% # of criticality 1 failure modes (-)329 1.48% 65.13% # of element to element interfaces requiring engineering control (-)320 1.44% 66.57% Ave. Isp on refer. trajectory (+)310 * 1.40% 67.97% # of parts (different, backup, complex) (-)296 1.33% 69.31% #of umbs. req'd to Launch Vehicle ( - ) 276 * 1.25% 70.55% # of engines (-)274 * 1.24% 71.79% Resistance to Space Environment (+) 268 * 1.21% 73.00% # of physically difficult to access areas (-)265 1.19% 74.19% # of active engine systems required to function (-)247 * 1.11% 75.30% Integral structure with propulsion sys. (+) 239 * 1.08% 76.38% Hours to refurbish propulsion system (-)237 1.07% 77.45% # of manhours (c/o, handle, assemble etc) on system between on and off cycles (Low Cycle Fatigue) or use (High Cycle Fatigue) (-)229 1.03% 78.48% # of modes or cycles (-)227 * 1.02% 79.50% # of ground power systems (-)226 * 1.02% 80.52% Mean time between major overhaul (+) 221 1.00% 81.52% Amount of energy release from unplanned reaction of propellant (-)219 * 0.99% 82.51% Margin, mass fraction (+)215 * 0.97% 83.48% Margin, thrust level / engine chamber press(+)211 * 0.95% 84.43% Transportation trip time (-)211 * 0.95% 85.38% # of engine restarts required (-)201 * 0.91% 86.29%

19. Attributes versus Programmatic Criteria - Matrix / Pareto PA-technology readiness at program acquisition milestone: TRL 6 + margin (+) PA-technology capability margin (performance as fraction of ultimate) (+) PA-# of other options available (+) PA-# major new technology development items (engines, airframes, TPS, etc) (-) PA-# items requiring major ground test articles and demonstration (example: new engines) (-) 1516171819 93393 93393 99993 99393 33393 750510450900300 161110207 SPST / SL-100 Space Propulsion

20. Attributes versus Programmatic Criteria - Matrix / Pareto Note: TRD - Technology Research and Development Stage PA - Program Acquisition Phase Program Acquisition Phase PA-# major new technology development items (engines, airframes, TPS, etc) (-)20% PA-technology readiness at program acquisition milestone: TRL 6 + margin (+) 16% PA-time required to establish infrastructure (schedule of R&D phase) (-)12% PA-total system DDT&E concept development and implementation cost (-)12% PA-infrastructure cost: initial system implementation (capital investment) (-)12% PA-technology capability margin (performance as fraction of ultimate) (+) 11% PA-# of other options available (+)10% PA-# items requiring major ground test articles and demonstration (example: new engines) (-) 7% Technology R & D Phase TRD-# technology breakthroughs required to develop and demonstrate (-)14% TRD-estimated time to reach TRL 6 from start of R&D (-)13% TRD-# operational effectiveness attributes addressed for improvement (+)13% TRD-Current TRL (+) 11% TRD-# full scale ground or flight demonstrations required (-) 11% TRD-cost to reach TRL -6 (-) 10% TRD-# operational effectiveness attributes previously demonstrated (+) 9% TRD-#related technology databases available (+) 7% TRD-# of new facilities required costing over $2M (-) 7% TRD-total annual funding by item at peak dollar requirements (-) 4% TRD-# multiuse applications including space transportation (+)3% SPST / SL-100 Space Propulsion

21. 05101520 PA-# items requiring major ground test articles and demonstration (example: new engines) (-) PA-# of other options available (+) PA-technology capability margin (performance as fraction of ultimate) (+) PA-time required to establish infrastructure (schedule of R&D phase) (-) PA-total system DDT&E concept development and implementation cost (-) PA-infrastructure cost: initial system implementation (capital investment) (-) PA-technology readiness at program acquisition milestone: TRL 6 + margin (+) PA-# major new technology development items (engines, airframes, TPS, etc) (-) TRD-# multiuse applications including space Transportation (+) TRD-total annual funding by item at peak dollar Requirements (-) TRD-#related technology databases available (+) TRD-# of new facilities required costing over $2M (-) TRD-# operational effectiveness attributes previously demonstrated (+) TRD-cost to reach TRL -6 (-) TRD-Current TRL (+) TRD-# full scale ground or flight demonstrations Required (-) TRD-estimated time to reach TRL 6 from start of R&D (-) TRD-# operational effectiveness attributes addressed for improvement (+) TRD-# technology breakthroughs required to develop and demonstrate (-) SCORE Attributes versus Programmatic Criteria - Matrix / Pareto SPST / SL-100 Space Propulsion

22. SpaceLiner 100 Propulsion Task Force SpaceLiner 100 Propulsion Assessment/Prioritization Process & Criteria Technology Evaluation Benefits (Technical) Attributes and Associated Design Criteria Benefits (Technical With Sense of Goodness and Normalized Weighting) Affordable / Low Life Cycle Cost Min. Cost Impact on Launch Sys. Low Recurring Cost Low Cost Sens. to Flt. Growth* Operation and Support Initial Acquisition Vehicle/System Replacement Raw % Score Weight No. 49 # of unique stages (flight and ground) (-) 483 5.3 % No. 75 # of active on-board space sys. req'd for propulsion ( - ) 454 4.9 % No. 78 On-board Propellant Storage & Management Difficulty in Space (-) 453 4.9 % No. 38 Technology readiness levels (+) 425 4.6 % No. 59 Mass Fraction required (-) 387 4.2 % No. 54 Ave. Isp on refer. trajectory (+) 310 3.4 % No. 70 # of umbs. req'd to Launch Vehicle ( - ) 276 3.0 % No. 58 # of engines (-) 274 3.0 % No. 79 Resistance to Space Environment (+) 268 2.9 % No. 82 Integral structure with propulsion sys. (+) 239 2.6 % No. 85 Transportation trip time (-) 211 2.3 % Dependable Highly Reliable Intact Vehicle Recovery Mission Success Operate on Command Robustness Design Certainty Raw % Score Weight No. 10 # of active components required to function including flight operations (-)527 5.7 % No. 87 Design Variability (-)464 5.0 % No. 14 # of different fluids in system (-)404 4.4 % No. 60 # of active engine systems required to function (-)247 2.7 % No. 48 # of modes or cycles (-)227 2.5 % No. 16 Margin, mass fraction (+)215 2.3 % No. 18 Margin, thrust level/engine chamber press (+)211 2.3 % No. 64 # of engine restarts required (-)201 2.2 %

23. SpaceLiner 100 Propulsion Task Force SpaceLiner 100 Propulsion Assessment/Prioritization Process & Criteria Technology Evaluation Benefits (Technical) Attributes and Associated Design Criteria Con’t Responsive Flexible Capacity Operable Process Verification Auto. Sys. Health Verification Auto. Sys. Corrective Action Ease of Vehicle/System Integration Maintainable Simple Launch on Demand Easily Supportable Resiliency Raw % Score Weight No. 37 # of different propulsion systems (-)582 6.3 % No. 66 System Margin (+)508 5.5 % No. 33 % of propulsion system automated (+)488 5.3 % No. 53 # of ground power systems (-)226 2.5 % Environmental Compatibility Minimum Impact on Space Environ. Minimum Effect on Atmosphere Minimum Environ. Impact all Sites Safety Vehicle Safety Personnel Safety Public Safety Equipment and Facility Safety Raw % Score Weight No. 5 # of toxic fluids (-)495 5.4 % No. 6 # of propulsion sub-systems with fault tolerance (+)398 4.3 % No. 4 Amount of energy release from unplanned reaction of propellant (-)219 2.4 % Public Support Benefit GNP Social Perception

24. SpaceLiner 100 Propulsion Task Force In-Space Propulsion Assessment/Prioritization Process & Criteria Candidate Technologies PROGRAMMATIC Assessment Criteria Program Acquisition Phase PA-# major new technology development items (engines, airframes, TPS, etc) (-) 20 % PA-technology readiness at program acquisition milestone: TRL 6 + margin (+) 16 % PA-time required to establish infrastructure (schedule of R&D phase) (-) 12 % PA-total system DDT&E concept development and implementation cost (-) 12 % PA-infrastructure cost: initial system implementation (capital investment) (-) 12 % PA-technology capability margin (performance as fraction of ultimate) (+) 11 % PA-# of other options available (+) 10 % PA-# items requiring major ground test articles & demonstration (ex: new engines) (-) 7 % Technology R & D Phase TRD-# technology breakthroughs required to develop and demonstrate (-) 14 % TRD-estimated time to reach TRL 6 from start of R&D (-) 13 % TRD-# operational effectiveness attributes addressed for improvement (+) 13 % TRD-Current TRL (+) 11 % TRD-# full scale ground or flight demonstrations required (-) 11 % TRD-cost to reach TRL –6 (-) 10 % TRD-# operational effectiveness attributes previously demonstrated (+) 9 % TRD-# related technology databases available (+) 7 % TRD-# of new facilities required costing over $2M (-) 7 % TRD-total annual funding by item at peak dollar requirements (-) 4 % TRD-# multi-use applications including space transportation (+) 3 %