1. National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology Quantification of Margins and Uncertainties for Model-Informed Flight System Qualification Dr. Lee D. Peterson Principal Engineer Mechanical Systems Division (35) Presented at the NASA Thermal and Fluids Analysis Workshop Hampton, VA 16 August 2011 System Performance Observations Simulation Uncertainty Project Requirement Test Data Margin © 2011 California Institute of Technology. Government sponsorship acknowledged.

2. 2 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology QMU (Quantification of Margins and Uncertainty): Managing flight margin and risk QMU background and overview Tools for high fidelity multiphysics models and simulations Tools for test selection and design Potential for QMU application on flight missions JPL is pursuing QMU technology to enable rigorous certification of models and simulations for extrapolation to poorly-testable flight conditions

3. 3 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology “All models are wrong, but some are useful.” Prof. George E.P. Box, U. of Wisconsin “Models answer questions to support decisions.” Dr. Greg Agnes, NASA JPL

4. 4 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology Surface Water Ocean Topography (SWOT) relies on a large, deployable antenna boom difficult to test Self-shadowing in LEO on a large, flexible structure Micron-scale dimensional error budget allocations Comparable magnitude for effects neglected by conventional tools Nonlinearity, thermal snap Ground test validation will need models to extrapolate to flight Model and simulation errors may drive design when comparable to subsystem error budgets.

5. 5 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology A spacecraft “Digital Twin” would predict flight performance under difficult-to-test conditions Model V&V Component Ground Tests Subsystem Ground Tests Flight Digital Twin Flight System Ground Tests Extrapolation Demonstration Model V&V OSSE Science Measurements QMU provides a formalism for establishing credibility of a Digital Twin for extrapolation to flight.

6. 6 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology QMU seeks to quantify margins and risk from both simulation and test uncertainties System Performance Observations Simulation Uncertainty Project Requirement Test Data Margin QMU can supplement traditional margin rules when experience is sparse.

7. 7 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology Simulation Uncertainty Tools Advanced multi-physics FEM software (Sierra) Uncertainty quantification (DAKOTA) Model CM and workflow software Uncertainty Quantification Convergent (high density) meshes Numerical and parametric errors Random uncertainty Non-random (epistemic) uncertainty Simulation Credibility Guidelines and Requirements NASA Modeling and Simulation Standard (7009) ASME Guidelines (10-2006) Project Rules Model Heritage Phenomena Identification and Ranking Table (PIRT) Predictive Capability Maturity Matrix (PCMM) Inverse Bayesian test planning tool Model Validation Bayesian UQ updates Test selection and design Margin and Risk Assessment (Based on idea from K. Alvin, SNLA) Analysis and practice are both key to establishing a QMU modeling, simulation and test campaign

8. 8 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology Rigorous Model Verification and Validation is the backbone of QMU Large activities over the past 10-15 years (especially in DOE labs) Recent NASA Standard for Models and Simulation (NASA-STD-7009, released July 2008) In response to the Columbia Accident Investigation Board (CAIB) report Embodies much of the modern model V&V language NASA, AIAA, ASME, DOE and DOD Guidelines and Recommended Practices

9. 9 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology Model “verification” is not model “validation” Computerized Model Conceptual Model Reality Model Qualification Model Verification Model Validation [AIAA, 1999] Programming Analysis Computer Simulation

10. 10 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology A validated model can be credible for extrapolation. system or environmental parameter physical and geometrical complexity Application Domain Validation Domain system or environmental parameter physical and geometrical complexity Application Domain Validation Domain system or environmental parameter physical and geometrical complexity Application Domain Validation Domain inference [Oberkampf et al, 2004] Extrapolation depends on whether the model gets the right answer … for the right reasons.

11. 11 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology What are key technologies for QMU of space systems? Realism Credibility Speed CAD-like meshes Nonlinearities Imperfections Randomness Couplings & interactions Rigorous model V&V Design-CAD-Simulation traceability Model parameterization and sensitivity High performance computing Efficient model iteration and sampling algorithms Templated model construction and CM Convergent (high density) meshes Examples UQ Results from SWOT Reflectarry Panel

12. 12 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology DOE labs faced similar challenges in their quest for model-based qualification without full system test DOE advanced modeling, simulation and model V&V technology over two decades Multi-$B investment in hardware, software and test methodologies Pervasive use of multiple physical domains, nonlinearity Specialized to DOE weapon performance Rigorous model V&V practices Complex Systems, Components, and Physics Diverse Applications (ref: A. Ratzel, SNL, 5/19/09) JPL has collaborated with Sandia to pilot the application of their tools and methods for spacecraft QMU.

13. 13 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology Initial Piloted Application of Sierra to SWOT Concept Integrated Sierra Structural-Thermal Model Single-Panel Integrated Sierra Simulation ThermalStructural

14. 14 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology Illustrative SWOT Panel UQ Study Results QMU enables systematic assessments off simulation uncertainty The Universe is Not Gaussian!

15. 15 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology But there is more to QMU than just “uncertainty quantification” Need to establish “simulation credibility” via application of rigorous process Model Credibility Assessment and Planning Tools Phenomena Identification and Ranking Table (PIRT) Inverse Bayesian Analysis Model V&V Component Ground Tests Subsystem Ground Tests Flight Digital Twin Flight System Ground Tests Extrapolation Demonstration Model V&V Flight-critical models must be developed using rigor similar to flight-critical hardware

16. 16 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology Phenomena Identification and Ranking Table is a tool for prioritizing model and test requirements

17. 17 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology New “Inverse Bayesian” tools can specify validation tests and analyses requirements Traditional QMU analysis answers the “forward” uncertainty problem “Given test and simulation uncertainty, what is the prediction uncertainty?” Inverse Bayesian approach answers the “backwards” problem “Given a required model prediction uncertainty, what are the requirements on component, subsystem and system tests?”

18. 18 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology QMU Inverse Bayesian tool can rank tests by impact on system prediction uncertainty Simple applications have shown potential to plan a test campaign that target model error improvement From Sankararaman, McLemore, Mahadevan, Bradford, Peterson, “Prioritization of System Integration Tests using Bayes Networks” Submitted to AIAA Journal. Model V&V Component Ground Tests Subsystem Ground Tests Flight Digital Twin Flight System Ground Tests Extrapolation Demonstration Model V&V QMU can reduce both cost and risk by targeting the most important tests

19. 19 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology Challenges remain in advancing QMU tools from current state to flight practice Projects are (rightly) reluctant to adopt models and simulations for extrapolating to flight Key is to integrate QMU of the model into flight system integration flow But will a hierarchical model V&V campaign succeed? Can model errors be reliably allocated to manage extrapolation risk? Large scale demonstration of QMU would enable its application to space missions. Model V&V Component Ground Tests Subsystem Ground Tests Flight Digital Twin Flight System Ground Tests Extrapolation Demonstration Model V&V

20. 20 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology System Performance Observations Simulation Uncertainty Project Requirement Test Data Margin Summary JPL has been piloting the use of QMU of space systems Key new technologies, tools and practices specific to spacecraft applications Potential to enable missions not currently practical Model V&V Component Ground Tests Subsystem Ground Tests Flight Digital Twin Flight System Ground Tests Extrapolation Demonstration Model V&V