1. Introduction to Risk Assessment in Engineering: With Application to Heat Shield Reliability Modeling Presented by: Austin Howard University of Idaho Mechanical Engineering Dept. Idaho Space Grant Consortium

2. 2 Austin Howard

3. 3 Outline  Introduction  Failure Mode Effect Analysis  Fault Trees  Event Trees  Obtaining Component Reliability  Monte Carlo Method  Case Study: Heat Shield Reliability Modeling  Summary

4. 4 Austin Howard Purpose of This Talk  Describe importance of risk assessment  Introduction to key tools, processes, and concepts related to risk analysis  Provide context with case study based on experiences at summer internship at NASA Ames 2006  Note: Risk assessment is its own discipline and therefore it is outside the scope of this talk to show you how to create/evaluate risk models

5. 5 Austin Howard Definition: Risk  Risk:  “The combination of the frequency, or probability, of occurrence and the consequence of a specified hazardous event” -www.bees.unsw.edu.au/ohs/definitions.html  One of many ways to calculate risk:  Risk=(Probability of failure)x(Severity of the Consequence)

6. 6 Austin Howard Risk  Risk is also a board game:

7. 7 Austin Howard Risk vs. Unreliability  Risk is not the same as Unreliability  Reliability: Probability that a device will function without failure over a specified period of time or amount of usage  Reliability is one of the (but not the only) factors that contributes to system risk  Reliability analysis is often used interchangeably with risk, but they are two different concepts  Engineers often present reliability statistics rather than risk values due to difficulty of measuring and comparing consequence severity

8. 8 Austin Howard Risk vs. Safety  Judging Risk  is a quantitative activity grounded by testing, and physical modeling  Judging Safety  is a qualitative, political activity  You must have a safety standard to judge system risk against otherwise risk is a relatively meaningless value in decision making and design assessment

9. 9 Austin Howard Deterministic vs Non-Deterministic  Deterministic model-model behaves predictably  In other words, for a constant input, you will always get the same output  Non-deterministic model-model with one or more choice points where different continuations are possible  In other words for a constant input, you will not always get the same output  Requires input from one or more: user, global variables, hardware timer, random numbers, stored data…

10. 10 Austin Howard Purpose of Risk Assessment  Purpose of Risk Assessment: Answering and effectively communicating the following questions/considerations: Haimes, Yacov Y. Risk Modeling, Assessment, and Management. Hoboken, NJ, USA: John Wiley & Sons, Incorporated, 2005. p 23. http://site.ebrary.com/lib/uidaho/Doc?id=10114200&ppg=47

11. 11 Austin Howard  Reputation  Customer Satisfaction/Safety  Warranty Costs  Repeat Business  Cost Analysis  Customer Requirements  Competitive Advantage Importance of Risk Analysis

12. 12 Austin Howard Cont…  Reduce long term cost http://klabs.org/DEI/References/design_guidelines/analysis_series/1314msfc.pdf

13. 13 Austin Howard Process

14. 14 Austin Howard Outline  Introduction  Failure Mode Effect Analysis (DFMEA)  Fault Trees  Event Trees  Obtaining Component Reliability  Monte Carlo Method  Case Study: Heat Shield Reliability Modeling  Summary

15. 15 Austin Howard Failure Mode Effect Analysis (FMEA)  Other wise known as:  Failure Mode Effect Criticality Analysis (FMECA)  Design Failure Mode Effect Analysis (DFMEA)  Process Failure Mode Effect Analysis (PFMEA)  Purpose  Define and guide a logical design process  Identify, quantify, and reduce design risk  Provide a traceable document for design and development  Justify design activities  Provide a means for continuous product improvement

16. 16 Austin Howard Cont…  Combines Possible Failure:  Severity (rate 1-10)  Occurrence (rate 1-10)  Detect-ability (rate 1-10)  Product of the parameters is called the RPN, this value describes the overall risk of each failure mechanism  High RPN numbers = high risks  Focus on these failure mechanisms first in risk mitigation process

17. 17 Austin Howard FMEA Process http://www.qualitytrainingportal.com/resources/fmea/fmea_process.htm

18. 18 Austin Howard Example: FMEA

19. 19 Austin Howard Outline  Introduction  Failure Mode Effect Analysis (DFMEA)  Fault Trees  Event Trees  Obtaining Component Reliability  Monte Carlo Method  Case Study: Heat Shield Reliability Modeling  Summary

20. 20 Austin Howard Fault Trees  At the top of a fault tree is a failure  Under the tree are all the possible faults that could lead to the top failure  Fault trees are used for viewing a system and the interactions between faults and possible paths to a failure  Fault trees can be built with software and combined with probabilities to produce reliability estimates

21. 21 Austin Howard Cont…  Paths from bottom to top of tree are termed cutsets, the shortest cutset is the minimum cutset  Symbols used: Haimes, Yacov Y. Risk Modeling, Assessment, and Management. Hoboken, NJ, USA: John Wiley & Sons, Incorporated, 2005. p 530. http://site.ebrary.com/lib/uidaho/Doc ?id=10114200&ppg=554

22. 22 Austin Howard Example: Fault Tree http://safety.transportation.org/htmlguides/implement/ProcAppJ.htm

23. 23 Austin Howard Outline  Introduction  Failure Mode Effect Analysis (DFMEA)  Fault Trees  Event Trees  Obtaining Component Reliability  Monte Carlo Method  Case Study: Heat Shield Reliability Modeling  Summary

24. 24 Austin Howard Event Trees  Goal of event tree  to determine the probability of an event based on the outcomes of each event in the chronological sequence of events leading up to it  By analyzing all possible outcomes using event tree analysis, you can determine the percentage of outcomes which lead to the desired result  Event trees can be built with software to produce reliability estimates

25. 25 Austin Howard Example: Event Trees http://www.ece.cmu.edu/~koopman/des_s99/safety_critical/

26. 26 Austin Howard Outline  Introduction  Failure Mode Effect Analysis (DFMEA)  Fault Trees  Event Trees  Obtaining Component Reliability  Monte Carlo Method  Case Study: Heat Shield Reliability Modeling  Summary

27. 27 Austin Howard Testing  Advantages  Can illuminate overlooked failure mechanisms  Some situations cannot be modeled accurately with current physical understanding  Turbulence  Limitations  Expensive  Time consuming  Need lots of data to be meaningful

28. 28 Austin Howard How Modeling Produces Unreliability Load Probability Curve Design Probability Curve Area=Probability of failure Mean Load Mean Design Spec Design Margin

29. 29 Austin Howard Modeling  Advantages  Can be relatively inexpensive/fast  Limitations  Easy to make incorrect assumptions/mistakes  Some situations are difficult/impossible to model accurately

30. 30 Austin Howard System/Sub-System Reliability Series Reliability A B C R tot = R A * R B * R C Full Redundancy A B C R tot = 1- (1- R A ) * (1 - R B ) * (1 - R C )

31. 31 Austin Howard Outline  Introduction  Failure Mode Effect Analysis (DFMEA)  Fault Trees  Event Trees  Obtaining Component Reliability  Monte Carlo Method  Case Study: Heat Shield Reliability Modeling  Summary

32. 32 Austin Howard The Essence of Monte Carlo  Monte Carlo: Method of modeling involving inputs from random or pseudo random numbers  Output produced has the similar characteristics to that of data collected from an experiment*  Similar scattering of data  The more “runs” of the model, the more pronounced the trends are *If input is correct - your model output is only as good as the information you put into the model

33. 33 Austin Howard What Monte Carlo Looks Like Vose, David; Quantitative Risk Analysis:A guide to Monte Carlo simulation modeling; 1996

34. 34 Austin Howard Outline  Introduction  Failure Mode Effect Analysis (DFMEA)  Fault Trees  Event Trees  Obtaining Component Reliability  Monte Carlo Method  Case Study: Heat Shield Reliability Modeling  Summary

35. 35 Austin Howard Heat Shields 101  Kinetic Energy: +Potential Energy:  Thermal Energy (hot)  Entry velocities between 7km/s(LEO)-11km/s (Lunar return),  Altitude ~400 km (+ for lunar return)  Blunt body advantage  Shuttle vs Apollo

36. 36 Austin Howard Cont… Apollo Shuttle Before After

37. 37 Austin Howard Case Study Objectives  Risk Assessment Objectives For Orion Heat Shield:  Obtain an estimation of the overall system reliability  Identify components/events most likely to cause failure  Identify sub-systems that may be too conservative  Determine sensitivity of design/modeling/testing/environmental parameters on system reliability  Determine where resources should be allocated in order to reduce risk most efficiently

38. 38 Austin Howard Failure Modes  TPS Failure Modes  Burnthrough of heat shield material  Crack  Damage  De-bonding  Hot spots  Flowthrough  Bondline overheat  Excessive conduction  Radiation absorption  System interface failure  e.g. electromagnetic interference, landing system interference

39. 39 Austin Howard The Software Used  SAFE – Space Architecture Failure Evaluation  Code in development at NASA Ames  Monte Carlo Simulation method  Input  Assembly architecture  Nominal reliabilities of components and events  Consequences of failure  Mission outline (events and segments)  The software generates hundreds or thousands of semi- random repetitions of the given scenario  The output  Histograms and mission summaries that engineers can use to determine when the system is likely to fail, what will cause failure, and how often system failures are likely to occur…

40. 40 Austin Howard Simple Example

41. 41 Austin Howard Risk Interaction Example  Micro-Meteoroid and Orbital Debris (MMOD)  Risk of significant sized particles hitting heat shield with significant velocity to cause damage  Risk of the MMOD damage causing/contributing to TPS failure

42. 42 Austin Howard Another Example  Environment modeling  Accurately predicting entry environment  Recession modeling based on predicted environment  Material selection/Thickness design based on recession modeling

43. 43 Austin Howard Organizing the Risks

44. 44 Austin Howard Visualizing Risk Interaction

45. 45 Austin Howard Calculating Risk Values

46. 46 Austin Howard The Model

47. 47 Austin Howard Predicting Reliability  Historical records  Apollo  Shuttle  Others  Physics based simulation tools  Testing  Ground Tests  Flight Tests

48. 48 Austin Howard Results of Summer Work  Reliability model:  Incorporates over 90 potential TPS risks  Each risk can fail in either a benign or catastrophic manner  Multiple benign failures have the ability to contribute to a catastrophic failure  All pre-entry factors influence risks during entry and landing phases  Risk Analysis Document  Outline for detailed sub-system interaction  Can be used to track changes and understand model  Can be used to help understand risk dependence on material choice and other design factors

49. 49 Austin Howard Outline  Introduction  Failure Mode Effect Analysis (DFMEA)  Fault Trees  Event Trees  Obtaining Component Reliability  Monte Carlo Method  Case Study: Heat Shield Reliability Modeling  Summary

50. 50 Austin Howard Summary  Risk analysis is a large topic that describes an entire discipline of engineering  Risk analysis is an iterative process  If used correctly, can save money, and lives!  Can aid in decision making process, justify actions  There are lots of tools available for engineers

51. 51 Austin Howard Cont…  The output of a risk assessment is only as good as the input  The engineer must have plenty of test data or a sound model before a valid risk model can be produced  Model output is meaningless without bounds on the solution

52. 52 Austin Howard Questions?