Research Heaven, West Virginia 1 FY 2004 Initiative: Risk Assessment of Software Architectures Hany Ammar, Katerina Goseva-Popstojanova, Ajith Guedem, Kalaivani Appukkutty, Walid AbdelMoez, and Ahmad Hassan LANE Department of Computer Science and Electrical Engineering West Virginia University Less risk, sooner WVU UI: Risk Assessment of Software Architectures

Research Heaven, West Virginia 2 Outline  Introduction  Severity Analysis Methodology  Requirement Risk Analysis Methodology  Relevance to NASA Case studies  Conclusion

Research Heaven, West Virginia 3 Introduction Risk is a combination of: The probability of failure and the consequence of the failure (Severity) Severity analysis is conducted on Unified Modeling Language (UML) models to quantify the severity of failure Severity is estimated based on cost of failure and Hazard analysis techniques: –Functional Failure Analysis (FFA) –Failure Mode and Effects Analysis (FMEA) –Fault Tree Analysis (FTA) Requirement Risk is assessed based on: Probability of failure (Normalized Dynamic Complexity) and Severity

Research Heaven, West Virginia 4 Outline  Introduction  Severity Analysis Methodology  Requirement Risk Analysis Methodology  Relevance to NASA Case studies  Conclusion

Research Heaven, West Virginia 5 Proposed Severity Analysis Methodology INPUT: DYNAMIC UML SPECIFICATIONS –System level scenarios related to use cases, annotated with the following: – Guide words (Omission, Commission, Late, etc) –Cost of hazard –Sequence diagram that show the component/connector interactions annotated with the following: –Component states, probability that a component fails in a given failure mode, and the cost of failure in each state –Messages transmitted between components, probability that connectors fail to transmit a message, and cost of failure of each message OUTPUT: COMPONENTS/CONNECTORS and SCENARIO SEVERITY

6 FMEA 2 Proposed severity analysis methodology FTA 3 (List of scenario level Hazards) (List of Component/Connector Failure Modes) Scenario Severity Component/Connector Severity UML Specs Use Case Diagrams, System level Scenario Diagrams Sequence Diagrams Component/ Connector interactions Component/ connector Cost-Severity Graph Scenario level Cost-Severity Graph 5 Cost of Failure Graph 4 FFA 1 Scenario Severity And Components / Connectors Severity

Research Heaven, West Virginia 7 Proposed Severity Analysis Methodology 1.Identify system hazards: states of the system that can contribute to accidents and mishaps Perform FFA using system scenario diagrams as an input 2.Identify component/connector failure modes Perform FMEA using UML Sequence Diagrams as an input 3.Construct a detailed cause-and-effect model, to record how failures propagate from component/connector level to the system level FTA is used to combine the outputs from FFA and FMEA 4.Develop the Cost of Failure Graph to estimate severity of each component/connector in a given scenario, or severity level of the scenario 5.Estimate severity of component/connector and system scenario using cost of failure graph and cost-severity graph The final result is a table of component/connector severity and system scenario severity.

Research Heaven, West Virginia 8 Step 1: FFA Table Event Name Class of failure FailureEffects on SystemCost of Failure ($) Comments CommandValueA Fault in processing command routine Heart is continuously triggered but device is still monitored by physician, need immediate fix or disable. 1000The component received the command misinterpret it VSenseOmissionTimer not set correctly No pace is generated for the heart, patient could die Some component’s timers does not work well PaceCommissionPacing hardware device malfunctioning Heart is always paced while patient condition requires only pacing the heart when no pulse is detected. Heart operation is irregular Some component’s sensor failed to sense the heart.

Research Heaven, West Virginia 9 Step 2: Example (Pacemaker AVI Scenario) Sequence diagram of the AVI scenario

Research Heaven, West Virginia 10 Step 2: FMEA Table ComponentFailure ModeEffect on the system Cause of failureCost of failure mode ($) ARToOn Value Error (AR stuck in Idle state) The component will not work and there is no pace of the heart The component miss interpret signal received from CG AR stuck in Refractory State The component will stay in Refractory state and there is no pace Connector VT-AR sends a wrong message, or component AR failes to understand the message The component receive GotVSense but there is no pace (AR stuck in Waiting state) The component will stay in waiting state and there is no pace The component sensor does not work

Research Heaven, West Virginia 11 Step 3: Fault Tree Analysis (FTA) Step 1:Top event hazard identified by applying FFA Step 2: Component/Conne ctor failure modes identified through FMEA Step3: Combining the results of steps 1 and 2 to build a cause-effect model by applying FTA Commission “Pace” Fault Tree

Research Heaven, West Virginia 12 Step 4: Cost of Failure Graph for AR Component System Level hazards (Fault Trees Top Event ) Commission “Pace” “Command” Value Error P(“Command” Value Error)=0.02 P(Commission “Pace”) = 0.65 Failure Modes probabilities of AR Component Consequence (Cost) of AR Failure Modes AR “stuck in Refractory” State P(“stuck in Refractory”) = 0.05 AR Cost Of Failure Sense TimeOut Error P(“Sense TimeOut Error”) = 0.1 PaceTimeOut Error P(“PaceTimeOut Error”) = 0.03 AR “stuck in Pace” State P(“stuck in Pace”) = 0.03 AR failed to handle ToOn P(“failed to handle ToOn”) =.09 AR “ stuck in Waiting” State P(“stuck in Waiting”) = 0.05 $ 1000 (Regular Care) $ 1000 (Regular Care) $ (Intensive Care) $ (Intensive Care) $ (Intensive Care) $ (Intensive Care) Sense TimeOut Error P(“Sense TimeOut” Error) = 0.1 P(Omission “VSense") = 0.5 Omission “VSense”

Research Heaven, West Virginia 13 Step 5: Severity of components/connectors Cost-Severity Graph

Research Heaven, West Virginia 14 Step 5: Component/connector severity and scenario severity Component/connector name Severity Connector CG-AR0.50 Connector CG-VT0.50 Connector AR-VT0.94 Connector VT-AR0.95 Component CG0.05 Component AR0.96 Component VT0.95 Severity of components/connectors in the AVI scenario ScenarioSeverity AVI Scenario 0.94 AVI scenario severity

Research Heaven, West Virginia 15 Outline  Introduction  Severity Analysis Methodology  Requirement Risk Analysis Methodology  Relevance to NASA Case studies  Conclusion

16 Proposed Requirement Risk Matrix RequirementsScenariosFailure Modes FM1FM2…FMn R1S1 Rf S2 R2S3 … Rm Risk factor of scenario S1 in Failure mode FM2 Requirements are mapped to UML use case and scenarios A failure mode refers to the way in which a scenario fails to achieve its requirement Failure mode of a scenario is determined based on the severity methodology According to Dr. Martin Feather’s DDP Process, “The Requirements matrix maps the impacts of each failure mode (should it occur) on each requirement.” Impact Matrix (DDP Process)

Research Heaven, West Virginia 17 Calculating Risk Factor The dynamic complexity is calculated at the scenario level The risk factor of a particular scenario (a), in a particular failure mode (b) is: Rf ab = Normalized Dynamic Complexity ab X Severity ab Severity of a scenario in a specific failure mode is estimated using the previously discussed severity analysis methodology [Slide 6]

Research Heaven, West Virginia 18 Calculating Complexity We estimate complexity as: Complexity = CC * Msg –CC : McCabe’s cyclomatic complexity of a scenario for each failure mode –Msg: The number of messages exchanged between the components in the sequence diagram to the point where the failure mode occurs in the scenario According to Fenton et al, “The product of cyclomatic complexity (CC) and sigFF was shown to be a good predictor of fault proneness...The combined metrics appear to be better than both sigFF and Cyclomatic complexity on their own and also better than the size metric”

Research Heaven, West Virginia 19 Proposed Requirement Risk Analysis Methodology STEP 1: Map the requirements to UML sequence diagrams For each Sequence diagram STEP 2: Construct the control flow graph of the scenario from the sequence diagram STEP 3: Identify the different failure modes on the sequence diagram and the control flow graph For each failure mode STEP 4: Assess the severity of the failure mode using Function Failure Analysis (FFA) STEP 5: Determine the cyclomatic complexity of the sub- control flow graph STEP 6: For the failure mode, measure the number of messages exchanged between the components in the sequence diagram. STEP 7: Calculate the complexity of the scenario for the failure mode as: Cyclomatic complexity * Number of messages STEP 8: Calculate the Risk factor of the scenario for the failure mode as: Complexity * Severity End For each failure mode End for each sequence diagram

20 FM1 Msg = 4 FM3 Msg = 9 FM2 Msg = 7 Atrial-Ventricular Inhibit Scenario: Control Flow Graph (Obtained from Sequence Diagram) Requirement R2 of Pacemaker : Mapped to Atrial-Ventricular Inhibit Scenario Failure Modes and Msg Values marked in the sequence diagram

Research Heaven, West Virginia 21 Calculating Risk Calculating Complexity Severity index of failure modes FM1, FM2 and FM3 of AVI scenario is 0.95 Risk is calculated as the product of Complexity and Severity index Failure Modes Cyclomatic Complexity (CC) Number of Messages (Msg) Complexity (CC * Msg) Normalized Complexity FM FM FM RequirementsScenarios Failure Modes FM1FM2FM3 R2AVI

Research Heaven, West Virginia 22 Requirement Risk Matrix for Pace Maker RequirementsScenarios Failure Modes FM1FM2FM3FM4FM5 R1Programming R2Atrial-Ventricular Inhibit R3Atrial-Atrial Inhibit

Research Heaven, West Virginia 23 Outline  Introduction  Severity Analysis Methodology  Requirement Risk Analysis Methodology  Relevance to NASA Case studies  Conclusion

Research Heaven, West Virginia 24 Severity Analysis of EOS Severity Analysis Methodology is applied on NASA’s Earth Observing System (EOS) Event NameActor sends or receives this event Status From IDB (external database) to system Uplink Command From system to Space craft Preplanned emergency scenario

Research Heaven, West Virginia 25 FFA Table for EOS SEQUENCE DIAGRAM EVENTFAILUREEFFECTSCOSTSEVERITY Preplanned Emergency System retrieves the Instrument status from IDB database IDB takes longer time to return the status Emergency cannot be handled at proper time Mission Loss Catastrophic System uplinks commands to the space craft Commands don’t reach the space craft at proper time Command groups cannot be up linked in time Mission Loss Catastrophic

Research Heaven, West Virginia 26 Requirement Risk Matrix for HCS case study RequirementsScenarios Failure Modes FM1FM2FM3FM4FM5 R1 Retry both Pumps R2 Single LT Mode

Research Heaven, West Virginia 27 Outline  Introduction  Severity Analysis Methodology  Requirement Risk Analysis Methodology  Relevance to NASA Case studies  Conclusion

Research Heaven, West Virginia 28 Conclusion We have developed a methodology to assess severity of failures of components, connectors, and scenarios based on UML models We have developed a methodology for assessing requirements based risk using normalized dynamic complexity and severity of failures These methodologies were applied on NASA case studies Our future work will address the validation of risk methodology using software artifacts and data from the Metrics Data Program