West Virginia University Architectural-Level Risk Analysis for UML Dynamic Specifications Dr. Sherif M. Yacoub Hewlett-Packard Laboratories Palo Alto, CA Alaa Ibrahim, and Hany H. Ammar Department of Computer Science and Electrical Engineering West Virginia University 9 th International Conference on Software Quality Management, SQM th -20 th April, 2001 Loughborough University, Loughborough, England

West Virginia University Outline  Research Objectives  Methodology  Towards an Automated Methodology  Process  Case Study: The Pacemaker example  Conclusions

West Virginia University  Architectural-Level Risk Assessment Methodology at the early stages of development (S. Yacoub, H. Ammar. ISSRE'00, IEEE Comp. Soc., October, 2000)  Automated Environment Automated Risk Assessment Research Objectives

West Virginia University Automated Risk Assessment (continued) Architectural-Level Risk Assessment Methodology (S. Yacoub, H. Ammar. ISSRE'00, IEEE Comp. Soc., October, 2000) Utilizes: Dynamic Metrics: Component Complexity cpx i Connector Complexity cpx ij (S. Yacoub, H. Ammar, and T. Robinson. Metrics'99, November 1999) Failure Mode Effect Analysis FMEA (MIL_STD 1629A to define Component Severity svrty i Connector Severity svrty ij) Component Dependency Graphs CDG (adopted from: S. Yacoub, B. Cukic, and H. Ammar. ISSRE'99 November 1999) Defines: Heuristic Component Risk Factor hrf i = cpx i x svrty i Heuristic Connector Risk Factor hrf ij = cpx ij x svrty ij Risk Aggregation Algorithm that produces HRF appl

West Virginia University Model the architecture of the system using simulation models (UML-RT). Perform complexity analysis using simulation traces. Perform severity analysis using FMEA and simulation runs. Develop heuristic risk factors for components and connectors. Develop Components Dependency Graph for risk assessment purposes. (System/Subsystems) Aggregate the risk factors using the graph traversal algorithm. Automated Risk Assessment Architectural-Level Risk Assessment Methodology (continued) 6 Steps

West Virginia University Component Complexity Factors Connector complexity Factors CDG “ hrf i and hrf ij unidentified” Formatted Excel charts Violation Tables UML Simulation Environment Simulation Settings Simulation Log and Violation Report Analysis Tool Timing Diag. Violation Table Analyst Rose Real Time tool Text File MS Excel Processing Macro Inspection Viewing Macro UML Model Observer Sub Run Settings Analysis Tool HRF MS Excel Risk Macro Excel sheets Severity Ranking Severity Analysis (Failure/Effect analysis) Automated Risk Assessment (continued) Automated Environment CARA Tool

West Virginia University  Model the architecture of the system together with the risk logging capability using Rose RealTime.  Adjust the simulation runs in the observer as desired.  Run the simulation and get two log files containing: Component complexities. Component Execution Time. A log of all the messages exchanged. Automated Risk Assessment Automated Environment (continued) Process

West Virginia University  Process the log with Excel Risk Macro and get: Transition Probabilities. Connector complexities. CDG “where Risk Factors = Severity Factors * Complexity Factors (hrf i = cpx i x svrty i )”  Perform severity analysis using FMEA and simulation runs.  Traverse the CDG using the Excel traversal macro. Automated Risk Assessment Automated Environment Process (continued)

West Virginia University Example: Pacemaker Main Use Case Diagram

West Virginia University 1) Develop a Simulation Model Capsule Diagram Example: Pacemaker

West Virginia University Atrial statechart Case Study: Pacemaker (continued)

West Virginia University Atrial statechart Case Study: Pacemaker (continued)

A sequence diagram for the AVI scenario

A sequence diagram for the Programming scenario ProgrammerReedSwitchCoilDriverCommunication Gnome AtrialVentricular ApplyMagnet EnableComm Pulse Receiving IDLE Pulse Count++, ResetTimer BitTimeout Decode(Count) Store Bit in Byte Byte Full? enqueue(byte) Yes ByteTimeOut Waiting for Bit IDLE Count = 1, SetTimer Pulse Count =0 Receiving ByteTimeOut OR IDLE Waiting For Byte Validating IsValid? Processing ToAVI ToON ToAVI Yes HerezaByte(ACK) No HerezaByte(NAK) Waiting to Transmit Waiting to Send Next Byte

West Virginia University 2) Perform Complexity Analysis A Transition between Composite States in a component’s Statechart Operational Complexity of a component using the scenario profile and its complexity per scenario.

West Virginia University 2) Perform Complexity Analysis (cont’d) A) Quantify Component Complexity Factors using dynamic complexity metrics.

West Virginia University 2) Perform Complexity Analysis (cont’d) Export Object Coupling (EOC) EOC with scenario profiles OQFS with scenario profiles the export coupling for component C i with respect to component C j, is the percentage of the number of messages sent from C i to C j with respect to the total number of messages exchanged during the execution of the scenario x

West Virginia University 2) Perform Complexity Analysis (cont’d) B) Quantify Connector Complexity Factors using dynamic coupling metrics.

West Virginia University 3) Perform Severity Analysis  In performing severity analysis, each potential failure mode is ranked according to the consequences of that failure mode.  Steps: Identifying Failure Modes  Failure modes of individual components. (Functional faults and state-based faults)  Failure modes of individual connectors. (Interface fault analysis)

West Virginia University 3) Perform Severity Analysis (cont’d)  Steps (cont’d): Conducting Effect Analysis  Inject the fault.  Simulate the faulty model.  Monitor output and compare to expected output.  Identify the effect of the fault. Rank Severity  Identify category: Minor, Marginal, Critical, or Catastrophic.  Assign severity index to each component i as (svrty i ), which takes a value of 0.25, 0.50, 0.75, and 0.95

West Virginia University  Worst case severity found for the RS, CD, CG, VT, and AR are Minor(0.25), Minor(0.25), Marginal(0.50), Catastrophic(0.95) and Catastrophic (0.95), respectively FMEA table for the Pacemaker components

West Virginia University FMEA table for the Pacemaker connectors

West Virginia University 4) Develop Risk Factors hrf i = cpx i x svrty i where: 0 <= cpx i <= 1, is the normalized complexity level (dynamic complexity for components or dynamic coupling for connectors), and 0<= svrty i < 1, is the severity level for the architecture element. Risk Factors for the components in the example

West Virginia University 4) Develop Risk Factors (cont’d) Comparison between risk factors based on static and dynamic metrics Risk Factors for the connectors in the pacemaker example

West Virginia University 5) Constructing the CDG

West Virginia University 6) Risk Aggregation Algorithm  The algorithm expands all branches of the CDG starting from the start node.  The breadth expansions of the graph represent logical "OR" paths. translated as the summation of aggregated risk factors weighted by the transition probability along each path.  The depth of each path represents the sequential execution of components: is given by the aggregate: HRF = 1 -  i (1- hrf i )

West Virginia University Risk Aggregation Algorithm Procedure AssessRisk Parameters consumes CDG, AE appl,(average execution time for the application) produces Risk appl Initialization: R appl = R temp = 1 (temporary variables for (1-RiskFactor) ) Time = 0 Algorithm push tuple, Time, R temp while Stack not EMPTY do pop, Time, R temp if Time > AE appl or C i = t;(terminating node) R appl += R temp ;(an OR path) else   children(C i ) push (, Time += EC i, R temp = R temp *(1-hrf i )*(1-hrf ij )*PT ij ) ( AND path) end end while Risk appl = 1- R appl end Procedure AssessRisk

West Virginia University Risk Aggregation Algorithm  The algorithm can be used for System-level Risk Assessment  The risk of the pacemaker that is found to be ~ 0.9 Subsystem-level Risk Comparison  Complex systems are composed of many subsystems.  The algorithm can be used to obtain a risk factor for a subsystem using risk factors of its individual components.  Compare risk factors of individual subsystems. Sensitivity Analysis  Sensitivity to Uncertainties in Component Risk Factors  Sensitivity to Uncertainties in Connector Risk Factors

West Virginia University Sensitivity Analysis The Pacemaker risk factor as function of connector risk factors (one at a time) The Pacemaker risk factor as function of component risk factors (one at a time)

West Virginia University Benefits  The approach helps in: Deciding which components in the architecture require more development resources. Deciding which connectors in the architecture are of highest risk. A high risk connector indicates that the interfaces between the corresponding components and the messaging protocol should be carefully designed. Studying how uncertainties in component risk factors affect the overall risk value of the system. Studying how uncertainties in connector risk factors affect the overall risk value of the system.

West Virginia University Conclusion : Benefits  The methodology is applicable early at the architectural level.  The methodology is based on dynamic metrics. We use dynamic metrics to account for the fact that a fault in a frequently executed component will frequently manifest itself into a failure.  The methodology is based on simulation of architecture models. Simulation helps in: Performing FMEA procedures. Calculating the CDG parameters such as probability of transitions. Obtaining dynamic metrics.

West Virginia University Conclusion : Issues  Using ordinal scale for measuring severity.  Effect of uncertainties in the scenario probabilities and the estimated average execution times.  Scalability issues, applying the methodology to a larger case study.  Methodology is limited to systems with statechart and sequence diagram specifications.

Questions...

West Virginia University Main Use Case Diagram