Model-Based Risk Assessment Walid AbdelMoez LANE Department of Computer Science and Electrical Engineering West Virginia University Morgantown, WV 26506-6109 This work is funded in part by grants from the NSF ITR Program and from the NASA Office of Safety and Mission Assurance (OSMA) SARP program managed through the NASA IV&V Facility, Fairmont, West Virginia. Tuesday November 28

Outline Introduction The Problem Research Objectives Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28

Introduction A sound architecture is the key to build a software system with high quality attributes. Risk assessment is an essential part in the management of software development. Performing it in the early phases of software development can enhance allocation of resources within the software lifecycle. Risk assessment provides useful means for identifying potentially troublesome software components that require careful development and allocation of more testing effort. Tuesday November 28

Outline Introduction The Problem Research Objectives Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28

Problem Statement In pursue of our research effort, we identify the following problems: How can we assess risk associated with the non-functional attributes of reliability and maintainability of a system based on quantitative means How to define a practical reliability-based risk assessment methodology that captures the dependencies between the system functionalities and among components Tuesday November 28

Outline Introduction The Problem Research Objectives Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28

Research Objectives The objectives of this research are: To develop reliability-based risk assessment methodology that considers functional dependencies To account for error propagation among the system components and to generalize the assumption of failure occurrence independence To define maintainability-based risk To develop a methodology for assessing maintainability-based risk for the system components based on the characteristics of the component and its interactions with other components. To automate the maintainability-based risk methodology Tuesday November 28

Outline Introduction The Problem Research Objectives Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28

Research Contribution In reliability-based risk assessment We develop a methodology to capture the dependencies among the system components and apply it on number of case studies. We generalize our methodology to accommodate relationships that model extensions and commonality within the UML use case model. It should be emphasized that, to the best of our knowledge, this is the first work which addresses relationships between use cases in assessing reliability-based risk. We apply The methodology on a real industrial case study which is a large command and control system used in a mission-critical application. Tuesday November 28

Research Contribution In maintainability-based risk assessment We introduce and define maintainability-based risk assessment for software architecture. We propose a generic methodology for estimating the maintainability-based risk when considering different types of maintenance: corrective, adaptive or perfective. We apply the proposed methodology on several case studies considering alternative maintenance types. We use Non-Homogeneous Poisson Process to get the maintainability-based risk estimate as a function of time when introducing new features in the system and/or due to adaptive maintenance. We automate the steps for the estimation methodology. Tuesday November 28

Outline Introduction The Problem Research Objectives Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28

Reliability-Based Risk Assessment Considering Dependencies among System Components In [Goseva-Popstojanova+2003], we estimate component’s reliability-based risk factor as a product of the dynamic complexity and the severity level of that component for a certain scenario To get an estimate for the system-level reliability-based risk factor for the component, we can use static cyclomatic complexity as a predictor for the probability of error occurrence in that component and multiply it by the system-level severity Such estimation assumes independence of error occurrences in the components of the system Tuesday November 28

Considering Error Propagation Probabilities in Assessing Components Reliability-Based Risk To generalize this assumption, we use error propagation probabilities among the components of the system in reliability-based risk assessment. The reliability-based risk factor of a system component Ci will be Where RRF = [rrfi] the Component Reliability-based Risk Factor IEP = [iepi] the Initial Error Probability EP = [epij] the Error Propagation Probabilities matrix CSv = [csvi] the Component Severity Tuesday November 28

Error Propagation Probability The Error Propagation Probability EP(A,B) = Prob([B](x)[B](x’) | xx’), where [B] denotes the function of component B, and x is an element of the connector X from A to B. S X A B x x` OR We interpret [B] to capture all the effects of executing component B, including the effect on the state of B and the effect on any outputs produced by B. Tuesday November 28

Estimating Error Propagation Analytically, the error propagation probability can be expressed in terms of the probabilities of the individual A-to-B messages and states, via the following formula: EP(AB) = Tuesday November 28

Related Work Voas ; Cigital labs 1997, analyzes error propagations between COTS components and presents an automated tool to simulate error propagation Michael et al.; RST corp. 1997, present an empirical study of data-state error propagation behavior. They argue that at a given location either all data state errors injected tend to propagate to the output or none does Hiller et al.; Chalmers Univ. 2001, analyze error propagation conceptually, introducing the concept of error permeability and discussing means to measure it Tuesday November 28

Case Study: Command and Control System The example we use to illustrate our work is a large command and control system that is used in a life-critical, mission-critical application This system was modeled using the Rational Rose Realtime CASE tool. It is a Computer Software Configuration Item (CSCI) that provides the following functions: Facilitating Communication, Control, Cautions Controlling a Secondary Electrical Power System Environmental Control Tuesday November 28

Case Study: Command and Control System Structure Diagram Tuesday November 28

Case Study: Command and Control System Messages Protocol and State Diagrams Tuesday November 28

The Framework of Experimental Error Propagation Analysis D. Nassar contributed to the experimental Framework Design Tuesday November 28

Experimental Validation of Error Propagation Probabilities Tuesday November 28

Correlation between analytical and empirical error propagation Tuesday November 28

Cyclomatic Complexity Components Reliability-Based Risk Pace Maker - Initial Error Probability Cyclomatic Complexity Initial Error Probabilities Stress CC is used to estimate IEP Tuesday November 28

A. Hassan assessed the severity level of the pace maker case study Pace Maker Analytical Error Propagation Probabilities Components Severity A. Hassan assessed the severity level of the pace maker case study Tuesday November 28

Comparing components reliability-based risk factors for pace maker case study Tuesday November 28

Comparing CM1 case study reliability-based risk factors Tuesday November 28

Experimental Validation of Components Reliability-Based Risk Assessment Comparing command and control system Componentsreliability risk factors Tuesday November 28

Outline Introduction The Problem Research Objectives Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28

Reliability-Based Risk Assessment Methodology with independent Use Cases For each use case For each scenario For each component Measure dynamic complexity Assign severity based on FMEA and hazard analysis Calculate risk factor For each connector Measure dynamic coupling Assign severity based on FEMA and hazard analysis Construct Markov model Calculate scenario level risk factor Calculate system level risk [TSE 2003] K. Goseva-Popstojanova , A. Hassan, A. Guedem, W. Abdelmoez, D. Nassar, H. Ammar, A. Mili, “Architectural-Level Risk Analysis using UML” IEEE Transaction on Software Engineering, Vol.29, No.10, October 2003 Tuesday November 28

Related Work In [Singh+ 2001], Singh et al. proposed an approach for reliability analysis of component-based systems based on UML. In this work authors assumed independent use cases. Houmb et al. presented the CORAS UML profile for risk assessment in [Houmb+ 2002] and demonstrated its use on an e-commerce system assuming independent use cases. In [Hawkins+ 2002] Hawkins et al. addressed the hazard and safety analysis of object-oriented systems. Although the approach proposed in [Hawkins+ 2002] starts from the use cases, relationships among them were not considered. Polynomial of widely used code level metrics such as Halstead measures and McCabe’s cyclomatic complexity Linear model based on a minimal set of design level software metrics to predict Software Maintainability Index Tuesday November 28

Relationships Between Use-Cases Non-Primitive Use-Case Terminal Use-Case Primitive Use-Case Tuesday November 28

Plan Itinerary sequence diagram Tuesday November 28

Purchase Ticket sequence diagram Tuesday November 28

Reliability-Based Risk Assessment with Functional Dependencies Aggregate Tuesday November 28

Algorithm for Reliability-Based Risk Assessment with Use –Case Relationships Determine terminal, primitive, and non-primitive use cases and their dependent scenarios For each primitive use case Determine the risk factor of each scenario Determine the use case risk factor by taking the max risk scenario For each terminal use case For each dependent scenario Recursively determine the risk factor of the scenario Determine the system risk using a weighted sum of the risk factors of the terminal use cases An implementation for this risk aggregation algorithm considering one scenario per use case was developed by Ajith and Venu. Tuesday November 28

Case Study: Command and Control System Tuesday November 28

Risk factors of the primitive use cases Tuesday November 28

The risk factor of Monitoring and Mode_Setting use cases Tuesday November 28

Distribution of the overall system risk factor Tuesday November 28

Outline Introduction The Problem Research Objectives Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28

Maintainability-based Risk Maintainability-based Risk is defined as: Probability of performing maintenance task * Cost of performing this task Risk caused by low system maintainability can be assessed by the following: Software components require frequent maintenance tasks High cost of maintenance effort - the required maintenance is growing out of control (changes propagate to many components). Loss of the system by aging - the system becomes out-dated and it is not worthy to maintain Tuesday November 28

Related Work [P.Oman 1994] introduced the Maintainability Index (MI) measure [Muthanna 2000] used design level metrics to statistically estimate the maintainability of software systems [Menzies+ 2000] assessed the maintainability of software systems by developing and simulating a model of system reachability [Steward 1981] presented Design Structure Matrices (DSMs) [Bohner+ 1996] summarized change impact analysis [Briand+ 1999B] investigated a probabilistic decision models based on coupling measurement to support impact analysis [Cohen+ 2000] introduced C-FAR system to trace and predict change propagation [Briand+ 2003] proposed a UML model-based approach to impact analysis that can allow early decision-making and a change planning process [Hassan+ 2004] suggested a number of heuristics to anticipate change propagation in software systems. [Tsantalis+ 2005] proposed a probabilistic approach to estimate the change proneness of an object-oriented design. Polynomial of widely used code level metrics such as Halstead measures and McCabe’s cyclomatic complexity Linear model based on a minimal set of design level software metrics to predict Software Maintainability Index Tuesday November 28

Outline Introduction The Problem Research Objectives Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28

Maintainability-based Risk Estimation Methodology Tuesday November 28

Maintenance change propagation Outgoing maintenance Incoming maintenance Tuesday November 28

Estimating Change Propagation V1 C1 C2 = 1 V11 V12 Change in Provided Service V13 Change . V13 Required Services = 0 Change Propagation Probabilities matrix CP=[cpij ] cpij is the probability that a change in Ci due to corrective/ perfective maintenance requires a change in Cj while maintaining the overall function of a system S cpij = P([Cj]  [Cj'] | [Ci]  [Ci'] ^ [S] = [S'] ) cpij is estimated by cpij = Tuesday November 28

Validation of Change propagation Probabilities Tuesday November 28

Estimating Size of Change V11 V1 V12 V13 Change Change in Provided Service M1 M2 M3 … M7 Receiving Component methods Size of change SC=[scij ] scij is defined as the ratio between the number of affected methods of the receiving component caused by the changes in the interface elements of the providing components and the total number of methods in the receiving component scij is estimated by scij = Tuesday November 28

Case Study : CM1 [NASA IV&V Facility – MDP Project] The UML Model of CM1 was Reversed Engineered by WVU (Nathan, Tom and Rajesh, Summer 2004) based on the CM1 software design specification Tuesday November 28

CM1: Change Propagation and Size of Change Matrices Tuesday November 28

Maintainability-based Risk Estimation Methodology in Context of Adaptive Maintenance SW Architecture Adaptive Maintenance Change reports (1) Estimate components Initial Change Probability (ICP): Normalized Frequency (2) Estimate Change Propagation (CP) probabilities (3) Estimate Size of Change (SC) ICP=[icpi] SC=[sci/j] CP=[cpi/j] (4) Estimate component risk factor Tuesday November 28

CM1 Maintainability-Based Risk in Adaptive Maintenance Context Tuesday November 28

Requirements Maturity Index SW Architecture Maintainability-based Risk Estimation Methodology in Context of Adaptive Maintenance-Requirements Changes Adaptive Maintenance Requirements Maturity Index SW Architecture (1) Estimate components Initial Change Probability (ICP): Tracing Requirements to Components (2) Estimate Change Propagation (CP) probabilities (3) Estimate Size of Change (SC) ICP=[icpi] SC=[sci/j] CP=[cpi/j] (4) Estimate component risk factor Tuesday November 28

CM1 UML Model Use Case Diagram Since we do not have real modifications data for CM1 requirements, we assume hypothetical maintenance scenario to demonstrate our methodology Transfer use case has 4 different scenarios We assume that there were only three scenarios. The required changes is to add an alternative scenario to the Transfer use case Tuesday November 28

CM1 UML Model Sequence Diagram “Transfer 3” We use the added scenario Transfer 3, to estimate the UCMI for Transfer use case. Where UT = the function point size of the use-case in the current release UC = the function point size of the change in the use case Tuesday November 28

Initial Change Probabilities for CM1 UCMI is proportional to component stability. Thus, we can map the use case stability into components stability. ICP of the system components icpi = DCoi * FanOuti * (1 – UCMI) Where DCoi is the normalized dynamic complexity of Ci in UCi, FanOuti the Fan out coupling of Ci in UCi Tuesday November 28

Maintainability-Based Risk of CM1 Tuesday November 28

Corrective Maintenance Maintainability-based Risk Estimation Methodology in Context of Corrective Maintenance SW Architecture Corrective Maintenance Error reports (1) Estimate components Initial Change Probability (ICP): Normalized Frequency (2) Estimate Change Propagation (CP) probabilities (3) Estimate Size of Change (SC) ICP=[icpi] SC=[sci/j] CP=[cpi/j] (4) Estimate component risk factor Tuesday November 28

Putting Maintainability-Based Risk to Work: Ranking Corrective Maintenance Tasks To order the corrective maintenance tasks for a certain project according to the importance of each task, we propose using the maintainability-based risk of the components that need to be fixed. the severity-level of failures that may be manifested from the errors in these components. Tuesday November 28

Prioritizing Corrective Maintenance Tasks for CM1 The CM1 case study has 98 error reports of components bugs. Assuming that these errors have not been yet fixed, we calculate the frequency of errors occurrences in the components of the system. Then, we estimate the initial change probability ICP of the components by normalizing the frequency of error occurrences by the total number of error reports. For maintenance tasks of components with high severity-levels, they should be fixed regardless of their corresponding maintainability-based risk because of the consequences of such potential failures on the system. On the other hand for maintenance tasks of components that have low severity-levels, we should examine the components maintainability-based risk. So, maintenance tasks of low severity-level and high maintainability-based risk should be avoided or delayed in the maintenance plan Critical Major Cat Cat. Minor Severity Level TMALI TIS SSI SCUI 1553 ICUI EDAC DPA DCX DCI CCM BIT Components Tuesday November 28

Worst Case Maintainability-Based Risk (Initial change probabilities set to one) Command and Control CM1 PaceMaker Tuesday November 28

Using Non-Homogeneous Poisson Process to Estimate Maintainability-Based Risk In [Burch+ 1997] and [Gefen+ 1996], the authors observed the non-homogeneous nature of maintenance request arrivals. In [Tan+ 2005], Tan et al. modeled the random arrival of maintenance requests using non-homogeneous Poisson process with time varying mean rate. Unlike previous maintainability-based risk models, NHPP provides us with an estimate which is a function of time The estimation procedure is more flexible as it relies on a statistical model to estimate ICP at different points of time. As the system gains more stability, we are able to acquire better estimates for the parameters of the statistical model. Thus, we can have a better predication of the maintainability-based risk Tuesday November 28

cs1 = {initial system features } Using NHPP to Estimate Maintainability-Based Risk (Adaptive/Perfective Maintenance) MP = {cs1 , cs2 , cs3 } cs1 = {initial system features } cs2 = { HeartBeat}, cs3 = { Transferc } Tuesday November 28

Using Non-Homogeneous Poisson Process to Estimate Maintainability-Based Risk Tuesday November 28

Initial Change Probabilities for the Components of CM1 Case Study Maintenance Request Arrival Generated using NHPP with Matlab simulation ICP obtained by Normalizing using max sum Tuesday November 28

Components Maintainability-Based Risk for CM1 Components Tuesday November 28

Outline Introduction The Problem Research Objectives Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28

Software Architecture Risk Assessment (SARA) Tool - Structural Description Tuesday November 28

Software Architecture Risk Assessment (SARA) Tool - Functional Description Tuesday November 28

Software Architecture Risk Assessment (SARA) Tool – Snap Shots Wang, Venu and Khader contributed in developing SARA tool Tuesday November 28

Outline Introduction The Problem Research Objectives Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28

Conclusion In this dissertation, we present risk assessment estimation methodologies based on architectural models of the software system Reliability-based risk assessment Used Error Propagation Probability in the assessment to account for the dependency among the components Generalized the reliability-based risk assessment to account for functional dependencies Maintainability-based risk assessment Introduced and defined maintainability-based risk Developed a general methodology for estimating the maintainability-based risk Automated the estimation methodology. Tuesday November 28

Outline Introduction The Problem Research Objectives Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28

Future Work The future work has the following aspects To apply the generalized reliability-based risk assessment methodology To validate the maintainability-based risk assessment estimation methodology, To apply methodology on other case studies/system architectures such as product lines To mine history repositories of the open-source projects To execute additional controlled experiments and perform pre/post analysis To extend the Software Architecture Risk Assessment (SARA) tool Tuesday November 28

Validation Prospects for Maintainability-Based Risk Estimation An experiment design should incorporate monitoring several maintenance projects with different size Bottom-up approach: ICP, CP, SC System components changes history Compare with domain expert subjective assessment Modular approach of our proposed maintainability-based risk assessment allow flexible improvements . Pilot exploratory studies CM1 case study from NASA Metrics Data Program MDP Controlled experiment on three case studies: Job Application, Colleague states and Borg Tuesday November 28

Imported error proneness for command and control system case study Tuesday November 28

Analytical error proneness of the components in steps Tuesday November 28

<<Extend>> relationship Tuesday November 28

<<Include>> relationship Tuesday November 28

Reliability-Based Risk Assessment with Functional Dependencies Tuesday November 28

DTMC of Retry_Both_Pumps use case Tuesday November 28

DTMC of the Monitoring use case Tuesday November 28

Multi-Step Change Propagation (i, j) := CP(Ci, Cj), i, j = 1,…,N (p) = (i, i1) ( i1, i2) … ( in-1, j) k(i, j)  n(i, j)  n(i, j) We use multi-step change propagation to predicting change propagation patterns. In [Clarkson+ 2000], Clarkson et al present a three-tiered classification of changes as follows Ripple of changes Wave of changes Avalanches of changes Ripple of change: the introduced changes will result in an acceptable behavior to be observed for the maintenance process. Changes are controlled and limited. Wave of change: the introduced changes will still result in an acceptable behavior to be observed for the maintenance process. Although there are many changes, they are under control. Avalanches of changes: the introduced changes will result in an unacceptable behavior to be observed in the maintenance process. There are many changes and they are uncontrolled. Tuesday November 28

Parameterization of the categorization of the change behavior  ( 0 <  < 1 ) is the propagation area significance,  ( 0 <  < 1 ) is the ripple threshold, and  ( 0 <  < 1 ) is the avalanche threshold Tuesday November 28

Graphical representation of the critical change propagation Tuesday November 28

Mn(Ci) of the components through multi-step change propagation Tuesday November 28

Using Change Propagation Probabilities to Assess Quality Attributes of Software Architectures The following methodology has been applied: Prepare a pair of architectures for the same application. One of the architectures is designed using design patterns while the other has no patterns. Apply the CP metric on both architectures. Apply other object-oriented metrics on both architectures. Analyze and compare the results. The architecture the employs software patterns should have a better quality in terms of extensibility and maintainability. Tuesday November 28

Job Application Case Study Tuesday November 28

Colleague States Case Study Tuesday November 28

Coupling Between Objects (CBO) Tuesday November 28

Response For a Class (RFC) Tuesday November 28

Message Passing Coupling (MPC) Tuesday November 28

Motivations Many types of risk are ushered when software systems undergo maintenance Some are similar to those we face while developing new software systems, e.g. project risk and usability risk. Maintainers often interact with complex and difficult to comprehend systems, which introduces types of risk unique to software maintenance Tuesday November 28

Motivations According to [Pigoski 1996], 60%-80% of the system budget is spent on maintenance Enhancements (perfective/ adaptive maintenance) account for 78%-83% of the maintenance effort Tuesday November 28

Improve the maintainability of the software architecture Goals and Objectives Improve the maintainability of the software architecture Identify risky components in terms of maintainability Use maintainability risk to manage system maintenance process Tuesday November 28

Maintainability-based Risk Estimation Methodology Tuesday November 28

Maintainability Based Risk in Perfective Maintenance Context Software system which is hard to maintain usually is plagued with bad smells Fowler and Beck presented a list of bad smells which can be identified at code level Some of the bad smells can be identified at architectural level Scale Index No of set() and get()/NOM NAM, WMC NUMPAR CP Matrix SC Matrix Tuesday November 28

Maintainability Based Risk in Perfective Maintenance Context Tuesday November 28