Download presentation
Presentation is loading. Please wait.
1
Model-Based Risk Assessment
Walid AbdelMoez LANE Department of Computer Science and Electrical Engineering West Virginia University Morgantown, WV This work is funded in part by grants from the NSF ITR Program and from the NASA Office of Safety and Mission Assurance (OSMA) SARP program managed through the NASA IV&V Facility, Fairmont, West Virginia. Tuesday November 28
2
Outline Introduction The Problem Research Objectives
Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28
3
Introduction A sound architecture is the key to build a software system with high quality attributes. Risk assessment is an essential part in the management of software development. Performing it in the early phases of software development can enhance allocation of resources within the software lifecycle. Risk assessment provides useful means for identifying potentially troublesome software components that require careful development and allocation of more testing effort. Tuesday November 28
4
Outline Introduction The Problem Research Objectives
Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28
5
Problem Statement In pursue of our research effort, we identify the following problems: How can we assess risk associated with the non-functional attributes of reliability and maintainability of a system based on quantitative means How to define a practical reliability-based risk assessment methodology that captures the dependencies between the system functionalities and among components Tuesday November 28
6
Outline Introduction The Problem Research Objectives
Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28
7
Research Objectives The objectives of this research are:
To develop reliability-based risk assessment methodology that considers functional dependencies To account for error propagation among the system components and to generalize the assumption of failure occurrence independence To define maintainability-based risk To develop a methodology for assessing maintainability-based risk for the system components based on the characteristics of the component and its interactions with other components. To automate the maintainability-based risk methodology Tuesday November 28
8
Outline Introduction The Problem Research Objectives
Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28
9
Research Contribution
In reliability-based risk assessment We develop a methodology to capture the dependencies among the system components and apply it on number of case studies. We generalize our methodology to accommodate relationships that model extensions and commonality within the UML use case model. It should be emphasized that, to the best of our knowledge, this is the first work which addresses relationships between use cases in assessing reliability-based risk. We apply The methodology on a real industrial case study which is a large command and control system used in a mission-critical application. Tuesday November 28
10
Research Contribution
In maintainability-based risk assessment We introduce and define maintainability-based risk assessment for software architecture. We propose a generic methodology for estimating the maintainability-based risk when considering different types of maintenance: corrective, adaptive or perfective. We apply the proposed methodology on several case studies considering alternative maintenance types. We use Non-Homogeneous Poisson Process to get the maintainability-based risk estimate as a function of time when introducing new features in the system and/or due to adaptive maintenance. We automate the steps for the estimation methodology. Tuesday November 28
11
Outline Introduction The Problem Research Objectives
Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28
12
Reliability-Based Risk Assessment Considering Dependencies among System Components
In [Goseva-Popstojanova+2003], we estimate component’s reliability-based risk factor as a product of the dynamic complexity and the severity level of that component for a certain scenario To get an estimate for the system-level reliability-based risk factor for the component, we can use static cyclomatic complexity as a predictor for the probability of error occurrence in that component and multiply it by the system-level severity Such estimation assumes independence of error occurrences in the components of the system Tuesday November 28
13
Considering Error Propagation Probabilities in Assessing Components Reliability-Based Risk
To generalize this assumption, we use error propagation probabilities among the components of the system in reliability-based risk assessment. The reliability-based risk factor of a system component Ci will be Where RRF = [rrfi] the Component Reliability-based Risk Factor IEP = [iepi] the Initial Error Probability EP = [epij] the Error Propagation Probabilities matrix CSv = [csvi] the Component Severity Tuesday November 28
14
Error Propagation Probability
The Error Propagation Probability EP(A,B) = Prob([B](x)[B](x’) | xx’), where [B] denotes the function of component B, and x is an element of the connector X from A to B. S X A B x x` OR We interpret [B] to capture all the effects of executing component B, including the effect on the state of B and the effect on any outputs produced by B. Tuesday November 28
15
Estimating Error Propagation
Analytically, the error propagation probability can be expressed in terms of the probabilities of the individual A-to-B messages and states, via the following formula: EP(AB) = Tuesday November 28
16
Related Work Voas ; Cigital labs 1997, analyzes error propagations between COTS components and presents an automated tool to simulate error propagation Michael et al.; RST corp. 1997, present an empirical study of data-state error propagation behavior. They argue that at a given location either all data state errors injected tend to propagate to the output or none does Hiller et al.; Chalmers Univ. 2001, analyze error propagation conceptually, introducing the concept of error permeability and discussing means to measure it Tuesday November 28
17
Case Study: Command and Control System
The example we use to illustrate our work is a large command and control system that is used in a life-critical, mission-critical application This system was modeled using the Rational Rose Realtime CASE tool. It is a Computer Software Configuration Item (CSCI) that provides the following functions: Facilitating Communication, Control, Cautions Controlling a Secondary Electrical Power System Environmental Control Tuesday November 28
18
Case Study: Command and Control System Structure Diagram
Tuesday November 28
19
Case Study: Command and Control System Messages Protocol and State Diagrams
Tuesday November 28
20
The Framework of Experimental Error Propagation Analysis
D. Nassar contributed to the experimental Framework Design Tuesday November 28
21
Experimental Validation of Error Propagation Probabilities
Tuesday November 28
22
Correlation between analytical and empirical error propagation
Tuesday November 28
23
Cyclomatic Complexity
Components Reliability-Based Risk Pace Maker - Initial Error Probability Cyclomatic Complexity Initial Error Probabilities Stress CC is used to estimate IEP Tuesday November 28
24
A. Hassan assessed the severity level of the pace maker case study
Pace Maker Analytical Error Propagation Probabilities Components Severity A. Hassan assessed the severity level of the pace maker case study Tuesday November 28
25
Comparing components reliability-based risk factors for pace maker case study
Tuesday November 28
26
Comparing CM1 case study reliability-based risk factors
Tuesday November 28
27
Experimental Validation of Components Reliability-Based Risk Assessment Comparing command and control system Componentsreliability risk factors Tuesday November 28
28
Outline Introduction The Problem Research Objectives
Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28
29
Reliability-Based Risk Assessment Methodology with independent Use Cases
For each use case For each scenario For each component Measure dynamic complexity Assign severity based on FMEA and hazard analysis Calculate risk factor For each connector Measure dynamic coupling Assign severity based on FEMA and hazard analysis Construct Markov model Calculate scenario level risk factor Calculate system level risk [TSE 2003] K. Goseva-Popstojanova , A. Hassan, A. Guedem, W. Abdelmoez, D. Nassar, H. Ammar, A. Mili, “Architectural-Level Risk Analysis using UML” IEEE Transaction on Software Engineering, Vol.29, No.10, October 2003 Tuesday November 28
30
Related Work In [Singh+ 2001], Singh et al. proposed an approach for reliability analysis of component-based systems based on UML. In this work authors assumed independent use cases. Houmb et al. presented the CORAS UML profile for risk assessment in [Houmb+ 2002] and demonstrated its use on an e-commerce system assuming independent use cases. In [Hawkins+ 2002] Hawkins et al. addressed the hazard and safety analysis of object-oriented systems. Although the approach proposed in [Hawkins+ 2002] starts from the use cases, relationships among them were not considered. Polynomial of widely used code level metrics such as Halstead measures and McCabe’s cyclomatic complexity Linear model based on a minimal set of design level software metrics to predict Software Maintainability Index Tuesday November 28
31
Relationships Between Use-Cases
Non-Primitive Use-Case Terminal Use-Case Primitive Use-Case Tuesday November 28
32
Plan Itinerary sequence diagram
Tuesday November 28
33
Purchase Ticket sequence diagram
Tuesday November 28
34
Reliability-Based Risk Assessment with Functional Dependencies
Aggregate Tuesday November 28
35
Algorithm for Reliability-Based Risk Assessment with Use –Case Relationships
Determine terminal, primitive, and non-primitive use cases and their dependent scenarios For each primitive use case Determine the risk factor of each scenario Determine the use case risk factor by taking the max risk scenario For each terminal use case For each dependent scenario Recursively determine the risk factor of the scenario Determine the system risk using a weighted sum of the risk factors of the terminal use cases An implementation for this risk aggregation algorithm considering one scenario per use case was developed by Ajith and Venu. Tuesday November 28
36
Case Study: Command and Control System
Tuesday November 28
37
Risk factors of the primitive use cases
Tuesday November 28
38
The risk factor of Monitoring and Mode_Setting use cases
Tuesday November 28
39
Distribution of the overall system risk factor
Tuesday November 28
40
Outline Introduction The Problem Research Objectives
Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28
41
Maintainability-based Risk
Maintainability-based Risk is defined as: Probability of performing maintenance task * Cost of performing this task Risk caused by low system maintainability can be assessed by the following: Software components require frequent maintenance tasks High cost of maintenance effort - the required maintenance is growing out of control (changes propagate to many components). Loss of the system by aging - the system becomes out-dated and it is not worthy to maintain Tuesday November 28
42
Related Work [P.Oman 1994] introduced the Maintainability Index (MI) measure [Muthanna 2000] used design level metrics to statistically estimate the maintainability of software systems [Menzies+ 2000] assessed the maintainability of software systems by developing and simulating a model of system reachability [Steward 1981] presented Design Structure Matrices (DSMs) [Bohner+ 1996] summarized change impact analysis [Briand+ 1999B] investigated a probabilistic decision models based on coupling measurement to support impact analysis [Cohen+ 2000] introduced C-FAR system to trace and predict change propagation [Briand+ 2003] proposed a UML model-based approach to impact analysis that can allow early decision-making and a change planning process [Hassan+ 2004] suggested a number of heuristics to anticipate change propagation in software systems. [Tsantalis+ 2005] proposed a probabilistic approach to estimate the change proneness of an object-oriented design. Polynomial of widely used code level metrics such as Halstead measures and McCabe’s cyclomatic complexity Linear model based on a minimal set of design level software metrics to predict Software Maintainability Index Tuesday November 28
43
Outline Introduction The Problem Research Objectives
Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28
44
Maintainability-based Risk Estimation Methodology
Tuesday November 28
45
Maintenance change propagation
Outgoing maintenance Incoming maintenance Tuesday November 28
46
Estimating Change Propagation
V1 C1 C2 = 1 V11 V12 Change in Provided Service V13 Change . V13 Required Services = 0 Change Propagation Probabilities matrix CP=[cpij ] cpij is the probability that a change in Ci due to corrective/ perfective maintenance requires a change in Cj while maintaining the overall function of a system S cpij = P([Cj] [Cj'] | [Ci] [Ci'] ^ [S] = [S'] ) cpij is estimated by cpij = Tuesday November 28
47
Validation of Change propagation Probabilities
Tuesday November 28
48
Estimating Size of Change
V11 V1 V12 V13 Change Change in Provided Service M1 M2 M3 … M7 Receiving Component methods Size of change SC=[scij ] scij is defined as the ratio between the number of affected methods of the receiving component caused by the changes in the interface elements of the providing components and the total number of methods in the receiving component scij is estimated by scij = Tuesday November 28
49
Case Study : CM1 [NASA IV&V Facility – MDP Project]
The UML Model of CM1 was Reversed Engineered by WVU (Nathan, Tom and Rajesh, Summer 2004) based on the CM1 software design specification Tuesday November 28
50
CM1: Change Propagation and Size of Change Matrices
Tuesday November 28
51
Maintainability-based Risk Estimation Methodology in Context of Adaptive Maintenance
SW Architecture Adaptive Maintenance Change reports (1) Estimate components Initial Change Probability (ICP): Normalized Frequency (2) Estimate Change Propagation (CP) probabilities (3) Estimate Size of Change (SC) ICP=[icpi] SC=[sci/j] CP=[cpi/j] (4) Estimate component risk factor Tuesday November 28
52
CM1 Maintainability-Based Risk in Adaptive Maintenance Context
Tuesday November 28
53
Requirements Maturity Index SW Architecture
Maintainability-based Risk Estimation Methodology in Context of Adaptive Maintenance-Requirements Changes Adaptive Maintenance Requirements Maturity Index SW Architecture (1) Estimate components Initial Change Probability (ICP): Tracing Requirements to Components (2) Estimate Change Propagation (CP) probabilities (3) Estimate Size of Change (SC) ICP=[icpi] SC=[sci/j] CP=[cpi/j] (4) Estimate component risk factor Tuesday November 28
54
CM1 UML Model Use Case Diagram
Since we do not have real modifications data for CM1 requirements, we assume hypothetical maintenance scenario to demonstrate our methodology Transfer use case has 4 different scenarios We assume that there were only three scenarios. The required changes is to add an alternative scenario to the Transfer use case Tuesday November 28
55
CM1 UML Model Sequence Diagram “Transfer 3”
We use the added scenario Transfer 3, to estimate the UCMI for Transfer use case. Where UT = the function point size of the use-case in the current release UC = the function point size of the change in the use case Tuesday November 28
56
Initial Change Probabilities for CM1
UCMI is proportional to component stability. Thus, we can map the use case stability into components stability. ICP of the system components icpi = DCoi * FanOuti * (1 – UCMI) Where DCoi is the normalized dynamic complexity of Ci in UCi, FanOuti the Fan out coupling of Ci in UCi Tuesday November 28
57
Maintainability-Based Risk of CM1
Tuesday November 28
58
Corrective Maintenance
Maintainability-based Risk Estimation Methodology in Context of Corrective Maintenance SW Architecture Corrective Maintenance Error reports (1) Estimate components Initial Change Probability (ICP): Normalized Frequency (2) Estimate Change Propagation (CP) probabilities (3) Estimate Size of Change (SC) ICP=[icpi] SC=[sci/j] CP=[cpi/j] (4) Estimate component risk factor Tuesday November 28
59
Putting Maintainability-Based Risk to Work: Ranking Corrective Maintenance Tasks
To order the corrective maintenance tasks for a certain project according to the importance of each task, we propose using the maintainability-based risk of the components that need to be fixed. the severity-level of failures that may be manifested from the errors in these components. Tuesday November 28
60
Prioritizing Corrective Maintenance Tasks for CM1
The CM1 case study has 98 error reports of components bugs. Assuming that these errors have not been yet fixed, we calculate the frequency of errors occurrences in the components of the system. Then, we estimate the initial change probability ICP of the components by normalizing the frequency of error occurrences by the total number of error reports. For maintenance tasks of components with high severity-levels, they should be fixed regardless of their corresponding maintainability-based risk because of the consequences of such potential failures on the system. On the other hand for maintenance tasks of components that have low severity-levels, we should examine the components maintainability-based risk. So, maintenance tasks of low severity-level and high maintainability-based risk should be avoided or delayed in the maintenance plan Critical Major Cat Cat. Minor Severity Level TMALI TIS SSI SCUI 1553 ICUI EDAC DPA DCX DCI CCM BIT Components Tuesday November 28
61
Worst Case Maintainability-Based Risk (Initial change probabilities set to one)
Command and Control CM1 PaceMaker Tuesday November 28
62
Using Non-Homogeneous Poisson Process to Estimate Maintainability-Based Risk
In [Burch+ 1997] and [Gefen+ 1996], the authors observed the non-homogeneous nature of maintenance request arrivals. In [Tan+ 2005], Tan et al. modeled the random arrival of maintenance requests using non-homogeneous Poisson process with time varying mean rate. Unlike previous maintainability-based risk models, NHPP provides us with an estimate which is a function of time The estimation procedure is more flexible as it relies on a statistical model to estimate ICP at different points of time. As the system gains more stability, we are able to acquire better estimates for the parameters of the statistical model. Thus, we can have a better predication of the maintainability-based risk Tuesday November 28
63
cs1 = {initial system features }
Using NHPP to Estimate Maintainability-Based Risk (Adaptive/Perfective Maintenance) MP = {cs1 , cs2 , cs3 } cs1 = {initial system features } cs2 = { HeartBeat}, cs3 = { Transferc } Tuesday November 28
64
Using Non-Homogeneous Poisson Process to Estimate Maintainability-Based Risk
Tuesday November 28
65
Initial Change Probabilities for the Components of CM1 Case Study
Maintenance Request Arrival Generated using NHPP with Matlab simulation ICP obtained by Normalizing using max sum Tuesday November 28
66
Components Maintainability-Based Risk for CM1 Components
Tuesday November 28
67
Outline Introduction The Problem Research Objectives
Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28
68
Software Architecture Risk Assessment (SARA) Tool - Structural Description
Tuesday November 28
69
Software Architecture Risk Assessment (SARA) Tool - Functional Description
Tuesday November 28
70
Software Architecture Risk Assessment (SARA) Tool – Snap Shots
Wang, Venu and Khader contributed in developing SARA tool Tuesday November 28
71
Outline Introduction The Problem Research Objectives
Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28
72
Conclusion In this dissertation, we present risk assessment estimation methodologies based on architectural models of the software system Reliability-based risk assessment Used Error Propagation Probability in the assessment to account for the dependency among the components Generalized the reliability-based risk assessment to account for functional dependencies Maintainability-based risk assessment Introduced and defined maintainability-based risk Developed a general methodology for estimating the maintainability-based risk Automated the estimation methodology. Tuesday November 28
73
Outline Introduction The Problem Research Objectives
Research Contribution Reliability-Based Risk Assessment Considering Dependencies among System Components Reliability-Based Risk Assessment with Functional Dependencies Maintainability-Based Risk Assessment Maintainability-Based Risk Assessment Estimation Methodology Software Architecture Risk Assessment (SARA) Tool Conclusion Future Work Tuesday November 28
74
Future Work The future work has the following aspects
To apply the generalized reliability-based risk assessment methodology To validate the maintainability-based risk assessment estimation methodology, To apply methodology on other case studies/system architectures such as product lines To mine history repositories of the open-source projects To execute additional controlled experiments and perform pre/post analysis To extend the Software Architecture Risk Assessment (SARA) tool Tuesday November 28
75
Validation Prospects for Maintainability-Based Risk Estimation
An experiment design should incorporate monitoring several maintenance projects with different size Bottom-up approach: ICP, CP, SC System components changes history Compare with domain expert subjective assessment Modular approach of our proposed maintainability-based risk assessment allow flexible improvements . Pilot exploratory studies CM1 case study from NASA Metrics Data Program MDP Controlled experiment on three case studies: Job Application, Colleague states and Borg Tuesday November 28
76
Imported error proneness for command and control system case study
Tuesday November 28
77
Analytical error proneness of the components in steps
Tuesday November 28
78
<<Extend>> relationship
Tuesday November 28
79
<<Include>> relationship
Tuesday November 28
80
Reliability-Based Risk Assessment with Functional Dependencies
Tuesday November 28
81
DTMC of Retry_Both_Pumps use case
Tuesday November 28
82
DTMC of the Monitoring use case
Tuesday November 28
83
Multi-Step Change Propagation
(i, j) := CP(Ci, Cj), i, j = 1,…,N (p) = (i, i1) ( i1, i2) … ( in-1, j) k(i, j) n(i, j) n(i, j) We use multi-step change propagation to predicting change propagation patterns. In [Clarkson+ 2000], Clarkson et al present a three-tiered classification of changes as follows Ripple of changes Wave of changes Avalanches of changes Ripple of change: the introduced changes will result in an acceptable behavior to be observed for the maintenance process. Changes are controlled and limited. Wave of change: the introduced changes will still result in an acceptable behavior to be observed for the maintenance process. Although there are many changes, they are under control. Avalanches of changes: the introduced changes will result in an unacceptable behavior to be observed in the maintenance process. There are many changes and they are uncontrolled. Tuesday November 28
84
Parameterization of the categorization of the change behavior
( 0 < < 1 ) is the propagation area significance, ( 0 < < 1 ) is the ripple threshold, and ( 0 < < 1 ) is the avalanche threshold Tuesday November 28
85
Graphical representation of the critical change propagation
Tuesday November 28
86
Mn(Ci) of the components through multi-step change propagation
Tuesday November 28
87
Using Change Propagation Probabilities to Assess Quality Attributes of Software Architectures
The following methodology has been applied: Prepare a pair of architectures for the same application. One of the architectures is designed using design patterns while the other has no patterns. Apply the CP metric on both architectures. Apply other object-oriented metrics on both architectures. Analyze and compare the results. The architecture the employs software patterns should have a better quality in terms of extensibility and maintainability. Tuesday November 28
88
Job Application Case Study
Tuesday November 28
89
Colleague States Case Study
Tuesday November 28
90
Coupling Between Objects (CBO)
Tuesday November 28
91
Response For a Class (RFC)
Tuesday November 28
92
Message Passing Coupling (MPC)
Tuesday November 28
93
Motivations Many types of risk are ushered when software systems undergo maintenance Some are similar to those we face while developing new software systems, e.g. project risk and usability risk. Maintainers often interact with complex and difficult to comprehend systems, which introduces types of risk unique to software maintenance Tuesday November 28
94
Motivations According to [Pigoski 1996], 60%-80% of the system budget is spent on maintenance Enhancements (perfective/ adaptive maintenance) account for 78%-83% of the maintenance effort Tuesday November 28
95
Improve the maintainability of the software architecture
Goals and Objectives Improve the maintainability of the software architecture Identify risky components in terms of maintainability Use maintainability risk to manage system maintenance process Tuesday November 28
96
Maintainability-based Risk Estimation Methodology
Tuesday November 28
97
Maintainability Based Risk in Perfective Maintenance Context
Software system which is hard to maintain usually is plagued with bad smells Fowler and Beck presented a list of bad smells which can be identified at code level Some of the bad smells can be identified at architectural level Scale Index No of set() and get()/NOM NAM, WMC NUMPAR CP Matrix SC Matrix Tuesday November 28
98
Maintainability Based Risk in Perfective Maintenance Context
Tuesday November 28
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.