1. Software Risk Assessment based on UML models
Requirements-based and Performance-based Risk Assessment
By: Kalaivani Appukkutty, MS thesis, WVU
This work is funded in part by grants to West Virginia University NASA Office of Safety and Mission Assurance (OSMA) Software Assurance Research Program (SARP) managed through the NASA Independent Verification and Validation (IV&V) Facility in Fairmont, West Virginia.

2. Overview Introduction Research Contribution
Requirements-based Risk Analysis
Performance-based Risk Analysis
Conclusions and Future work

3. Introduction Risk is the possibility of suffering loss.
Software risk is defined as a combination of two factors: probability of failure of software and the severity of the failure
Risk Assessment is the process of identifying the potential risk and their impact
It identifies potentially troublesome software components that require careful development and allocation of more testing efforts
It can be performed at various phases of life software cycle.

4. Requirements-based Risk Assessment
Research Objectives
Requirements-based Risk Assessment
To identify the requirements that are at high risk early in the software life cycle
To assess requirements-based risk from UML models
To develop a methodology that can be automated
Performance-based Risk Assessment
To assess performance based risk taking into account the severity of performance failures
To develop a methodology based on the UML profile for performance, schedulability and time.

5. Requirements-based Risk
Software risk is the product of :
The probability of malfunctioning (failure) of the software
The consequence of malfunctioning (severity)
The probability of failure is proportional to the complexity of the system.
Requirements-based risk is assessed for a requirement in a specific failure mode

6. Requirements-based Risk
Requirements are mapped to UML use case and scenarios
A failure mode of a scenario refers to the way in which a scenario fails to achieve its requirement
The risk factor of a particular scenario (a), in a particular failure mode (b) is:
Rfab = Normalized Dynamic Complexityab X Severityab

7. Calculating Complexity
We estimate complexity as:
Complexity = CC * Msg
CC : McCabe’s cyclomatic complexity of a scenario for each failure mode
Msg: The number of messages exchanged between the components in the sequence diagram to the point where the failure mode occurs in the scenario
According to Fenton et al, “The product of cyclomatic complexity (CC) and sigFF was shown to be a good predictor of fault proneness ... The combined metrics appear to be better than both sigFF and Cyclomatic complexity on their own and also better than the size metric”

8. Requirement Risk Matrix
Requirements R1 R2 … Rm
Scenarios S1 S2 S3
Failure Modes
FM1
FM2 …
FMn Rf
Risk factor of scenario S1 in Failure mode FM2

9. Requirement-based Risk Assessment Methodology
STEP 1: Map the requirements to UML sequence diagrams
For each Sequence diagram
STEP 2: Construct the control flow graph of the scenario from the sequence diagram
STEP 3: Identify the different failure modes on the sequence diagram and the control flow graph
For each failure mode
STEP 4: Assess the severity of the failure mode using Function Failure Analysis (FFA)
STEP 5: Determine the cyclomatic complexity of the sub-control flow graph
STEP 6: For the failure mode, measure the number of messages exchanged between the components in the sequence diagram.
STEP 7: Calculate the complexity of the scenario for the failure mode as:
Cyclomatic complexity * Number of messages
STEP 8: Calculate the Risk factor of the scenario for the failure mode as: Complexity * Severity
End For each failure mode
End for each sequence diagram

10. Cardiac Pacemaker system
A cardiac pacemaker is an implanted device that assists cardiac functions by pacing the heart, when the heart pulse fails.

11. Atrial-Ventricular Inhibit Scenario: Control Flow Graph
Requirement R2 of Pacemaker : Mapped to Atrial-Ventricular Inhibit (AVI) Scenario
Atrial-Ventricular Inhibit Scenario: Control Flow Graph
Failure Modes and Msg Values marked in the sequence diagram
FM1
Msg = 4
FM3
Msg = 9
FM2
Msg = 7

12. Calculating Risk Complexity Severity
Failure
Modes
Cyclomatic
Complexity (CC)
Number of
Messages (Msg)
Complexity
(CC * Msg)
Normalized
FM1 3 4 12
0.111
FM2 6 7 42
0.389
FM3 9 54
0.500
Severity
Severity is assessed using Function Failure Analysis (FFA)
The severity factor corresponding to the failure of requirement R2 in failure modes FM1, FM2 and FM3 is 0.95 (catastrophic category)
Risk
Requirements
Scenarios
Failure Modes
FM1
FM2
FM3 R2
AVI
0.11
0.37
0.475

13. Requirement Risk Matrix (Cardiac Pacemaker)
Requirements
Scenarios
Failure Modes
FM1
FM2
FM3
FM4
FM5 R1
AVI
0.11
0.37
0.475 R2
AAI
0.151
0.283
0.566 R3
Programming
0.017
0.233

14. Requirement Risk Chart (Cardiac Pacemaker)

15. Performance-based Risk Analysis
Considers the non-functional performance requirements of the system, which can sometimes be more important than the functional-requirements
Highlights the performance critical components and scenarios at an architectural-level
Uses the software and the system execution models which are well defined ways of modeling the performance aspects of a system

16. Performance-based Risk Analysis Methodology
For each scenario in each use case and for each scenario is that use case
STEP 1 - Assign demand vector to each action/interaction in sequence diagram and build a Software Execution Model for that scenario
STEP 2 - Add hardware platform characteristics on the deployment diagram and conduct stand-alone analysis
STEP 3 - Devise the workload parameters; build a System Execution Model and conduct contention-based analysis and estimate probability of performance failure
STEP 4 - Conduct severity analysis and estimate severity of performance failure for the scenario
STEP 5 - Estimate the performance risk of the scenario; Identify high-risk components

17. Step 1
Each “action” in the sequence diagram such as number of cpu instructions, amount of disk access and the data passed from one component to another
A software execution model is built which considers the maximum service demand path of the scenario execution

18. Step 2
The annotated Deployment Diagram is used for identifying the hardware platform characteristics on which the software system is deployed
The system execution model is built by mapping the software execution model to the hardware component service times
A standalone analysis is conducted by considering total service time of the scenario against the response time objective of that scenario

19. Step 3
Contention-based analysis is performed by for objective workload and response time of the scenario
is the total demand of the scenario and the is the maximum demand in that scenario
Z1: Failure probability (Z1) = 0
Z2: Failure probability (Z2) = (UB-OBJ) / (UB-LB)
Z3: Failure probability (Z3) = 1
UB stands for Upper Bound, LB for Lower Bound and OBJ stands for Performance Objective

20. Step 4 and 5
Step 4
Severity analysis is conducted to obtain the severity factor of the performance failures of the scenarios. We use a framework for conducting severity analysis of the performance failures of the scenario.
The framework uses
Function Failure Analysis(FFA)
Failure Mode Effect Analysis(FMEA)
Fault Tree Analysis(FFT)
Step 5
Identification of performance-critical components - we first estimate the overall residence time of each component in a given scenario and then normalize it with the response time of the scenario
A high-risk component is said to be the one with a higher normalized service time in a scenario or across many scenarios

21. The methodology is illustrated with a typical e-commerce case study

22. Sequence Diagram Annotations and Execution Graph
The sequence diagram shows the PlaceRequisition scenario and the various performance annotations

23. Demand Vectors
The sequence diagram is converted to a software execution graph and the execution path with the longest service demand is considered for estimating the
PLACE-REQ
RA4
CA4
CI3
CA5
RS1
DOA1
OS1
SPLIT

24. Service Time and Demand Vectors
The deployment diagram is annotated with hardware characteristics
The completion time of the Place Requisition scenario is equal to seconds

25. Estimating Failure Probability
The performance objectives are :
A response time objective of 1.5 seconds
A workload of 15 customers
The results are:
UB= sec
LB=1.35 sec and OBJ=1.5 sec
Probability of performance failure =
The severity assigned for this scenario is catastrophic (0.95)
The risk factor of this scenario is

26. Performance-based Risk: E-commerce case study

27. Residence Time of Components
The components with taller bars(high-risk) and colored red(high-severity) are the ones that are more performance-critical and require further investigation and testing

28. Conclusion and Future Work
The requirements based risk assessment methodology presented, fills the gap between completely domain-expert dependant methods that are applied at the requirement analysis stage and formal analytical methods that do not assess risk at the requirements level
The performance-based risk assessment methodology gives a systematic approach to use UML profile for performance to assess risk based on failure probability and severity

29. Conclusion and Future Work
Future work will focus on developing tool support to assist the analyst with the steps of the methodology.
We also intend to apply the methodologies and supporting tools on large scale applications.

30. Publications
K. Appukkutty , Hany H. Ammar, K.Goseva-Popstojanova, “Software Requirement Risk Assessment Using UML”, Accepted in The 3rd ACS/IEEE International Conference on Computer Systems and Applications, Jan 2005.
K. Appukkutty , Hany H. Ammar, K.Goseva-Popstojanova, “Early Risk Assessment of Software Systems”, Accepted in The 3rd International Conference on Computer Science, Software Engineering, Information Technology, e-Business, and Applications, Dec 2004.
V. Cortellessa, K. Goseva-Popstojanova, K. Appukkutty, A. Guedem, A. Hassan, R. Elnaggar, W. Abdelmoez, and H. H. Ammar “Performance-based Risk Analysis of UML Models” submitted to IEEE Transactions on Software Engineering