IV&V Facility 1 FY 2002 Initiative IV&V of UML Hany Ammar, Katerina Goseva-Popstojanova, V. Cortelessa, Ajith Guedem, Diaa Eldin Nassar, Walid AbdelMoez, Ahmad Hassan, and Rania Elnaggar LANE Department of Computer Science and Electrical Engineering West Virginia University Ali Mili, Bo Yu College of Computing Science New Jersey Institute of Technology Less risk, sooner - A Catch Phrase by Coach Menzies WVU UI: Architectural-level Risk Assessment

IV&V Facility 2 Outline Objectives What we can do Why UML UML & NASA Project Overview Architecture-Based Risk Analysis The Risk Assessment Methodology Performance – based risk Accomplishments Future Work Publications

IV&V Facility 3 Objectives Automated techniques V&V of dynamic specifications –Performance and timing analysis –Fault-injection based analysis, Less risk, sooner – Risk assessment Technologies: –UML –Architectures –Risk assessment methodology Benefits: –Find & rank critical use cases, scenarios, components, connectors Before bad software After bad software The ARIANE 5 explosion

IV&V Facility 4 What We Can Do Identify and rank critical components based on risk factors and severity classes How?- details follow Minor Major Critical Catastrophic Not contributing

IV&V Facility 5 Why UML Unified modeling language –Rational software –The three amigos: Booch Rumbaugh, Jacobson. International standard in system specification An international standard In system specification

IV&V Facility 6 UML & NASA Increasing use at NASA Informal (very) survey –Google search: –“rational rose nasa” –10,000 hits –3 definite projects, just in first ten We use a case study based on the UML specs of a component of the International Space Station

IV&V Facility 7 Project Overview FY01 Developed of an automated simulation environment for UML dynamic specification, suggested an observer component to detect errors Conducted performance and timing analysis of the NASA case study FY02 Develop a fault injection methodology Define a fault-model for components at the specification level Develop a methodology for architecture-based risk analysis Determine critical use case List Determine critical component/connector list (based on recent paper by Yacoub & Ammar on IEEE Trans. on Software Engineering, June 02) FY03 Develop a methodology for performance-based/reliability-based risk assessment Validation of the risk analysis methodology on several NASA projects

IV&V Facility 8 Architecture-Based Risk Analysis Develop architecture-based approach for risk assessment –Overall system/subsystem –Different use cases –Key scenarios associated with use cases Heavily used scenarios Scenarios that are used infrequently but perform critical functions Develop components and connectors risk factors –Define components risk factors as Normalized dynamic complexity * Severity –Estimate dynamic complexity measure based UML sequence diagrams and state charts –Estimate severity measure based FEMA and hazard analysis –Consistent with the NASA definition of risk Probability of an undesired event * Consequences if that event should occur –Define connectors risk factor as Normalized dynamic coupling * Severity

IV&V Facility 9 Risk Assessment Methodology For each use case –For each scenario For each component –Measure dynamic complexity –Assign severity based on FMEA and hazard analysis –Calculate risk factor For each connector –Measure dynamic coupling –Assign severity based on FEMA and hazard analysis –Calculate risk factor Construct Markov model Calculate scenario level risk factor Determine critical component/connector list –Calculate use case level risk factors –Rank the scenarios based on risk factors, Determine critical scenarios list Calculate system level risk Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope

IV&V Facility 10 Risk Assessment Methodology For each use case –For each scenario For each component –Measure dynamic complexity –Assign severity based on FMEA and hazard analysis –Calculate risk factor For each connector –Measure dynamic coupling –Assign severity based on FEMA and hazard analysis –Calculate risk factor Construct Markov model Calculate scenario level risk factor Determine critical component/connector list –Calculate use case level risk factors –Rank the scenarios based on risk factors, Determine critical scenarios list Calculate system level risk Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope

IV&V Facility 11 NASA CASE STUDY Use Case Diagram

IV&V Facility 12 Risk Assessment Methodology For each use case –For each scenario For each component –Measure dynamic complexity –Assign severity based on FMEA and hazard analysis –Calculate risk factor For each connector –Measure dynamic coupling –Assign severity based on FEMA and hazard analysis –Calculate risk factor Construct Markov model Calculate scenario level risk factor Determine critical component/connector list –Calculate use case level risk factors –Rank the scenarios based on risk factors, Determine critical scenarios list Calculate system level risk Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope

IV&V Facility 13 Both Pumps Retry- scenario

IV&V Facility 14 Risk Assessment Methodology For each use case –For each scenario For each component –Measure dynamic complexity –Assign severity based on FMEA and hazard analysis –Calculate risk factor For each connector –Measure dynamic coupling –Assign severity based on FEMA and hazard analysis –Calculate risk factor Construct Markov model Calculate scenario level risk factor Determine critical component/connector list –Calculate use case level risk factors –Rank the scenarios based on risk factors, Determine critical scenarios list Calculate system level risk Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope

IV&V Facility 15 Component Dynamic Complexity Dynamic Complexity of a component O i in scenario x is defined as : Where CC x (o i ) =t i – c i + 2 Is the cyclomatic complexity of a component O i in scenario x C x (o i ) : A finite set of states for a component O i for a scenario x, c i is the cardinality of this set. T x (o i ) : A finite set of transitions from one state to another for O i component in a scenario x, t i is the cardinality of this set. O x : is the set of components collaborating during the execution of a scenario

IV&V Facility 16 Component Severity (FEMA)

IV&V Facility 17 Risk Assessment Methodology For each use case –For each scenario For each component –Measure dynamic complexity –Assign severity based on FMEA and hazard analysis –Calculate risk factor For each connector –Measure dynamic coupling –Assign severity based on FEMA and hazard analysis –Calculate risk factor Construct Markov model Calculate scenario level risk factor Determine critical component/connector list –Calculate use case level risk factors –Rank the scenarios based on risk factors, Determine critical scenarios list Calculate system level risk Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope

IV&V Facility 18 Dynamic coupling for connector between component O i and component O j Connector Dynamic Coupling MT x (O i,O j ) : is the set of messages sent from component O i to component O j during the execution of scenario x MT x : is the set of total messages exchanged between all components during the execution of scenario x. is the percentage of the number of messages sent from O i to O J with respect to the total number of messages exchanged during the scenario x

IV&V Facility 19 Connector Dynamic Coupling Dynamic Coupling for Connector between component C1 and component C2 is EOC(C 1,C 2 ) =2/5=0.4 M 1 M 2 M 3 M 4 M 5 C1C1 C2C2 C3C3

IV&V Facility 20 Connector Severity (FEMA)

IV&V Facility 21 Risk Assessment Methodology For each use case –For each scenario For each component –Measure dynamic complexity –Assign severity based on FMEA and hazard analysis –Calculate risk factor For each connector –Measure dynamic coupling –Assign severity based on FEMA and hazard analysis –Calculate risk factor Construct Markov model Calculate scenario level risk factor Determine critical component/connector list –Calculate use case level risk factors –Rank the scenarios based on risk factors, Determine critical scenarios list Calculate system level risk Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope

IV&V Facility 22 The Markov Chain Model for Both Pumps Retry- scenario FAILURE STATES OF VARIOUS SEVERITIES PFMC_LT Minor Major Critical Catastrophic FRITC S SCITC S RPCM_MT RPCM_LT PFMC_M T S T

IV&V Facility 23 Risk Assessment Methodology For each use case –For each scenario For each component –Measure dynamic complexity –Assign severity based on FMEA and hazard analysis –Calculate risk factor For each connector –Measure dynamic coupling –Assign severity based on FEMA and hazard analysis –Calculate risk factor Construct Markov model Calculate scenario level risk factor Determine critical component/connector list –Calculate use case level risk factors –Rank the scenarios based on risk factors, Determine critical scenarios list Calculate system level risk Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope

IV&V Facility 24 Distribution of risk factors of each scenario over the severity classes

IV&V Facility 25 Risk Assessment Methodology For each use case –For each scenario For each component –Measure dynamic complexity –Assign severity based on FMEA and hazard analysis –Calculate risk factor For each connector –Measure dynamic coupling –Assign severity based on FEMA and hazard analysis –Calculate risk factor Construct Markov model Calculate scenario level risk factor Determine critical component/connector list –Calculate use case level risk factors –Rank the scenarios based on risk factors, Determine critical scenarios list Calculate system level risk Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope

IV&V Facility 26 Overall System risk distribution over the severity classes MINORMARGINALCRTICALCATASTROPHIC The overall system risk factor is:

IV&V Facility 27 Sensitivity analysis of components

IV&V Facility 28 Risk Assessment Methodology For each use case –For each scenario For each component –Measure dynamic complexity –Assign severity based on FMEA and hazard analysis –Calculate risk factor For each connector –Measure dynamic coupling –Assign severity based on FEMA and hazard analysis –Calculate risk factor Construct Markov model Calculate scenario level risk factor Determine critical component/connector list –Calculate use case level risk factors –Rank the scenarios based on risk factors, Determine critical scenarios list Calculate system level risk Rank the use cases based on risk factors, Determine critical use case list Determine critical component / connector list in the system scope

IV&V Facility 29 Determine Critical Component/Connector List MinorMajorCritical Catastrophic Not contributing

IV&V Facility 30 Performance failure is the inability of the system to meet its performance objective(s) Define components performance-based risk as Normalized component demand factor * Severity Performance – based risk X1X1 X2X2 X3X3 T 11 T 21 T 22 T 31 T 23 T 12 T 32 D 12 D 21 D 22 D 23 D 31 D 32 D 11 = is demand for resource (e.g, CPU, disk, etc.) in state T ij (state j of component i ) Scaling vector the resource demands accordingly to the corresponding service times of the resources scales

IV&V Facility 31 Total demand of component x i in a scenario S k is Normalized demand factor of component x i in Scenario S k DF i = (. SC T ) / (. SC T ) where m is total number of components and l total number of states for a given component in a given scenario Overall demand of a scenario S k is Performance – based risk

IV&V Facility 32 Accomplishments Developed analytical techniques and a methodology for Architecture-Based Risk Analysis A lightweight approach based on static analysis of dynamic specifications is developed and automated A tool will be presented in the Tools session Applied the methodology and tool to the NASA case study

IV&V Facility 33 Future Work The main thrust of our future work will be in the development of a cohesive methodology for performance- based /reliability- based risk assessment Compare risk factors based on other Complexity and coupling metrics obtained from static analysis of UML dynamic specs. –COSMIC-Full Function Point measurement maybe a good complexity predictor. –COCOMO II’s effort prediction may be another good complexity predictor Validation of methodology using several NASA case studies

IV&V Facility 34 Publications 1.Sherif M. Yacoub, Hany H. Ammar, “A Methodology for Architecture-Level Reliability Risk Analysis,” IEEE Transactions on Software Engineering, June 2002, pp H. H. Ammar, T. Nikzadeh, and J. B. Dugan "Risk Assessment of Software Systems Specifications," IEEE Transactions on Reliability, September Hany H. Ammar, Sherif M. Yacoub, Alaa Ibrahim, “A Fault Model for Fault Injection Analysis of Dynamic UML Specifications,” International Symposium on software Reliability Engineering, IEEE Computer Society, November Rania M. Elnaggar, Vittorio Cortellessa, Hany Ammar, “A UML-based Architectural Model for Timing and Performance Analyses of GSM Radio Subsystem”, 5th World Multi- Conference on Systems, Cybernetics and Informatics, July. 2001, Received Best Paper Award 5.Ahmed Hassan, Walid M. Abdelmoez, Rania M. Elnaggar, Hany H. Ammar, “An Approach to Measure the Quality of Software Designs from UML Specifications,” 5th World Multi- Conference on Systems, Cybernetics and Informatics and the 7th international conference on information systems, analysis and synthesis ISAS July Hany H. Ammar, Vittorio Cortellessa, Alaa Ibrahim “Modeling Resources in a UML-based Simulative Environment”, ACS/IEEE International Conference on Computer Systems and Applications (AICCSA'2001), Beirut, Lebanon, June A. Ibrahim, Sherif M. Yacoub, Hany H. Ammar, “Architectural-Level Risk Analysis for UML Dynamic Specifications,” Proceedings of the 9th International Conference on Software Quality Management (SQM2001), Loughborough University, England, April 18-20, 2001, pp