بسم الله الرحمن الرحيم الحمد لله ، والصلاة والسلام على رسول الله Introduction to Risk Management And Software Architecture Risk Assessment Hany H. Ammar LANE Department of Computer Science and Electrical Engineering West Virginia University, Morgantown, West Virginia, USA, and Faculty of Computers and Information, Cairo University, Cairo, Egypt

OUTLINE Risk Management Software Architectures Software Architecture Risk Assessment Reliability-based risk Performance-based risk Maintainability-based risk Component Ranking Conclusions Next Steps SW Architecture Risk Assessment Keynote Presentation MySEC’06

Risk Management SW Architecture Risk Assessment Keynote Presentation MySEC’06

SW Architecture Risk Assessment Keynote Presentation MySEC’06

For NASA Programs RISK MANAGEMENT: An organized, systematic decision-making process that efficiently identifies risks, assesses or analyzes risks, and effectively reduces or eliminates risks to achieving program goals. RISK: A Program “Risk” is any circumstance or situation that poses a threat to: crew or vehicle safety, Program controlled cost; Program controlled schedule; or major mission objectives, and for which an acceptable resolution is deemed unlikely without a focused management effort SW Architecture Risk Assessment Keynote Presentation MySEC’06

Risk Management Cycle Identify: Identify that a risk exits and give it a meaningful name. Analyze: Determine the severity of the risk according to the risk matrix. If the risk is negligible (low to medium severity, low likelihood of occurrence), stop here. However, if the risk could cause damage to the system or the system's users, continue. Plan: Decide how to combat the risk based on the risk's severity and likelihood of occurrence. Mitigate: Follow the plan formulated in the previous phase as closely as possible to combat the risk. If this approach does not work, return to the previous phase and make a new plan. If the plan does work, continue analyzing the risk to determine whether it has been reduced to an acceptable severity level. Track: Once the risk has been mitigated to an acceptable severity level, the risk should be tracked to ensure the continued control of the risk. If at any time the risk seems to resurface, the risk management cycle should begin again, starting with the analysis phase. SW Architecture Risk Assessment Keynote Presentation MySEC’06

SW Architecture Risk Assessment Keynote Presentation MySEC’06

Risk Definition According to NASA Software Safety Technical Standard, risk is defined as: “exposure to the chance of injury or loss. It is a function of the possible frequency of occurrence of the undesired event, of the potential severity of resulting consequences, and of the uncertainties associated with the frequency and severity”. For software intensive systems, a risk is a combination of a likelihood of occurrence of an abnormal event or failure and the potential consequences or severity of that event or failure to a system's operators, users, or environment SW Architecture Risk Assessment Keynote Presentation MySEC’06

Likelihood of Occurrence Risk Matrix Severity Likelihood of Occurrence Probable Occasional Remote Improbable Catastrophic High High-Medium Medium Critical Medium-Low Marginal Low Negligible SW Architecture Risk Assessment Keynote Presentation MySEC’06

SW Architecture Risk Assessment Keynote Presentation MySEC’06

NASA IV&V Facility NPD 2820.1C for Software IV&V Policy states: "Task the IV&V Facility in Fairmont, West Virginia to manage the performance of all IV&V for software identified per the established criteria, and for any other safety critical software (as defined in NASA-STD-8719.13)" SW Architecture Risk Assessment Keynote Presentation MySEC’06

IV&V Function Software Independent Verification & Validation (IV&V) is a systems engineering process employing rigorous methodologies for evaluating the correctness and quality of the software product throughout the software life cycle. Software IV&V is adapted to the characteristics of the project. Different projects require different level of IV&V SW Architecture Risk Assessment Keynote Presentation MySEC’06

IV&V Lifecycle Activities System Requirements Review Preliminary Design Review Critical Design Review System Test Mission Readiness Review System Retirement S/W FQT Launch Initial IVVP Signed Baseline IVVP Signed - IV&V provides support and reports for Project milestones Technical Analysis Reports document major phases IVVP is updated to match changes in Project IV&V Provides CoFR IV&V Final Report Concept Phase 2.0 Requirements Phase 3.0 Design Phase 4.0 Implementation Phase 5.0 Test Phase 6.0 Operations & Maintenance Phase 7.0 IV&V Phase Independent Support 1.0 Note: numbers correspond to IV&V WBS Life-cycle IV&V is designed to mesh with the Project schedule and provide timely inputs to mitigate risk Dialog between the IV&V Facility and the Project must begin before SRR For most Projects, IV&V ends (and the Final Report is delivered) on or about MRR. Some Projects have extended S/W development post-launch or major upgrades/maintenance (e.g. Shuttle, MER) SW Architecture Risk Assessment Keynote Presentation MySEC’06

Software Project Resolution Project Resolution is commonly categorized into three resolution types: Successful Projects Completed and operational, and: On Schedule On Cost With all originally specified features and functions Challenged Projects Completed and operational, but: Behind Schedule------- Project Risk Over Cost--------------- Project Risk With fewer features and functions than originally specified ------------ Product Risk Failed Projects: Cancelled before completion or never implemented SW Architecture Risk Assessment Keynote Presentation MySEC’06

Software CHAOS The Standish Group has examined 30,000 Software Projects in the US since 1994. This "CHAOS" research has revealed a decided improvement in IT project management with the implementation of standards and practices such as IV&V. This improvement correlates with the rise in project success depicted in the chart below: Project Resolution History (1994-2000) The Standish Group International, Inc.: Extreme CHAOS (2001) - The 2001 update to the CHAOS report. http://www.standishgroup.com/sample_research/PDFpages/extreme_chaos.pdf SW Architecture Risk Assessment Keynote Presentation MySEC’06

Error Detection/Correction Early error detection and correction are vital. The cost to correct software errors multiplies during the software development lifecycle. Early error detection and correction reduce costs and save time. Direct Return on Investment of Software Independent Verification and Validation: Methodology and Initial Case Studies, James B. Dabney and Gary Barber, Assurance Technology Symposium, 5 June 2003. SW Architecture Risk Assessment Keynote Presentation MySEC’06

IV&V Characteristics Includes Risk Identification and Mitigation Techniques Provides Independent Evaluation / Assessment of: Are we building the product right? = Verification Are we building the right product? = Validation Requires Technical, Managerial and Financial Independence Makes a value added contribution, everyone shares the same mission success objective For NASA Management - Provides Mission Assurance For Project Management - Provides Unbiased Source of Help Helps deliver Risk Identification and Mitigation Increased Quality and Safety Improved Timeliness and Reliability Reduced Rework Cost NPD 8730.4: Requires NASA programs and projects that contain mission or safety critical software to document decisions concerning the use of IV&V. SW Architecture Risk Assessment Keynote Presentation MySEC’06

OUTLINE Risk Management Software Architectures Software Architecture Risk Assessment Reliability-based risk Performance-based risk Maintainability-based risk Component Ranking Conclusions Next Steps SW Architecture Risk Assessment Keynote Presentation MySEC’06

Software Architecture SW Architecture Risk Assessment Keynote Presentation MySEC’06

Why Software Architecture ? Software architecture is a major determinant of software quality (performance, deployment, maintenance and product evolution) “Software architecture is not only concerned with structure and behavior, but also with usage, functionality, performance, resilience, reuse, comprehensibility, economic and technology constraints and tradeoffs” - The Rational Unified Process, 2002 SW Architecture Risk Assessment Keynote Presentation MySEC’06

Why Software Architecture ? The Carnegie Mellon Software Engineering Institute (SEI) developed the Architectural Tradeoff Analysis Method (ATAM) for qualitative evaluation of architectures by expert evaluators and validated its usefulness in practice. This process not only permits evaluation of specific architecture quality attributes (e.g., modifiability, performance, security, and reliability) but also allows engineering tradeoffs to be made among possibly conflicting quality goals. The work presented in this talk complements ATAM by providing scenario-based quantitative approach for architecture risk assessment SW Architecture Risk Assessment Keynote Presentation MySEC’06

Software Architecture Views Architecture views provide the basis for reasoning about the appropriateness and quality of the architecture for achieving system quality goals. Some common architectural views include the logical view, which represents system functions; key system abstractions and their dependencies; and data flows the module decomposition view, which represents the hierarchical decomposition of the system's functionality into units of implementation. This decomposition can include objects, procedures, functions, and their relationships. the communicating-processes view, which represents processing threads, their synchronization, and the data flows between them the deployment view, which shows how the software is allocated to hardware including processors, storage, and external devices or sensors, along with the communications paths that connect them SW Architecture Risk Assessment Keynote Presentation MySEC’06

More on Views Views serve as the primary vehicle for communicating the architecture to its stakeholders, the primary engineering handle for designing quality attributes into the system, and the primary window into the architecture for evaluators who are checking it for fitness of purpose. SW Architecture Risk Assessment Keynote Presentation MySEC’06

What is software architecture ANSI/IEEE Std 1471-2000, Recommended Practice for Architectural Description of Software-Intensive Systems Architecture is defined by the recommended practice as the fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution SW Architecture Risk Assessment Keynote Presentation MySEC’06

What is software architecture A simplified Definition A software architecture is defined by a configuration of architectural elements--components, connectors, and data--constrained in their relationships in order to achieve a desired set of architectural properties. SW Architecture Risk Assessment Keynote Presentation MySEC’06

Software Architecture Elements A component is an abstract unit of software instructions and internal state that provides a transformation of data via its interface A connector is an abstract mechanism that mediates communication, coordination, or cooperation among components. SW Architecture Risk Assessment Keynote Presentation MySEC’06

Software Architecture Elements A datum is an element of information that is transferred from a component, or received by a component, via a connector. A configuration is the structure of architectural relationships among components, connectors, and data during a period of system run-time. An architectural style is a coordinated set of architectural constraints that restricts the roles/features of architectural elements and the allowed relationships among those elements within any architecture that conforms to that style. Detailed example: http://sunset.usc.edu/SSCS/toc.html Standard Satellite Control Segment Reference Architecture SW Architecture Risk Assessment Keynote Presentation MySEC’06

Example of an Architecture Development Process SW Architecture Risk Assessment Keynote Presentation MySEC’06

A Simple Example of Software Architecture Using UML2 EXAMPLE: SATELLITE CONTROL SYSTEM Use-Case Diagram SW Architecture Risk Assessment Keynote Presentation MySEC’06

A Simple Example of Software Architecture Using UML2 SATELLITE CONTROL SYSTEM Architecture composition SW Architecture Risk Assessment Keynote Presentation MySEC’06

A Simple Example of Software Architecture Using UML2 SATELLITE CONTROL SYSTEM Architecture structure SW Architecture Risk Assessment Keynote Presentation MySEC’06

A Simple Example of Software Architecture Using UML2 SATELLITE CONTROL SYSTEM Architectural behavior SW Architecture Risk Assessment Keynote Presentation MySEC’06

OUTLINE Risk Management Software Architectures Software Architecture Risk Assessment Reliability-based risk Performance-based risk Maintainability-based risk Component Ranking Conclusions Next Steps SW Architecture Risk Assessment Keynote Presentation MySEC’06

Software Architecture Risk Assessment This work is funded in part by grants to West Virginia University Research Corp. from the NSF (ITR) Program, and from the NASA Office of Safety and Mission Assurance (OSMA) Software Assurance Research Program (SARP) managed through the NASA Independent Verification and Validation (IV&V) Facility, Fairmont SW Architecture Risk Assessment Keynote Presentation MySEC’06

What keeps satellites working 24/7 ? Project Overview Risk Assessment of software architecture components, usage scenarios, and requirements Risk definition is based on * Frequency of abnormal events * Severity or consequences of events Reliability-based risk, ( FY 95, 06) Performance-based risk (FY01, 05) Requirement–based risk (FY02, 05) Severity Analysis (FY 01,04) Maintainability-based risk (FY 03, 06) Component Ranking (FY 06) What keeps satellites working 24/7 ? SW Architecture Risk Assessment Keynote Presentation MySEC’06

Software Architecture Risk Assessment Develop an architecture-based approach for risk assessment Components\connectors, requirements, and scenario risk, Define several types of risk factors Reliability-based Risk [IEEE Trans. on Rel 2001, on SE, 2002, 2003] Probability of failure * Severity or Consequences of this failure Maintainability-based Risk [RAMS 06, ICSM 05, ICSM 04] Probability of performing maintenance task * Cost of performing this task The losses caused by low system maintainability can be: High cost of maintenance effort Loss of the system by aging Performance-based Risk [IEEE Trans. SE, Jan. 2005] Probability of missing timing or performance requirements * Severity or Consequences SW Architecture Risk Assessment Keynote Presentation MySEC’06

Importance / Benefits Components Risk Factor Components Risk Improve allocation of development and IV&V resources based on quantitative assessment of product metrics Identify the high risk components of the system in terms of Reliability/Maintainability/Performance SW Architecture Risk Assessment Keynote Presentation MySEC’06

Software Architecture Risk Assessment Methodology Requirements Model System Architecture Model Maintainability-based Risk Analysis Performance-based Reliability-based Software Architecture Risk Assessment Components Ranking Components Risk Factors SW Architecture Risk Assessment Keynote Presentation MySEC’06

OUTLINE Risk Management Software Architectures Software Architecture Risk Assessment Reliability-based risk Performance-based risk Maintainability-based risk Component Ranking Conclusions Next Steps SW Architecture Risk Assessment Keynote Presentation MySEC’06

Reliability-based Risk Analysis [IEEE Trans. Reliability] “Risk Assessment of Software Systems Specifications”, [With Nikzadeh and Dugan, 2001] [IEEE Trans. SE] “A methodology for Architecture-level Reliability Risk Analysis” [With Yacoub, June 2002] [IEEE Trans on SE] “Architectural-Level Risk Analysis using UML” [With, Goseva-Popstjanova et al, October 2003] SW Architecture Risk Assessment Keynote Presentation MySEC’06

Component/Connector Risk Factors Software Architecture Risk Assessment Methodology: Estimating Reliability-based Risk Sw Architecture Model (For each Scenario) For each component Measure dynamic complexity Determine severity level using FMEA and hazard analysis For each Connector Measure dynamic coupling Determine severity level using FMEA and hazard analysis Component/Connector Risk Factors [IEEE Trans. SE] “A methodology for Architecture-level Reliability Risk Analysis” [Yacoub & Ammar, June 2002] SW Architecture Risk Assessment Keynote Presentation MySEC’06

CM1case study Components Reliability-Risk Factors SW Architecture Risk Assessment Keynote Presentation MySEC’06

Scenario-Based Reliability Risk For each use case For each scenario For each component Measure dynamic complexity Assign severity based on FMEA and hazard analysis Calculate risk factor For each connector Measure dynamic coupling Assign severity based on FEMA and hazard analysis Construct Markov model Calculate scenario level risk factor SW Architecture Risk Assessment Keynote Presentation MySEC’06

NASA CASE STUDY Use Case Diagram SW Architecture Risk Assessment Keynote Presentation MySEC’06

DTMC of Retry_Both_Pumps Use Case SW Architecture Risk Assessment Keynote Presentation MySEC’06

Distribution of Scenario Risk Factors over severity classes SW Architecture Risk Assessment Keynote Presentation MySEC’06

Sensitivity Analysis of Scenario risk SW Architecture Risk Assessment Keynote Presentation MySEC’06

Severity Analysis Methodology FFA 1 Severity Analysis Methodology Cost of Failure Graph 4 UML Specs Use Case Diagrams, System level Scenario Diagrams Sequence Component/ Connector interactions FTA 3 (List of scenario level Hazards) (List of Component/Connector Failure Modes) Component/ connector Cost-Severity Graph Scenario level 5 Scenario Severity And Components/ Connectors FMEA 2 Scenario Severity Component/Connector Severity Functional Failure Analysis (FFA), Fault Tree Analysis (FTA), Failure Mode and Effects Analysis (FMEA), SW Architecture Risk Assessment Keynote Presentation MySEC’06

Severity Analysis Process Steps Identification of “system hazards”: identifying states of the system that can contribute to accidents and mishaps. FFA is the suggested technique using use case as an input. Performing FMEA to find out component/connector failure modes. Construction of a detailed cause-and-effect model, to record how faults propagate from component /connector level through the system. FTA is the suggested technique. Correlate the top event system failure with component/connector failure to estimate the severity of each component/connector. SW Architecture Risk Assessment Keynote Presentation MySEC’06

FFA Example (Pacemaker AVI Scenario) SW Architecture Risk Assessment Keynote Presentation MySEC’06

FFA Example (Pacemaker AVI Scenario) SW Architecture Risk Assessment Keynote Presentation MySEC’06

Failure Mode and Effect Analysis (FMEA) It is an inductive qualitative method of analysis that is used to identify elements having significant effects on system function in considered application. FMEA is used to evaluate the ways components of the system can fail and the effects these failures can have. FMEA identify single failure modes that either directly result in or contribute significantly to system failure SW Architecture Risk Assessment Keynote Presentation MySEC’06

Component Severity (FEMA) SW Architecture Risk Assessment Keynote Presentation MySEC’06

Fault Tree Analysis (FTA) SW Architecture Risk Assessment Keynote Presentation MySEC’06

Use-Cases Dependencies Non-Primitive Use-Case Terminal Use-Case Primitive Use-Case SW Architecture Risk Assessment Keynote Presentation MySEC’06

Reliability-Based Risk Assessment with Use –Case Relationships Determine terminal, primitive, and non-primitive use cases and their dependent scenarios For each primitive use case Determine the risk factor of each scenario Determine the use case risk factor by taking the max risk scenario For each terminal use case For each dependent scenario Recursively determine the risk factor of the scenario Determine the system risk using a weighted sum of the risk factors of the terminal use cases SW Architecture Risk Assessment Keynote Presentation MySEC’06

Reliability-Based Risk Assessment with Functional Dependencies Aggregate SW Architecture Risk Assessment Keynote Presentation MySEC’06

OUTLINE Risk Management Software Architectures Software Architecture Risk Assessment Reliability-based risk Performance-based risk Maintainability-based risk Component Ranking Conclusions Next Steps SW Architecture Risk Assessment Keynote Presentation MySEC’06

Maintainability-based Risk [ICSM 05] AbdelMoez, Goseva, Ammar, Mili, Fuhrman, “Architectural level Maintainability Based Risk Assessment” [RAMS 06] AbdelMoez, Goseva, Ammar,” Methodology for Maintainability Based Risk Assessment”, Jan 2006. SW Architecture Risk Assessment Keynote Presentation MySEC’06

Importance / Benefits Maintainability-based Risk According to Pigoski, 60%-80% of the system budget is spent on maintenance Enhancements (perfective/ adaptive maintenance) account for 78%-83% of the maintainer effort SW Architecture Risk Assessment Keynote Presentation MySEC’06

Importance / Benefits Maintainability-based Risk Unisys holds the NASA contract to maintain and support 14 million lines of ground software for the space shuttle There were 3,800 requirement changes made to the software after the loss of Challenger. These changes resulted in 900 software releases, of which 30 applied to the mission-control center with 3 of these being major upgrades Reference: IEEE Software, Vol.6,  No.1,  pp. 116-119 http://hebb.cis.uoguelph.ca/~dave/343/Lectures/introduction.html SW Architecture Risk Assessment Keynote Presentation MySEC’06

Importance / Benefits Trade off Analysis for Perfective Maintenance Risk Factor Components Risk Components Patterns Maintainability risk for perfective maintenance (open source case study Borg) SW Architecture Risk Assessment Keynote Presentation MySEC’06

Requirments maturity Index / change / error reports Software Architecture Risk Assessment Methodology: Maintainability-based Risk SW Architecture Requirments maturity Index / change / error reports (1) Estimate components Initial Change Probability (ICP) (2) Estimate Change Propagation (CP) probabilities (3) Estimate Size of Change (SC) ICP=[icpi] SC=[sci/j] CP=[cpi/j] (4) Estimate component risk factor SW Architecture Risk Assessment Keynote Presentation MySEC’06

Maintenance change propagation Outgoing maintenance Incoming maintenance SW Architecture Risk Assessment Keynote Presentation MySEC’06

Estimating Change Propagation V1 C1 C2 = 1 V11 V12 Change in Provided Service V13 Change . V13 Required Services = 0 Change Propagation Probabilities matrix CP=[cpij ] cpij is the probability that a change in Ci due to corrective/ perfective maintenance requires a change in Cj while maintaining the overall function of a system S cpij = P([Cj]  [Cj'] | [Ci]  [Ci'] ^ [S] = [S'] ) cpij is estimated by cpij = SW Architecture Risk Assessment Keynote Presentation MySEC’06

Estimating Size of Change V11 V1 V12 V13 Change Change in Provided Service M1 M2 M3 … M7 Receiving Component methods Size of change SC=[scij ] scij is defined as the ratio between the number of affected methods of the receiving component caused by the changes in the interface elements of the providing components and the total number of methods in the receiving component scij is estimated by scij = SW Architecture Risk Assessment Keynote Presentation MySEC’06

Parameterization of the categorization of the change behavior  ( 0 <  < 1 ) is the propagation area significance,  ( 0 <  < 1 ) is the ripple threshold, and  ( 0 <  < 1 ) is the avalanche threshold SW Architecture Risk Assessment Keynote Presentation MySEC’06

CM1 Maintainability-Based Risk in Adaptive Maintenance Context SW Architecture Risk Assessment Keynote Presentation MySEC’06

Case Study: NASA CM1 UML Model Structure Diagram The UML-RT Model of CM1 was Developed by WVU students (Nathan, Tom and Rajesh, Summer 2004) based on the CM1 software design specification SW Architecture Risk Assessment Keynote Presentation MySEC’06

Change Propagation Probabilities for CM1 The Change Propagation probabilities CP is estimated using the CM1 UML model The Change Propagation probabilities CP can be automatically estimated from UML-RT models, or java source SW Architecture Risk Assessment Keynote Presentation MySEC’06

Size of Change for CM1 The Size of Change metrics SC is estimated using the CM1 UML model The Size of Change metrics SC Probabilities CP can be automatically estimated from UML-RT models, or Java source SW Architecture Risk Assessment Keynote Presentation MySEC’06

Requirments maturity Index / change / error reports Software Architecture Risk Assessment Methodology: Maintainability-based Risk SW Architecture Requirments maturity Index / change / error reports (1) Estimate components Initial Change Probability (ICP) (2) Estimate Change Propagation (CP) probabilities (3) Estimate Size of Change (SC) ICP=[icpi] SC=[sci/j] CP=[cpi/j] (4) Estimate component risk factor SW Architecture Risk Assessment Keynote Presentation MySEC’06

Maintainability-based Risk For corrective maintenance (case study CM1) ICP is estimated using error reports data SW Architecture Risk Assessment Keynote Presentation MySEC’06

Prioritizing Corrective Maintenance Tasks for CM1 The CM1 case study has 98 error reports of components bugs. Assuming that these errors have not been yet fixed, we calculate the frequency of errors occurrences in the components of the system. Then, we estimate the initial change probability ICP of the components by normalizing the frequency of error occurrences by the total number of error reports. For maintenance tasks of components with high severity-levels, they should be fixed regardless of their corresponding maintainability-based risk because of the consequences of such potential failures on the system. On the other hand for maintenance tasks of components that have low severity-levels, we should examine the components maintainability-based risk. So, maintenance tasks of low severity-level and high maintainability-based risk should be avoided or delayed in the maintenance plan Critical Major Cat Cat. Minor Severity Level TMALI TIS SSI SCUI 1553 ICUI EDAC DPA DCX DCI CCM BIT Components SW Architecture Risk Assessment Keynote Presentation MySEC’06

using change reports data Maintainability-based Risk Maintainability risk for Adaptive maintenance (case study CM1) ICP is estimated using change reports data SW Architecture Risk Assessment Keynote Presentation MySEC’06

Initial Change Probabilities for CM1 based on an Added Scenario SW Architecture Risk Assessment Keynote Presentation MySEC’06

Maintainability-based Risk of CM1 For Adaptive Maintenance for the Added Scenario SW Architecture Risk Assessment Keynote Presentation MySEC’06

OUTLINE Risk Management Software Architectures Software Architecture Risk Assessment Reliability-based risk Performance-based risk Maintainability-based risk Component Ranking Conclusions Next Steps SW Architecture Risk Assessment Keynote Presentation MySEC’06

Performance Based Risk [IEEE Trans. SE, Jan. 2005] Cortellessa, Goseva, Ammar “Model-Based Performance Risk Analysis” SW Architecture Risk Assessment Keynote Presentation MySEC’06

Performance – Based Risk Performance is a non-functional software attribute that plays a crucial role in wider application domains spreading from safety-critical systems to e-commerce or web applications. We introduce the concept of performance-based risk, which is risk resulting from software failures originated from behaviors that do not meet performance requirements. Performance failure is the inability of the system to meet its performance objective(s) Performance-based risk is defined as: Probability of performance failure  Severity of the failure SW Architecture Risk Assessment Keynote Presentation MySEC’06

Performance Based Risk Methodology For each Use Case For each scenario STEP 1 – Assign demand vector to each “action” in Sequence diagram; build a Software Execution Model STEP 2 – Add hardware platform characteristics on the Deployment diagram; conduct stand-alone analysis STEP 3 – Devise the workload parameters; build a System Execution Model; conduct contention-based analysis and estimate probability of failure as a violation of a performance objective STEP 4 – Conduct severity analysis and estimate severity of performance failure for the scenario STEP 5 – Estimate the performance risk of the scenario; Identify high-risk components SW Architecture Risk Assessment Keynote Presentation MySEC’06

Build System Execution Model and Conduct Contention-based Analysis (Step 3) Throughput: Response Time: where N is the number of requests, is the sum of all demands in the scenario, and is the maximum demand in that scenario. This slide should be deleted since it refers to D, X, and N and these are not defined, the next slide is enough SW Architecture Risk Assessment Keynote Presentation MySEC’06

Asymptotic bounds and Failure probability estimate (Step 3) Failure probability (Z1) = 0 Failure probability (Z2) = =0.7958 Failure probability (Z3) = 1 SW Architecture Risk Assessment Keynote Presentation MySEC’06

Identify High-risk Components (Step 5) Estimate the overall residence time of each component in a given sequence diagram Sum the time of all processing steps that belong to that component in a given scenario Normalize it with the response time of the sequence diagram Components that contribute significantly to the scenario’s response time are the high-risk components This is not clear, Is this also is based on the workload SW Architecture Risk Assessment Keynote Presentation MySEC’06

Identify High-risk Components (Step 5) This is not clear, Is this also is based on the workload. A diagram containing several scenarios even if it is for another case study (E-commerce) would be important to show what kind of results would be available when we analyze all the scenarios, a 3-D bar chart would be good to have This 3-D graph shows the components on x-axis, scenarios on y-axis and normalized service times on the z-axis SW Architecture Risk Assessment Keynote Presentation MySEC’06

OUTLINE Risk Management Software Architectures Software Architecture Risk Assessment Reliability-based risk Performance-based risk Maintainability-based risk Component Ranking Conclusions Next Steps SW Architecture Risk Assessment Keynote Presentation MySEC’06

Ranking or Classifying Components With Multiple Risk Factors SW Architecture Risk Assessment Keynote Presentation MySEC’06

Ranking Components based on Risk Factors SW Architecture Risk Assessment Keynote Presentation MySEC’06

Software Architecture Risk Assessment Methodology: Components Ranking Component Clustering tree of Reliability-Risk Complexity Severity Fan Out 0.50 0.99 1.00 0.67 0.50 0.06 0.21 0.25 0.40 0.99 0.20 0.31 0.99 0.50 0.40 0.50 0.50 0.99 0.36 0.75 0.38 0.40 0.75 0.60 0.99 0.08 0.50 0.06 0.08 0.50 0.06 Ranking of the 12 components starting from most critical to least critical according to domain experts knowledge is 8,3,10,7,12,9,2,5,11,6,1,4 SW Architecture Risk Assessment Keynote Presentation MySEC’06

Software Architecture Risk Assessment Methodology: Components Ranking Component Clustering tree of Maintainability-Risk Change Propagation Probabilities Normalized Maintenance Impact Fan Out Normalized Maintenance Impact Fan In 0.42 0.46 0.23 0.32 0.19 0.20 0.27 0.17 0.20 0.25 0.22 0.13 0.17 0.11 0.10 0.17 0.17 0.19 0.09 0.13 0.17 0.11 0.12 0.17 0.11 0.14 0.17 0.13 0.17 0.09 0.13 0.17 Component 2 followed by components 9,7, and 5 have the highest maintainability risk SW Architecture Risk Assessment Keynote Presentation MySEC’06

Tool Support SW Architecture Risk Assessment Keynote Presentation MySEC’06

Technology Readiness Level The Software Architecture Risk Assessment Tool Support SW Architecture Risk Assessment Keynote Presentation MySEC’06

Potential Applications The Software Integrity Level Assessment Process (SILAP) at NASA IV&V: determines the set of tasks to be performed on each software component and focus the emphasis and intensity of the IV&V effort on the most important areas (http://ims.ivv.nasa.gov/sharedfiles/documents/IVV_09-1.doc) SW Architecture Risk Assessment Keynote Presentation MySEC’06

Potential Applications NASA Perspective on Risk Assessment by M. Stamatelatos NRC Workshop on Stepping Stones in Space, February 2004 SW Architecture Risk Assessment Keynote Presentation MySEC’06

Conclusions Risk Management is vital to the success of projects and products A risk Assessment process is needed Software Architecture is a major determinant of software quality Software Architecture can be used to manage project and product risks Development of a methodology and a process for software architecture risk assessment Continued development of a software architecture risk assessment tool to support the methodology SW Architecture Risk Assessment Keynote Presentation MySEC’06

Next steps Implement and validate the prototype of the software architecture risk assessment tool Explore the extent to which the NASA IVV risk assessment process (e.g., SILAP) can be enhanced by the techniques and the tools developed in this project SW Architecture Risk Assessment Keynote Presentation MySEC’06

Acknowledgements Faculty Graduate Students: Ali Mili, Prof., NJIT Amir Zaid, Assistant Prof. (Visiting, Spring and Summer 2005) Mark Shereshevsky, Research Assistant Prof. Katerina Guseva, Assistant Prof. Vittorio Cortellessa, Assistant Prof. (Visiting) Graduate Students: Ph.D students: Sherif Yacoub*, Khaild Lateef*, Walid Abdelmoez*, Fadi Haj-Said, Ahmad Hassan*,, Masters students: Kalaivani, Rajesh Gunnalan, Wang, Swetha, Khader, Israr SW Architecture Risk Assessment Keynote Presentation MySEC’06

Papers Published Vittorio Cortellessa, Katerina Goseva-Popstojanova, Kalaivani Appukkutty, Ajith R. Guedem, Ahmed Hassan, Rania Elnaggar, Walid Abdelmoez, Hany H. Ammar, “Model-Based Performance Risk Analysis, IEEE Transactions on Software Engineering, January 2005, (Vol. 31, No. 1), pp.3-20. Katerina Goseva-Popstojanova, Ahmed Hassan, Ajith Guedem, Walid Abdelmoez, Diaa Eldin M. Nassar, Hany Ammar, Ali Mili, "Architectural-Level Risk Analysis Using UML", IEEE Transactions on Software Engineering, October 2003 (Vol. 29, No. 10), pp.946-960. S. Yacoub, H. Ammar, “A Methodology for Architectural-Level Reliability Risk Analysis,” IEEE Transactions on Software Engineering, Vol. 28, No. 6, June 2002. W. AbdelMoez, K. Goseva-Popstojanova, H.H. Ammar,” Methodology for Maintainability-Based Risk Assessment”, Proc. of the 52nd Annual Reliability & Maintainability Symposium (RAMS 2006), Newport Beach, Ca., January 23-26, 2006.   Israr P. Shaik , W. Abdelmoez, R. Gunnalan, M. Shereshevsky, A. Zeid, H.H. Ammar, A. Mili, C. Fuhrman, “Change Propagation for Assessing Design Quality of Software Architectures”, Proc. of 5th IEEE/IFIP Working Conference on Software Architecture (WICSA), Pittsburgh, Pa., USA, November 6-9, 2005. AbdelMoez, W., I. Shaik, R. Gunnalan, M. Shereshevsky, K. Goseva-Popstojanova, H.H. Ammar, A. Mili, C. Fuhrman, “Architectural level Maintainability Based Risk Assessment”, Proc. of poster papers in IEEE International Conference on Software Maintenance (ICSM 2005), Budapest, Hungary, September 25-30,2005. W. Abdelmoez, D. M. Nassar, M. Shereshevsky, N. Gradetsky, R. Gunnalanm and H. H. Ammar, Bo Yu, and Ali Mili "Error Propagation in Software Architectures". In Proceedings of the 10th International Symposium on Software Metrics (METRICS'04), September 11 - 17, 2004 , IEEE Comp. Soc., pp 384-393 Abdelmoez, W., M. Shereshevsky, R. Gunnalan, H. H. Ammar, Bo Yu, S. Bogazzi, M. Korkmaz, A. Mili, "Software Architectures Change Propagation Tool (SACPT),” 20th IEEE International Conference on Software Maintenance (ICSM'04) September 11 - 14, 2004 , Chicago, Illinois, IEEE Comp. Soc., pp 517 A. Hassan, K. Goseva-Popstojanova, and H. Ammar, “UML Based Severity Analysis Methodology”, Proceedings of the 2005 Annual Reliability and Maintainability Symposium (RAMS 2005), Alexandria, VA, January 2005. SW Architecture Risk Assessment Keynote Presentation MySEC’06