3/14/2006USC-CSE1 Ye Yang, Barry Boehm Center for Software Engineering University of Southern California COCOTS Risk Analyzer and Process Usage Annual Research Review Mar. 14 th, 2006

3/14/2006USC-CSE2 Outline Motivation COCOTS Model COCOTS Risk Analyzer Evaluation Process Usage: Risk-Based Prioritization Conclusions

3/14/2006USC-CSE3 Motivation Enable COTS integration risk analysis with COCOTS cost estimation inputs Identify relative risk levels of COTS-based development (CBD) Provide recommendations to improve risk management practices

3/14/2006USC-CSE4 COCOTS Model - Calibrated to 20 industry projects

3/14/2006USC-CSE5 COCOTS Glue Code Sub-model

3/14/2006USC-CSE6 COCOTS Risk Analyzer

3/14/2006USC-CSE7 Knowledge Base Contents –Risk Rules (RR) –Risk level scheme –Common risk mitigation strategy Constructing approach –Expert Delphi Survey –Empirical study results –Literature review

3/14/2006USC-CSE8 Risk Rule A CBD risk situation –a combination of two cost attributes at their extreme ratings Risk Rule (RR) –An identified risk situation is formulated as a risk rule. E.g. one example RR: IF ((COTS Product Complexity > Nominal) AND (Integrator’s Experience on COTS Product < Nominal)) THEN there is a project risk.

3/14/2006USC-CSE9 Risk Situation Identification Total # of Delphi responses: 5 # of responses % of responses # of risk situations >=3>50%24 240%26 120%28 24 Risk Rules formulated in the knowledge base >=50%40%20% (Percentage of responses over total)

3/14/2006USC-CSE10 Risk Potential Rating for Cost Factors Mapping between cost factor’s rating to its risk potential rating:

3/14/2006USC-CSE11 Risk Level Scheme Assignment of risk probability levels: Risk levelQuantifier Severe0.4 Significant0.2 General0.1 Quantitative weighting scheme:

3/14/2006USC-CSE12 Productivity Range Reflects the cost consequence of risk occurring Combines both expert judgment and industry data calibration

3/14/2006USC-CSE13 Project Risk Quantification Project Overall Risk: –Riskprob ij corresponds to the nonlinear relative probability of the risk occurring –The product of PR i and PR j represents the cost consequence of the risk occurring Risk interpretation: –Normalized scale: 0 ~ 100 –100 represents the situation where each cost factor is rated at its most expensive extremity –0 ~ 5: low risk; 5 ~ 15: medium risk; 15 ~ 50: high risk; 50 ~ 100: very high risk

3/14/2006USC-CSE14 Risk Mitigation Recommendations Knowledge base built on previous empirical study results, e.g.: Risk RuleRisk SituationMitigation Advice APCPX_ACIPC (High, Very Low) Complex integration with inexperienced personnel Consider more compatible COTS; re-staffing; training; consultant mentoring ACREL_ACPMT (High, Low) High-reliability application dependent on immature COTS Consider more mature COTS; reliability-enhancing COTS wrappers; risk-based testing ACPER_AAREN (High, Very Low) Unvalidated architecture with COTS performance shortfalls Benchmark current and alternative COTS choices; reassess performance requirements vs. achievables

3/14/2006USC-CSE15 Evaluation Results Data: 9 USC e-services projectsData: 7 COCOTS calibration projects

3/14/2006USC-CSE16 Process Usage – An Example COTS A and B are our strongest COTS choices –But there is some chance that they have incompatible HCI’s –Probability of loss P(L) COTS C is almost as good as B, and it is compatible with A

3/14/2006USC-CSE17 Risk-Driven CBD Process Framework

3/14/2006USC-CSE18 Different Risk Strategy Resulting in Different Process

3/14/2006USC-CSE19 Conclusions CBD brings a host of unique risk items Many risk techniques/tools require intensive user inputs COCOTS Risk Analyzer provides a handy way to automate the CBD risk analysis by leveraging on existing knowledge and expertise in both cost estimation and risk mgmt. Case study shows how it supports process decisions following the risk based prioritization strategy

3/14/2006USC-CSE20 Backup Slides

3/14/2006USC-CSE21 Risk Potential Rating Captures the underlying relation between cost attributes and the impact of their specific ratings on project risk –4 Levels OK, Moderate, Risk Prone, and Worst Case Two types of treatments –Transforming continuous Size representation into discrete risk potential ratings –Mapping cost driver ratings into risk potential ratings

3/14/2006USC-CSE22 Risk Potential Rating for Size Delphi Responses for Size Rating (Size in KSLOC):

3/14/2006USC-CSE23 Risk Based Prioritization Strategy