1. Preliminary Results from a State- of-the-Practice Survey on Risk Management in Off-The-Shelf Component-Based Development Jingyue Li 23 Nov. 2004

2. Agenda Background Research questions Questionnaire design Data collection Results Discussion Future work

3. Background Risks are factors that may adversely affect a project, unless project managers take appropriate countermeasures. Risks in OTS component-based development cover different phases of a project.

4. Typical risks in OTS-component based development PhaseIDPossible risks Project planR1The project was delivered long after schedule. R2Effort to select OTS components was not satisfactorily estimated. R3Effort to integrate OTS components was not satisfactorily estimated. RequirementR4Requirement were changed a lot. R5OTS components could not be sufficiently adapted to changing requirements. R6It is not possible to (re) negotiate requirements with the customer, if OTS components could not satisfy all requirements. Component integration R7OTS components negatively affected system reliability. R8OTS components negatively affected system security R9OTS components negatively affected system performance R10OTS components were not satisfactorily compatible with the production environment when the system was deployed Maintenance and evolutionR11It was difficult to identify whether defects were inside or outside the OTS components R12It was difficult to plan system maintenance, e.g. because different OTS components had asynchronous release cycles R13It was difficult to update the system with the last OTS component version Provider RelationshipR14Provider did not provide enough technical support/ training R15Information on the reputation and technical support ability of provider were inadequate

5. Typical risk management strategies IDRisk management strategies M1Customer had been actively involved in the “acquire” vs. “build” decision of OTS components. M2Customer had been actively involved in OTS component selection. M3OTS components were selected mainly based on architecture and standards compliance, instead of expected functionality M4OTS components qualities (reliability, security etc.) were seriously considered in the selection process M5Effort in learning OTS component was seriously considered in effort estimation M6Effort in black-box testing of OTS components was seriously considered in effort estimation M7Unfamiliar OTS components were integrated first M8Did integration testing incrementally (after each OTS component was integrated ) M9Local OTS-experts actively followed updates of OTS components and possible consequences M10Maintained a continual watch on the market and looked for possible substitute components M11Maintained a continual watch on provider support ability and reputation

6. Research questions RQ1: What are the most frequent risks that software project managers met? RQ2: Can performed risk mitigation actions help to mitigate the corresponding risks?

7. Questionnaire design Background questions to collect information of the company, project, and respondents Main questions about risk and risk management Background questions to collect information about OTS components

8. Data collection Sample definition – A project  Finished, possibly with maintenance  Used one or more OTS components Sample selection strategies  Norway  Italy  Germany

9. Results Currently 42 projects (33 from Norway, 9 from Italy) Cover several application domains Most respondents have solid IT background

10. Results for RQ1 Risk R4 is the most frequent risk. Risks R2, R3, R6, R12, and R14 are classified as the second most frequent risks because they have an up-skewed distribution. Risk R1, R5, R9, R10, R11 and R13 are the third most frequent risks. Risk R7, R8, and R15 are the least frequent risks. These risks have either a lower median or a down-skewed distribution.

11. Results for RQ1 (cont’) Some risks were more frequent than others: Requirement relevant risks (R4 and R6) Cost-estimation risks (R2 and R3) Maintenance plan risk (R12) Provider support risks (R14) Some risks are less frequent: Reliability (R7) Security (R8)

12. Result for RQ2 Risk management action M4, M5, and M8 are the most frequently used. Risk management action M3 and M6 are the second most frequent. Other risk management actions as M1, M2, M7, M9, M10, and M11 are the least frequent.

13. Results for RQ2 (cont’) Quality control methods, such as quality evaluation in selection (M4) and incremental testing (M8) were used much in practice. Possible effort in learning OTS components was seriously considered (M5). Risk management methods relevant to customers (M1, M2) and providers (M9, M10, and M11) were seldom used

14. Results for RQ2 (cont’) RisksRisk management actionsCorrelationP-value R2M4-.307.018 R3M4-.327.013 Relationships between risks and risk management actions Investigated the most frequently used risk management actions (M4, M5 and M8) Used Somers’d method in SPSS 11.0 Risk management action is independent variable and risk is dependent variable. Reported only significant results

15. Discussion Comparison with related work  Requirement relevant risks are the most frequent risks.  Estimation of selection and integration costs in OTS-based projects is perceived as a challenge. Most proposed risk mitigation methods focus on solving this problem by having experienced project manager or a formal cost-estimation model. Our results show that giving complete estimation on the possible effort in OTS component quality evaluation helped to mitigate these risks.  OTS components’ negative effect on the reliability and security of the system is not as frequent as assumed.

16. Discussions (cont’) Threats to validity  Construct validity  Internal validity  Conclusion validity  External validity

17. Future work The data collection is still on-going. More data will be gathered to give further support to conclusions. More qualitative studies to investigate the underlying cause-effect of risk management strategies