CBSE Seminar -4 Feb OSLO 1 Risk management and Process Improvement of Off-The-Shelf Based Development Jingyue Li Reidar Conradi, Odd Petter N. Slyngstad, Norwegian University of Science and Technology Marco Torchiano, Maurizio Morisio, Dip.Automatica e Informatica, Politecnico di Torino Christian Bunse Fraunhofer IESE

CBSE Seminar -4 Feb OSLO 2 Agenda Research design Research design BackgroundBackground Research questionsResearch questions Sample selectionSample selection Results Results Selected samplesSelected samples Answers to research questionsAnswers to research questions Discussions Discussions Conclusions and future work Conclusions and future work

CBSE Seminar -4 Feb OSLO 3 Research design – Motivation Pre-study background Pre-study background This study followings a pre-study with 16 structured interviews in Norway, from Oct to Feb This study followings a pre-study with 16 structured interviews in Norway, from Oct to Feb Focused on SPI in COTS-based developmentFocused on SPI in COTS-based development Respondents shared a lot of experiences on risk management in COTS-based developmentRespondents shared a lot of experiences on risk management in COTS-based development Limitations of the pre-studyLimitations of the pre-study  Small sample size  Sample selected on convenience Motivation of this main study Motivation of this main study State-of-the-practice surveyState-of-the-practice survey Randomly selected much larger samples to validate conclusions of the pre-studyRandomly selected much larger samples to validate conclusions of the pre-study Also included Open Source ComponentAlso included Open Source Component

CBSE Seminar -4 Feb OSLO 4 Research design – research questions RQ1 - How to improve the development process in projects using OTS components. RQ1 - How to improve the development process in projects using OTS components. RQ2 - How to predict possible risks (problems) in projects using OTS components? RQ2 - How to predict possible risks (problems) in projects using OTS components? RQ3 - What are the effective methods to mitigate risks in projects using OTS components? RQ3 - What are the effective methods to mitigate risks in projects using OTS components? RQ4 - What are the similarities and differences between projects using COTS and OSS components? RQ4 - What are the similarities and differences between projects using COTS and OSS components?

CBSE Seminar -4 Feb OSLO 5 Research design – sample selection Norway Norway Germany Germany Italy Italy (Sample selection reported in later presentation)

CBSE Seminar -4 Feb OSLO 6 Research results – selected samples Current data Current data Total 86 projectsTotal 86 projects NorwayNorway  46 projects from 38 companies  One company filled in 4, one filled in 3, and one filled in 2.  In other companies, we selected only one project each company GermanyGermany  29 projects from 29 companies ItalyItaly  11 projects from 11 companies Data collection is still on-going in Germany and Italy Data collection is still on-going in Germany and Italy

CBSE Seminar -4 Feb OSLO 7 Research results – selected companies

CBSE Seminar -4 Feb OSLO 8 Research results – selected companies (cont’) Small (0-19) Medium (20-99) Large (more than 100)

CBSE Seminar -4 Feb OSLO 9 Research results – selected projects

CBSE Seminar -4 Feb OSLO 10 Research results – selected respondents

CBSE Seminar -4 Feb OSLO 11 85% respondents have more than 3 years experience on OTS-based development 85% respondents have more than 3 years experience on OTS-based development Most respondents have the Bachelor degree in informatics, 10% have Ph.D degree. Most respondents have the Bachelor degree in informatics, 10% have Ph.D degree. Research results – selected respondents (cont’)

CBSE Seminar -4 Feb OSLO 12 Research question RQ1  How to improve the development process in projects using OTS components? Overall development processOverall development process  Do I need to change my main development process dramatically in projects using OTS?  What activities and roles should be added? OTS selection processOTS selection process  Formal decision making process?  Familiar with component process?

CBSE Seminar -4 Feb OSLO 13 RQ1: Do I need to change my main development process dramatically? More than 80% projects members decided their main development process (Waterfall, incremental, etc.) before they started to think about using OTS. More than 80% projects members decided their main development process (Waterfall, incremental, etc.) before they started to think about using OTS. It actually worked. It actually worked.

CBSE Seminar -4 Feb OSLO 14 RQ1: What should be added? Activities Activities ”Acquire” vs. ”build” decision”Acquire” vs. ”build” decision OTS component selectionOTS component selection Learning OTS componentLearning OTS component Build glueware and/or addwareBuild glueware and/or addware A new role (OTS knowledge keeper) A new role (OTS knowledge keeper) Germany (100%)Germany (100%) Norway (37%)Norway (37%) Italy (9%)Italy (9%)

CBSE Seminar -4 Feb OSLO 15 RQ1: What is the proper OTS selection process? Formal decision making process (by 15% used) Formal decision making process (by 15% used) Selecting evaluation criteria (factors)Selecting evaluation criteria (factors) Collecting and assigning values to these criteriaCollecting and assigning values to these criteria Applying formal decision making algorithms such as MAUT or MCDA etc.Applying formal decision making algorithms such as MAUT or MCDA etc. Familiar with component process (by 85% used) Familiar with component process (by 85% used) Search internetSearch internet Limited to 2-3 componentsLimited to 2-3 components Download demo version and try it, then decideDownload demo version and try it, then decideOr Recommended from internal/external expertsRecommended from internal/external experts

CBSE Seminar -4 Feb OSLO 16 Research question RQ2 How to predict possible risks in projects using OTS components? How to predict possible risks in projects using OTS components? What were the most frequent risks (problems) in practice?What were the most frequent risks (problems) in practice? Was there any relationship between those risks (problems) and the project profile?Was there any relationship between those risks (problems) and the project profile?

CBSE Seminar -4 Feb OSLO 17 RQ2: Typical risks PhaseRisks Project planThe project was delivered long after schedule Effort to select OTS components was not satisfactorily estimated Effort to integrate OTS components was not satisfactorily estimated RequirementRequirements were changed a lot OTS components could not be sufficiently adapted to changing requirements It is not possible to (re) negotiate requirements with the customer, if OTS components could not satisfy all requirements Component integration OTS components negatively affected system reliability OTS components negatively affected system security OTS components negatively affected system performance OTS components were not satisfactorily compatible with the production environment when the system was deployed

CBSE Seminar -4 Feb OSLO 18 RQ2: Typical risks (cont’) PhaseRisks Maintenance and evolution It was difficult to identify whether defects were inside or outside the OTS components It was difficult to plan system maintenance, e.g. because different OTS components had asynchronous release cycles It was difficult to update the system with the last OTS component version Provider Relationship Provider did not provide enough technical support/ training Information on the reputation and technical support ability of provider were inadequate

CBSE Seminar -4 Feb OSLO 19 RQ2: Frequency of typical risks (problems) in OTS based development

CBSE Seminar -4 Feb OSLO 20 RQ2: Frequency of typical risks in OTS based development (cont’) Most frequent risks Most frequent risks Effort to integrate OTS components was not satisfactorily estimatedEffort to integrate OTS components was not satisfactorily estimated Keep up with requirements evolutionKeep up with requirements evolution Identify defects inside or outside OTS componentIdentify defects inside or outside OTS component Least frequent risks Least frequent risks Negative reliability effectNegative reliability effect Negative security effectNegative security effect Negative performance effectNegative performance effect Lack provider informationLack provider information

CBSE Seminar -4 Feb OSLO 21 RQ2: Relationship between typical risks (problems) and project context The more different OTS-components used in the project, the more frequent the following risks: The more different OTS-components used in the project, the more frequent the following risks: Identify whether defects were inside or outside the OTS componentsIdentify whether defects were inside or outside the OTS components It was difficult to update the system with the last version OTS componentsIt was difficult to update the system with the last version OTS components Provider did not provide enough technical support/trainingProvider did not provide enough technical support/training

CBSE Seminar -4 Feb OSLO 22 RQ2: Relationship between typical risks (problems) and project context (cont’) The higher the general experience on OTS-based development in projects, the less frequent the following risks: The higher the general experience on OTS-based development in projects, the less frequent the following risks: Effort to integrate OTS c components was not satisfactorily estimatedEffort to integrate OTS c components was not satisfactorily estimated It was difficult to identify whether defects were inside or outside the OTS componentsIt was difficult to identify whether defects were inside or outside the OTS components

CBSE Seminar -4 Feb OSLO 23 RQ2: Relationship between typical risks (problems) and project context (cont’) The project with an OTS knowledge keeper had less frequency on the following risks than project without OTS knowledge keeper: The project with an OTS knowledge keeper had less frequency on the following risks than project without OTS knowledge keeper: Difficult ot identify risks inside or outside OTS componentsDifficult ot identify risks inside or outside OTS components Lack the information of the vendors’ reputation and support abilityLack the information of the vendors’ reputation and support ability

CBSE Seminar -4 Feb OSLO 24 Research question RQ3 What are the effective methods to mitigate risks in projects using OTS components? What are the effective methods to mitigate risks in projects using OTS components? Which strategies had been frequently used in practice?Which strategies had been frequently used in practice? What were the effective strategies?What were the effective strategies?

CBSE Seminar -4 Feb OSLO 25 RQ3: Proposed risk management strategies Customer had been actively involved in “acquire” vs. “build” decision Customer had been actively involved in OTS component selection OTS components were selected mainly based on architecture and standards compliance, instead of expected functionality OTS components qualities (reliability, security etc.) were seriously considered during selection Effort in learning OTS component was seriously considered in effort estimation

CBSE Seminar -4 Feb OSLO 26 RQ3: Proposed risk management strategies (cont’) Effort in black-box testing of OTS components was seriously considered in effort estimation Unfamiliar OTS components were integrated first   Did integration testing incrementally (after each OTS component was integrated)   Local OTS-experts actively followed updates of OTS components and possible consequences   Maintained a continual watch on the market and looked for possible substitute components   Maintained a continual watch on provider support ability and reputation

CBSE Seminar -4 Feb OSLO 27 RQ3: Frequency of using proposed risk management strategies in practice

CBSE Seminar -4 Feb OSLO 28 RQ3: Frequency of using proposed risk management strategies in practice (cont’) The most frequently used risk management strategies: The most frequently used risk management strategies: OTS components qualities were seriously considered in the selection processOTS components qualities were seriously considered in the selection process Unfamiliar OTS components were integrated firstUnfamiliar OTS components were integrated first Did integration testing incrementallyDid integration testing incrementally Local OTS-experts actively followed updates of OTS components and possible consequences The least frequently used risk management strategies: The least frequently used risk management strategies: Involve customers in theInvolve customers in the “acquire” vs. “build” decision Invove customers in OTS selectionInvove customers in OTS selection

CBSE Seminar -4 Feb OSLO 29 RQ3: What were effective risk management strategies ? Risks Effective risk management method Estimate selection effort OTS components qualities (reliability, security etc.) were seriously considered in the selection process Estimate integration effort OTS components qualities (reliability, security etc.) were seriously considered in the selection process Follow requirement changes Maintained a continual watch on the market and looked for possible substitute components Plan maintenance OTS components qualities (reliability, security etc.) were seriously considered in the selection process Lack provider support Maintained a continual watch on provider support ability and reputation

CBSE Seminar -4 Feb OSLO 30 RQ3: Risk management recommendations in OTS-based projects Avoid risk Avoid risk Do not use too many different OTS components in one projectDo not use too many different OTS components in one project Manage risk Manage risk Manage the knowledge of OTS properly (Have a OTS expert and share OTS experience regularly)Manage the knowledge of OTS properly (Have a OTS expert and share OTS experience regularly) Spend enough time on OTS quality evaluation. Hand-on trial is necessarySpend enough time on OTS quality evaluation. Hand-on trial is necessary Do not marry specific OTS. Be ready for possible replacementDo not marry specific OTS. Be ready for possible replacement Maintain a continual watch on provider support ability and reputationMaintain a continual watch on provider support ability and reputation

CBSE Seminar -4 Feb OSLO 31 Research question RQ4 What are the similarities and differences between projects using COTS and OSS components? What are the similarities and differences between projects using COTS and OSS components? Are there any similarities and differences in:Are there any similarities and differences in:  Company, project, system profile ?  Motivation of using them ?  Frequency of risks (problems) ?

CBSE Seminar -4 Feb OSLO 32 RQ4: Selected samples – COTS projects vs. OSS projects 56 projects used only COTS 56 projects used only COTS 25 projects used only OSS 25 projects used only OSS 5 projects used both COTS and OSS (not considered in data analysis) 5 projects used both COTS and OSS (not considered in data analysis)

CBSE Seminar -4 Feb OSLO 33 RQ4: Are there any similarities and differences in company profile ?

CBSE Seminar -4 Feb OSLO 34 RQ4: Are there any similarities and differences in company profile ? (cont’)

CBSE Seminar -4 Feb OSLO 35 RQ4: Are there any similarities and differences in project profile ?

CBSE Seminar -4 Feb OSLO 36 RQ4: Are there any similarities and differences in system profile ?

CBSE Seminar -4 Feb OSLO 37 Our conclusion Our conclusion There is no difference in company, project and system profile between projects using COTS and OSS.There is no difference in company, project and system profile between projects using COTS and OSS. RQ4: Are there any similarities and differences in company, project, and system profile ?

CBSE Seminar -4 Feb OSLO 38 RQ4: Are there any similarities and differences in motivation of using COTS vs. OSS ? Commonalities Commonalities Shorter time-to-marketShorter time-to-market Less development and maintenance effortLess development and maintenance effort Higher reliabilityHigher reliability Differences Differences COTSCOTS  Follow the market trend  Paid software will give good reliability  Good support OSSOSS  New technology  Free source code  Avoid the risk in OSS evolution

CBSE Seminar -4 Feb OSLO 39 RQ4: Are there any similarities and differences in frequency of risks (problems) ? Commonalities Commonalities Requirement changed a lot and it was difficult to keep up with these changesRequirement changed a lot and it was difficult to keep up with these changes Differences Differences COTS: higher risk on following evolution of both requirements and COTS componentCOTS: higher risk on following evolution of both requirements and COTS component OSS: higher risk on getting good supportOSS: higher risk on getting good support

CBSE Seminar -4 Feb OSLO 40 Questions ?