1 SPIKE Seminar Process Improvement and Risk Management in OTS (Off-The-Shelf) Component-Based Development Jingyue Li Norwegian University of Science and Technology 7 th September 2004

2 SPIKE Seminar Agenda Section 1: An Empirical Study of Variations in COTS-based Software Development Process Section 2: Risk management in OTS (COTS and Open Source) based development

3 SPIKE Seminar Section 1: An Empirical Study of Variations in COTS-based Software Development Process State-of-the-art Research motivation and research questions Research methods Results Discussions

4 SPIKE Seminar State-of-the-art Process of the whole software development lifecycle –Boehm et al.: Both the waterfall model and evolutionary development process were regarded as unsuitable for COTS-based development. –NASA COTS-based development process –SEI EPIC (Evolutionary Process for Integrating COTS-based system) –Seven theses about COTS-based development COTS component selection and evaluation processes –MAUT, MCDA etc. (decision making algorithm drvien ) –RCPEP (requirement driven)

5 SPIKE Seminar Research Motivation Limitations of previous studies –Focused on propose new revised process without empirical evidence –Small project sample with similar contexts –Difficult for project managers to customize their processes based on their project context (developer skill, applicaiton domain, requirement ”toughness” etc.). Our motivation –To find out how to customize a COTS-based development process based on their project context.

6 SPIKE Seminar Research Questions RQ1: What was the actual process used in the projects using COTS Components. RQ2:What are the commonalities and possible variations in COTS-based development processes. RQ3: What are process scenarios of the projects using COTS components successfully and not successfully? What are the relationship between possible risks and process scenarios

7 SPIKE Seminar Research Methods Semi-structured interviews –Interview guide include both open and close questions –Definitions: COTS Is either provided by some other organizations in the same company, or provided by external companies as a commercial product. Is integrated into the final delivered system. Is not a commodity, i.e. not shipped with an operating system, not provided with the development environment, not generally included in any pre-existing platforms. Is not controllable by the user, in terms of provided features and their evolution. Our addition: This normally means “black box”, i.e. no source code available. Pre-test of interview guide Data collection method Data analysis method

8 SPIKE Seminar Results - Background information Companies (13 Norwegian IT companies) –1 small sized companies (3<staff size <15) –7 medium sized companies (15 =<staff size <=100 ) –5 large sized companies (staff size > 100) Projects (16 finished project using one or more COTS components) Respondents –3 IT manager, 4 project manager, 5 software architects, 4 developers –9 of them have more than 10 years SW development experience COTS Component samples –Following COM/DCOM, CORBA, EJB, etc. –C++, or Java library

9 SPIKE Seminar Results - Answers to research question RQ1 RQ1: What was the actual process used in the projects using COTS Components. Data: –Process can be classified into three categories: Pure waterfall Waterfall with prototyping Incremental mixted with prototyping –The main process was decided before they started to think about using COTS component or not. Answer: the so-called COTS-based development process is to customize the traditional development process due to the use of COTS components.

10 SPIKE Seminar Results - Answers to research question RQ2 RQ2:What are the commonalities and possible variations in COTS-based development processes. Data: Commonalities –Some new activites added Build vs. buy decision COTS component selection Learn and understand COTS component Build glueware and addware –New roles added COTS component knowledge keeper (12 projects)

11 SPIKE Seminar Results - Answers to research question RQ2 (cont’) Data: Variations –The phase to make the build vs. buy decision and the COTS component selection –The COTS component selection and evaluation process SP_fam: Familiarity-based selection process SP_unfam: Internet search, trial-based selection process –Step 1: First, developers used a search engine to browse the Internet, using keywords to express decided functionality. This usually gave them a handful of candidates. –Step 2: If there were more than 2 to 3 possible candidates, they selected 2 to 3 of them very quickly based on some key issues, such as license issues, cost and vendor reputation etc. –Step 3: After a small number of candidate COTS components had been decided, they downloaded a demo version of these from the web and test the COTS components.

12 SPIKE Seminar Results - Answers to research question RQ2 (cont’) Answer: There are some common new activities and one new role added in the COTS-based development process. The possible variations are when and how to perform these new activities

13 SPIKE Seminar Results - Answers to research question RQ3 RQ3: What are process scenarios of the projects using COTS components successfully and not successfully? What are the relationship between possible risks and process scenarios Data: –12 successful projects and 4 unsuccessful projects –6 Scenarios –6 Lessons learned

14 SPIKE Seminar Results - Answers to research question RQ3 (cont’) IDCharacterizationsLesson learned Sc1Two projects used waterfall processes. Made the build vs. buy decision and the COTS component selection in the implementation phase. Selected unfamiliar COTS components. Lesson 1: The COTS component selection should be implemented no later than the design phase in the waterfall processes. Lesson 2: They should inform the customer after they decided to use COTS components. Sc2One project used waterfall process. Selected COTS components in design phase but mainly on their functionalities. Selected unfamiliar COTS components. Lesson 3: They should consider more on how to integrate the COTS components. If the COTS components were hard to be integrated, they should build alternative components themselves. Sc3One project used incremental & prototyping process. Selected the COTS components in design phase but mainly on their functionalities. Selected unfamiliar COTS components. Lesson 3: See above Scenarios of 4 unsuccessful projects

15 SPIKE Seminar Results - Answers to research question RQ3 (cont’) Scenarios of 12 successful projects IDCharacterizationsLesson learned Sc4Two projects used waterfall process. Made the build vs. buy decision and selected COTS components in the requirement phase. Selected familiar COTS components. Lesson 4: Selected the COTS components mainly based on the easiness of integration. Sc5One project used waterfall with some prototyping process. Made the build vs. buy decision and selected COTS components in design phase. Selected familiar COTS components. Lesson 4: See above Sc6Nine projects used incremental with prototyping processes. Seven projects selected COTS components in requirement with familiar COTS components. Two projects selected COTS component in design phase with unfamiliar COTS components. Lesson 5: Using COTS components in the incremental & prototyping process helped the success of the project. Lesson 6: Rank the integration of unfamiliar COTS components as high risk task and integrate such components before other components.

16 SPIKE Seminar Results - Answers to research question RQ3 (cont’) Answer: COTS-based development could be performed successfully using waterfall and evolutionary processes. However, different process scenairos may face different risks.

17 SPIKE Seminar Variations in COTS-based Software development Processes SP_fam: Familiarity-based selection process SP_unfam: Internet search, trial-based selection process Possible COTS-based development process customizations

18 SPIKE Seminar Discussions Comparing with related work –Boehm et al. –EPIC & NASA process –Proposed COTS selection processes Possible threats to validity –Internal validity –External validity –Construct validity –Conclusion validity

19 SPIKE Seminar Conclusions Use of COTS components can be done as part of traditional development processes (e.g. waterfall and evolutionary) – there is no special “COTS-based development process”. Successful use of COTS components in such processes, however, requires that some new activities and roles are introduced in order to reduce risks. Typical new activities are the build vs. buy decision, COTS component selection, and COTS component integration. A new role is that of a knowledge keeper. Two COTS component selection processes have been verified in practice, i.e. familiarity-based selection process and process combining Internet searches with hands- on trials. Two of the new activities, the build vs. buy decision and the COTS component selection, can be placed in different development phases (requirements, design, or implementation). This depends on project context, especially on the familiarity with possible COTS components and the flexibility of requirements. A set of explanatory scenarios and guidelines have been synthesized to assist in the customization of the traditional development processes with the new activities and roles.

20 SPIKE Seminar Section 2: Risk management in OTS (COTS and Open Source) based development State-of-the-art Research motivation and research questions Research methods Results Discussions

21 SPIKE Seminar State-of-the-art OTS component based development processes Risks in component-based development –Component functionalities may not sufficiently match system requirements. –Whenever an assembler employs a component from a less-reputable source, it is difficult to gauge the quality of the corresponding component-based system. –Component-based assembly requires personnel with a different set of skills than the one employed in traditional life-cycle development etc. Risks in COTS component based development –Insufficient effort planned for integration, i.e. glue-code development and testing. –COTS components could not be sufficiently adapted to changing system requirements. –Ineffective management of the vendor relationship etc. Risks in Open Source Component based development –Security –Quality etc.

22 SPIKE Seminar State-of-the-art (cont’) Proposed risk management strategies –Involving customers into the “acquire” vs. “build” decision. –Significant quality features must be tested before finally selecting OTS components. –A most likely OTS component learning curve must be accounted for in cost estimation. –Involving OTS-knowledgeable individuals in all analytical processes. –Using a stringent configuration control etc.

23 SPIKE Seminar Research framework Decide the whole development process (When and how? RQ2) Make vs. Buy decision (Why? RQ1) OTS selection (When and how? RQ2) OTS integration Have found the proper OTS ? OTS maintenance and evolution Y Start project level risk mitigation (RQ3) Y N Non-OTS process Decide to use OTS ? N Start OTS level risk management (RQ4) Start project level risk identification and evaluation (RQ3) P1 P2 P3 P4

24 SPIKE Seminar Research questions RQ1: What are the main motivations of using OTS components instead of building in-house? RQ2: What are the real common OTS-based process models, and what are the possible variances? RQ3: What project-level risks have been identified and their relationships with the project contexts? Which risk reduction methods have been performed from the project level and the results? RQ4: What OTS component-level risks have been identified and their relationships with OTS component characteristics? Which risk reduction methods have been done for specific OTS components and the results?

25 SPIKE Seminar Research methods Survey (questionnaire) Sample selection strategies –Norway We gathered a company list from the Norwegian “Census Bureau ”. We included mostly companies which were registered as IT companies. Based on the number of employees, we selected the 115 largest IT companies (100 IT companies and 15 IT departments in the largest 3 companies in 5 other sectors), 150 medium-sized software companies (20-99 employees), and 100 small-sized companies (5-19 employees) as the original contacting list. –Italy We first got software companies from yellow pages. We then randomly selected companies from them. For these randomly selected companies, we read their web-site to ensure they are software companies or not. 196 companies were finally clarified as software companies, and were used as the original contacting list. –Germany We selected name list from a company list from an organization similar to the Norwegian “Census Bureau”. We then used the existing IESE customer database to get contact information of relevant IT/Software companies, in line with the Norwegian selection.

26 SPIKE Seminar Research methods (cont’) Data collection procedure –The final questionnaire was first designed and pre-tested in English (internal and external preview). It was then translated into the native languages and published on the SESE web survey tool at Simula Research Lab. –Possible respondents were contacted by telephone first. If they have suitable OTS-based projects and would like to join our study, a username and password will be sent to them, so that they can log into the SESE web tool to fill in the questionnaire (they can also use paper version). –The respondents who didn’t want to answer the questionnaire was also registered so that we could find the underlying reasons, e.g., no OTS- based projects, “busy”, not interested etc.

27 SPIKE Seminar Preliminary results Data collection is still on-going 15 filled-in questionnaire (9 from Norway, 6 from Italy) till the middle of August. Summarized the answers to RQ3 based on current data.

28 SPIKE Seminar Answers to RQ3 - Project level risks IDRisksRelative frequency (1..5) RgRequirements were changed a lot3.33 RbEffort to select OTS components was not satisfactorily estimated2.79 RoProvider did not provide enough technical support/ training2.33 RcEffort to integrate OTS components was not satisfactorily estimated2.29 RlIt was difficult to update the system with the last OTS component version2.29 RaThe project was delivered long after schedule2.20 RhOTS components could not be sufficiently adapted to changing requirements2.20 RiProject could not (re)negotiate requirements with the customer, if OTS components could not satisfy all requirements 2.08 RjIt was difficult to identify whether defects were inside or outside the OTS components1.93 RkIt was difficult to plan system maintenance, e.g. because different OTS components had asynchronous release cycles 1.93 RfOTS components negatively affected system performance1.87 RnInformation on the reputation and technical support ability of provider were inadequate1.85 RmOTS components were not satisfactorily compatible with the production environment when the system was deployed 1.80 RdOTS components negatively affected system reliability1.60 ReOTS components negatively affected system security1.50

29 SPIKE Seminar Answers to RQ3 (cont’) – Project level risk management IDRisk management actionRelative frequency MhDid integration testing incrementally (after each OTS component was integrated3.64 MdOTS components qualities (reliability, security etc.) were seriously considered in the selection process3.33 MeEffort in learning OTS component was seriously considered in effort estimation2.87 MaCustomer had been actively involved in the “acquire” vs. “build” decision of OTS components2.73 MfEffort in black-box testing of OTS components was seriously considered in effort estimation2.46 MbCustomer had been actively involved in OTS component selection2.33 MgUnfamiliar OTS components were integrated first2.29 MjMaintained a continual watch on the market and looked for possible substitute components2.20 MiLocal OTS-experts actively followed updates of OTS components and possible consequences2.14 McOTS components were selected mainly based on architecture and standards compliance, instead of expected functionality 2.13 MkMaintained a continual watch on provider support ability and reputation.2.07

30 SPIKE Seminar Answers to RQ3 (cont’) –Relationships between risks and risk management RisksRisk management actionsCorrelationP-value RaMa, RaMb RisksRisk management actionsCorrelationP-value RdMd RdMh RisksRisk management actionsCorrelationP-value RhMa RhMb Cost-estimation risk Quality risk Scope and requirement risk Somers’d in SPSS Version 11.0

31 SPIKE Seminar Preliminary conclusions Comparing with related work –The scope and requirement risks are the relative most frequent risks. Our results also show that customer involvement can help to reduce the scope and requirement risk. Customers can not only be involved into the “acquire” vs. “build” decision, they can also be involved in the selection of OTS component. –Estimation of selection and integration costs in OTS-based projects is perceived challenging. Most proposed risk mitigation methods focus on solving this problem by experienced project manager or a formal cost- estimation model. Our results show that customer involvement may help make it easy to estimate. –Most previous studies regard quality (reliability, security, or performance) risks are very challenging in OTS component-based development. Our results show that they are not as frequent as assumed. Project managers used careful selection and incremental testing to help to mitigate OTS components’ negative effect on the quality of the system. –For maintenance risks, these are perceived important but with less level of control.

32 SPIKE Seminar Discussions Methodology lessons learned –Definition of IT companies? –Sample selection (Representative IT companies may not be the representative sample of OTS component-development) –Compare the representativeness of different selection strategies in different countries –Motivation of survey? –Compare with structure interview

33 SPIKE Seminar Questions & Comments ?