1. COTS Project Types CeBASE COTS research Ye Yang, Jesal Bhuta, Dan Port

2. COTS Definition For the purposes of these classifications, we use the following definition of COTS “A software product, developed by a third party (who controls its ongoing support and evolution), bought, licensed, or acquired for the purposes of integration into a larger system as an integral part, i.e. that will be delivered as part of the system to the customer of that system (i.e. not a tool), which might or might not allow modification at the source code level, but may include mechanisms for customization, and is bought and used by a significant number of systems developers.”

3. MBASE Milestone Elements

4. MBASE Milestone Elements (2)

5. COTS Based Systems A COTS Based System (CBS) is any system that more than 50% of a systems functional requirements are implemented through use of COTS via glue code, tailoring, and assessment effort. –Glue code is the custom development needed to integrate COTS packages within an application external to the packages themselves –Tailoring effort is the activities relating to adapting COTS packages internally for use within an application –Assessment effort is the suitability evaluation and analysis of the desired attributes of the COTS packages for an application * The above definitions are compatible COCOTS definitions

7. CBS Approaches There are a variety of approaches to developing COTS based systems (CBS). –Find and use COTS packages as is (“turnkey”) –Find and adapt COTS packages as needed (“adaptation”) –Use COTS as components in custom built application (“integration”) These approaches differ considerably and driven primarily by the systems shared vision and desired characteristics & priorities (DC&P’s). –Effort distributed between Glue code Assessment Tailoring

8. CS577B COTS Effort Distribution Source: CS577 Project Effort Data 2000-2002

9. CBS Types Roughly, a CBS project can be classified as either Assessment Intensive Tailoring Intensive Application Intensive These classifications provide an overview of the potential risks, attributes, and development effort priorities. –Use for strategic and tactical development planning

10. Assessment Intensive An assessment intensive CBS project will have –The DC&P’s covered without much modification (or tailoring) by a single COTS framework or through the simple integration of several. –The primary effort is focused on identifying collection of candidate COTS products and evaluating (i.e. assessing) a feasible set of products whose existing capabilities directly address a desired operational concept.

11. Tailoring intensive CBS A tailoring intensive CBS project is where –The DC&P’s are covered by a single (or very few) COTS framework(s). –The primary effort is on adapting a COTS framework whose existing general capabilities can be customized (i.e. tailored) in a feasible way to satisfy the majority of a systems capabilities or operational scenarios.

12. Application Intensive The application intensive CBS project will have –Some nontrivial DC&P’s covered by COTS, and some that must be covered by custom development. –The primary effort is to identify COTS packages that can feasibly used as components to address subsets of the required system capabilities. –Typically there are significant “buy versus build” decisions Critical factors are: available COTS products, schedule limitations, budget limitations, and desired current and future capabilities

13. Effects of CBS Types The CBS type effects many aspects of a project –MBASE development guidelines –Glue code, assessment, tailoring effort –Risk factors –Requirements flexibility –System evolution and maintenance –Cost factors

14. CBS project Type Guidelines System shared vision Desired characteristics & priorities (DC&P’s) System shared vision Desired characteristics & priorities (DC&P’s) Relation of candidate COTS products to DC&P’s Standard MBASE Application Guidelines Application -Intensive CBS Guidelines Tailoring- Intensive CBS Guidelines Assessment -Intensive CBS Guidelines I IIIIIIV

15. CBS project Type Selection Matrix Evaluate with respect to business case and organization attributes

16. Assessment Intensive CBS Ye Yang

17. Assessment Intensive CBS Assumption: –COTS packages available to cover most of desired system capabilities identified according to system shared vision among stakeholders. –A considerable number of COTS package combination needs to be assessed against established evaluation criteria.

18. Assessment Intensive CBS- Characteristics 1 The primary effort is focused on identifying collection of candidate COTS products and evaluating a feasible set of products whose existing capabilities directly address a desired operational concept. –About 60-80% of COTS related effort is assessment effort, and accounts for 14% of overall effort.

19. Assessment Intensive CBS- Characteristics 2 Need high flexibility in requirements and design due to COTS uncertainty –COTS capabilities compose requirements –Requirements cannot be fixed (or “hard”) before final COTS selection –We use Desired Characteristics & Priorities (DC&P’s) to guide requirements gathering and flexibility

20. Assessment Intensive CBS- Characteristics 3 System architecture is focused on a COTS package configuration model, since the class model, object model and interaction model are not available for most COTS products. –Example: Rework effort for CS577 2001-2002 team8’s SSAD

21. Assessment Intensive CBS- Characteristics 4 A business case analysis for each COTS solution is highly recommended for the detailed COTS assessment –COCOTS Model for effort and schedule estimation Assessment, glue code, tailoring effort tradeoffs –Return on Investment analysis Value, risk, cost tradeoffs –COTS specific risks and mitigation costs vendor volatility, upgrade lifecycle, etc. Example: Dot-Project infeasibility in CS577A 2001 team 8

22. Identifying Assessment CBS You have an assessment intensive CBS when you have the previously stated characteristics. –High requirement volatility, high domain adaptability, low developer resources… –Business case shows feasibility of focusing effort on COTS assessment (rather than glue code, tailoring, or build from scratch)

23. Assessment Intensive CBS Project Guidelines Assessment -Intensive CBS Assess COTS candidates with respects to DC&P’s Develop glue code; Tailor COTS mix to DC&P’s Integrate, test, transition IOC Evolve as tailored COTS mix Follow Tailoring-Intensive COTS Guideline Singl e COT S Yes No, COTS Mix Solution

24. Example Project profile –USC Collaborative Services System (team 8 CS577 2001-2002) DC&P’S –Project management –File management –Online Discussion Board –Group Calendar

25. Example- COTS Assessment Steps Plan: set the evaluation effort scope and provide a baseline for evaluation process. COTS Identification: search for candidate COTS packages based on the set of DC&P’s Establish evaluation criteria based on functional, performance, architectural, and financial characteristics of DC&P’s Conduct multiple cycle of evaluation to collect evaluation data against the defined evaluation criteria. Perform Hand-on Experiments on COTS to verify vendor’s claim Compare and contrast the evaluation data and make COTS final selection.

26. Major risks-1 Many new non-technical activities introduced into software engineering process –Establishing evaluation criteria, –Planning and executing evaluation procedures, –Collecting and analyzing data from evaluation process, –Making COTS recommendations, which require training and experience.

27. Major Risks-2 Within the COTS evaluation process: –You may miss some possible COTS candidates. Stay as broad as possible when doing the initial search for candidates. –You might not include all key aspects for establishing the evaluation criteria set. Judgment of weight distributions require experienced and knowledgeable people –Introducing new COTS candidates is likely and require re-planning or contingency planning. Limits on schedule and budget must be accounted for

28. Major Risks-3 You may not get all the features of certain COTS products as advertised by its vendor. –Detailed assessment beyond literature review or vendor provided documentation should be performed in the form of hands-on experiments.

29. Example - Comparison of results from initial assessment and detailed assessment Evaluation RequirementCOTS candidate 1- eProject COTS candidate 2- Blackboard COTS candidate 3- iPlanet collaborative package InitialDetailedInitialDetailedInitialDetailed Users can upload/modify/download/delete authorized files ********* *None Project management, including planning group tasks, timelines and tracking ************None Message Board*************None User Interface********* * Admin Interface***************None Group Calendar********* * Detailed analysis provides greater assurance of COTS characteristics with respect to vendor documentation (although at significant effort)

30. Major Risks-4 The organization may not have the ability or willingness to accept the impact on its operation due to imposed COTS requirements. –Example: CS577 2001-2002 team 8: –COTS: Blackboard –Required operational change: User access mode

31. Major Risks-5 Difficulty in finding suitable time for key personal meetings to discuss COTS assessment may incur significant and unpredictable schedule delay –Based on analysis from student project weekly effort reports, team interaction and client interaction are the top-2 activities for assessment intensive CBS projects.

32. Tailoring Intensive CBS Jesal Bhuta

33. Tailoring Intensive CBS Assumption: –COTS packages have some sort of a tailoring/development interface using which the COTS package can be adapted to the DC&P’s –Little or no assessment is required (if required it is already complete) in selecting the COTS package CS 577 2001 – 2002 –Team 20 (577 MBASE in BORE) –Team 21 (Quality Management Through BORE)

34. Tailoring Intensive CBS Characteristic 1 The primary effort is on tailoring the existing adaptable capabilities of the COTS package to satisfy the majority of a systems core capabilities or operational scenarios. –CS 577 2001 – 2002 –Team 20 (577 MBASE in BORE) –Team 21 (Quality Management Through BORE)

35. Tailoring Intensive CBS Characteristic 2 Need moderate flexibility in requirements and design due to limitations in adaptability of COTS packages –Capabilities (and adaptability) of the COTS packages may only partially satisfy the system requirements –System requirements may need to be adapted to fit within COTS limitations Significant source of risk, focus on DC&P’s to manage requirements change Example: QA in BORE security levels requirement change

36. Tailoring Intensive CBS Characteristic 3 The focus of the system architecture depends upon the type of the COTS package tailoring required in order to satisfy the DC&P’s –GUI Based Tailoring (ex. Spearmint) No focus on system architecture –Parameter Based Tailoring (ex. Windows Media Player) Little or no focus on system architecture –Programmable based Tailoring Interface (ex. Hyperwave) Moderate focus on system architecture

37. Tailoring Intensive CBS – Characteristic 4 The Tailoring intensive CBS development usually require some amount of time for developer training to get acquainted to the COTS package. –A Learning Curve must be accounted for during the system development process

38. Identifying Tailoring CBS You have an tailoring intensive CBS when you have the previously stated characteristics. –Business case shows feasibility of buying and tailoring COTS packages to implement major system capabilities

39. Tailoring Intensive CBS Project Guide Tailoring- Intensive CBS Tailor COTS Framework to DC&P’s Test, Transition, IOC Test, Transition, IOC Evolve as Tailoring- Intensive CBS Get acquainted with the COTS framework Ensure that the core features to be used, function at an acceptable level

40. Risk 1 Version change may result in re-tailoring of COTS product –A constant feedback by the client to the vendor about the features used and enhancements required may help in ensuring that the features used by the current CBS continue in the future versions of the project Ex. “The LOGO element specifies a graphic file to display as a logo. Not supported after Windows Media Player version 6.4.”

41. Risk 2 Inadequate vendor support may result in significant project delays –Hiring consultants with experience in using COTS product (Non-CS577 mitigation) –Ensuring an efficient Phone/Email support at least during the development phase

42. Risk 3 Faulty Vendor claims (COTS “surprises”) may result in Significant Delays (Ex) Oracle intermedia claims to search for all unicode characters however the search is not carried as specified –Pre-testing of core features to be used before beginning of the actual development

43. Risk 4 COTS package incompatibilities (integration clash) –In case of multiple COTS packages being used to implement the DC&P’s there may exist incompatibilities between COTS solutions Pre-testing of compatibilities between packages to be used before beginning of the actual development

44. Risk 5 Added complexity of un-used COTS capabilities –Adaptable and Tailoring capable COTS packages usually come with additional complexity and cost that is incurred in the process of making the package tailorable Ex Microsoft Office Package

45. Assessment/Tailoring Intensive CBS Life Cycle and Mbase Inception Identify the Type of CBS Narrow the list of Vendors depending on available information Elaboration Develop and Test Functional Prototypes to ensure the functioning Core Capabilities Collect Detailed Information and Evaluate the COTS vendors short-list Construction 1 Obtain a shared Vision amongst stakeholders. Identify why to go CBS (window of opportunity, independence from evolutionary maintenance, …). Develop and collect Information on available COTS product in the market (RFI). Make a list of products which can satisfy the Core requirements and undergo a primary evaluation them LCOLCACCDIOC Construction 2 Train the Developers to use the COTS product Implement and test the Core requirements of the system Implement the low priority requirements of the system Support personnel must be trained to work with the COTS API Maintaining constant contact with the developer in terms of specifying evolution requirements Identify the COTS product (s) that shall implement the system

46. Application Intensive CBS Dan Port

47. Application Intensive CBS Assumptions: –COTS packages are available to satisfy significantly valuable subsets of system capabilities or project limitations dictate use (e.g. schedule, skill, or complexity factors) Buy vs. build cost-benefit ROI –Integration overhead is minimal with respect to custom development –Low assessment and tailoring effort required

48. Assessment Intensive CBS- Characteristics 1 The primary effort is focused on identifying system components and requirements that may be risky to custom implement and mapping them to a collection of candidate COTS products. –CS577 2001-2002 Team 15 DART’s use of MySQL and Tomcat

49. Application Intensive CBS- Characteristics 2 COTS expected to implement system requirements Low requirements volatility and adaptability Significant glue code effort expected COTS packages used are not specialized to a particular application domain (generic capabilities) COTS components dependent on application to function Many possible COTS packages may apply Many evolutionary requirements with respect to custom components and anticipated COTS features

50. Identifying Application CBS You have an assessment intensive CBS when you have the previously stated characteristics. –Business case shows feasibility of buying components to implement major system capabilities and focusing effort on COTS integration (glue code)

51. Application Intensive CBS Project Guidelines Application- Intensive CBS Tailor COTS Develop glue code, integrate COTS, custom components Choose best mix of COTS, custom components Develop custom components Integrate, test, transition IOC Assess COTS candidates with respects to DC&P’s Evolve as application- intensive CBS

52. Example Project profile –CS577 Team 15 DART 2001-2002 –Tomcat, MySQL Team 17 Web-mail 2000-2001 –Tomcat Team 8 Fulltext Title Search 2000-2001 –MySQL, Tomcat, Zope Team 9 Student/Staff Directories 2001-2002 –Zope

53. Major risks-1 Spending too much effort locating appropriate COTS products (too little value for cost) Overly optimistic expectation of COTS quality attributes COTS package incompatibilities (integration clash) Added complexity of un-used COTS capabilities Imposed black box testing of COTS components Inadequate COTS assessments

54. Major risks-2 Overly optimistic COTS package learning curve Compromising hard requirements to make use of particular COTS COTS “surprises”, capability shortfalls Application dependent on COTS vendors (legacy support, upgrades, bugs, etc.)

55. CBS Type Comparisons

56. 1Team Interaction7Feasibility Rationale Document 2Client Interaction8COTS Final Selection 3COTS Initial Filtering9Control and Monitoring 4Life Cycle Planning10COTS Tailoring 5Project Web-site11Transition and Support Planning 6Training and Preparation12Critical Component Implementation Source: CS577 2001-2002 Project Effort Data

57. CBS project Type Comparisons