1. COTS Based Systems: Benefits, Potential Risks and Mitigation Techniques Ronald J. Kohl Chair, GEIA IT&I TC Titan Systems Co rjkohl@prodigy.net

2. Contents Definitions Benefits of COTS Risks with Using COTS Mitigation approaches Some final thoughts

3. Definitions Commercial Off The Shelf (COTS): Commercially available product acquired in ‘as is’ condition, typically with built-in ‘tailoring’ capabilities but typically with no source code access Other Non-Developed Item (NDI) types MOTS (Modified Off The Shelf) GOTS (Government Off The Shelf) Reuse products, shareware, Open Source Custom: Developed software, control of source code and development team

4. Definitions Wrapper: Custom built software that provides an interface between a COTS product and some other software Application Service Provider (ASP): a new business model that allows for the ‘rental’ of COTS capabilities, in lieu of purchasing the COTS product Enterprise Application Integration (EAI): the activities needed to integrate a variety of products (COTS, etc) and services (ASP, etc) in support of creating an Enterprise solution

5. Kohl’s Lemma Using COTS products in lieu of custom built software is a trade between control of... Faster and cheaper vs Better (but ‘cheaper’ is under review!)

6. Potential Benefits of Using COTS Reduced development cost/schedule Reduced maintenance cost(?) Product ‘improvements’ paid by the vendor “Proven” product Wide user base to identify problems Wide user base to build operational life Available skill base Industry investment in technology base

7. Potential Risk Areas

8. CHART 8 COTS Risks ‘ Make vs Buy’ decision has become the ‘Make vs Buy vs Rent’ decision Some projects have abandoned the ‘make vs buy’ decision process (assumes that COTS is the way to go) Introduction of ASPs further complexifies this decision Different lifecycle model (vs custom SW) Requirements development (driven by COTS features) Testing (black box vs white box) Systems evolution vs product evolution (could diverge!) Maintenance (product volatility, vendor supportability)

9. CHART 9 COTS Risks Product Volatility product changes when the Vendor chooses driven by market forces, not necessarily your needs not aligned with your fiscal cycles product lifetime usually much less than system lifetime No/little insight into product or process Product flaws, documentation, source code, test reports Development processes, tools or skills can conflict with CMM Level 3 Acquisition requirements

10. CHART 10 COTS Risks System incompatibilities – H/W Platforms – Business processes – Operational environment Underestimated total program costs Integration Verification and Validation Training O&M

11. COTS Risks (con’t) Vendor viability – as a business entity – as a technology provider – susceptible to acquisition, by off-shore companies? Risk to maintenance Vendor support and stability Vendor dependency on error reporting/fixing Likelihood of ‘wrappers’ Unique operational environment Stringent program requirements/needs

12. COTS Risks (con’t) Testing is sometimes inadequate – total systems testing still mandatory – product testing should be across lifecycle Multiple COTS product integration Interoperability (plays well together) Overlap Code/Dormant Code Product problem determination, responsibility Increase in complexity of CM Increase in complexity of system evolution Increase in complexity/cost of wrappers Total Cost of Ownership

13. Mitigation Techniques

14. Employ good systems engineering practices – these are still large, complex systems – they contain many risks Conduct a ‘make vs buy (vs rent?)’ trade study – do not assume that ‘buy’ is the only viable option – let your needs define the criteria for such a study – included full lifecycle concerns (development, O&M)

15. Mitigation Techniques Gain Marketplace and vendor knowledge – conduct business viability analysis (early and often) – gain confidence in vendor’s quality (early and often) – identify alternate vendors (early and often) – may require a new role within project and/or enterprise! Gain product knowledge Learn all you can, as early as you can, however you can Shop early & often, allow for requirements changes Identify usage in similar applications (ask vendor, UG involvement, web sources, etc) may require a new role within project

16. Mitigation Techniques (con’t) Build around ‘flagship’ COTS product –typical in ERP systems –Reduces multiple vendor issues Early vendor involvement throughout the life cycle –Typical in both ERP and engineering systems Overall robust system design that can withstand the unexpected and ‘weak links’ Robust verification plan and environment

17. Mitigation Techniques (con’t) Contractually require product ‘insight’ requirements – problem reports, test plans/scripts, etc – product development processes – skill base –Don’t be surprised if vendors balk Product and/or Vendor certification ? –Can be the GO/NOGO decision in Safety Critical systems Source code escrow –Is almost always an expensive option!

18. System Architecting Heuristics Focus on those things which must change the least –Know where to put the basement, very early –Know where to put the load bearing walls, early –Design the whole building but don’t build the 22nd floor before the basement Understand those things which will likely change the most –new technologies (COTS, HCIs, Processors) –Political impacts (new admin, in my district, etc) –decide/baseline/ buy as late as possible, but still support the project schedule

19. Summary Employ good systems engineering Shop early, shop often Test Drive early, often Know the vendor, early and often Know the marketplace Know your options Acquire the right skills for COTS-based systems Defer final decision/purchase as late as possible! But buy only when you are a fully informed buyer!!