Determining the Suitability of COTS in Mission Critical Systems VT/NVC Feb. 15, 2002 Ronald J. Kohl Chief Systems Engineer Titan Systems Co.

Contents Definitions Why use COTS Risks with Using COTS What’s all the fuss with Mission Critical? Mitigation approaches Summary

Definitions Commercial Off The Shelf (COTS) Commercially available product acquired in ‘as is’ condition, perhaps with ‘tailoring’ capabilities Other Non-Developed Item (NDI) types MOTS (Modified Off The Shelf) GOTS (Government Off The Shelf) Reuse products shareware Open Source Custom Home grown or home maintained, control of source code and development team

What is ‘Mission Critical’? Everyone has a definition They’re all different, but similar Ron’s Defn: –Those parts of an enterprise or system which are essential to the success of that enterprise or system Could be Hardware, Software, Procedures or People!!!

So what’s different about Mission Critical Systems? Non-critical –meet functional rqmts –meet performance rqmts, often –be available, often –work correctly most of the time –maintainable –ease of recovery Mission Critical –meet all rqmts and nothing more –do so, all the time –be available, all the time –work correctly, always –quality of maintenance –rigorous recovery requirements

Potential Benefits of Using COTS Reduced development cost/schedule Reduced maintenance cost Product ‘improvements’ at no cost “Proven” product Wide user base to identify problems Wide user base to build shelf life Available skill base Industry investment in technology base

CHART 7 COTS Risks Product Volatility product features change when and to what the Vendor chooses No/little insight into product * Unknown product flaws Limited, poor, or no documentation No source code Unknown development processes or skills May not meet program requirements * Product features not as advertised (more or fewer) Product not suited for intended operational use Difficult to find human-life rated products Product lifetime may be less than program life

COTS Risks (con’t) Underestimated total program costs Integration costs, Verification costs and O&M costs Risk to maintenance Unpredictable vendor support and vendor stability Dependency on vendor to identify flaws that are applicable to program Vendor resistant to accepting/fixing externally identified flaws (requires “proof”) High probability of mods or ‘wrappers’ * Interfaces/protocol not standard with industry Unique operational environment Stringent program requirements/needs

Mitigation Techniques Gain Marketplace and vendor knowledge – shop early & often Gain product knowledge prior to baselining requirements - Learn all you can, as early as you can, however you can COTS standards for program Use of redundant vendors Early vendor involvement throughout the life cycle Product and/or Vendor certification *

Mitigation Techniques (con’t) Robust verification plan and environment * Early prototyping, allowing time for design/requirements changes Overall robust system that can withstand the unexpected Source code escrow * Up front systems engineering evaluations Product ‘insight’ requirements * Product simulators/models *

Summary: COTS Evaluation Guidelines Maturity of COTS marketplace in the domain Stable/quality vendors in this marketplace Quality products in this marketplace Similar usage of this COTS in related applications Certifiable for Critical Applications Insights into development products and processes Fidelity of product simulations/models Product Change/Maintenance Plans Quality of and alternatives for product support

Some additional heuristics Prioritize requirements Recognize the basement requirements from the roof requirements Recognize system elements that may be ‘volatile’, make final decisions as late as possible Establish acceptance criteria, early! Employ effective analytical methods

Some ‘open problems’ Complete description of the COTS-based systems development and operational life- cycle model Effective cost estimation algorithms More objective eval/selection criteria and methods Verification/validation methods