© 2003 by Carnegie Mellon University page 1 Process Principles for COTS-Based Systems 5 September 2003 Software Engineering Institute Carnegie Mellon University Pittsburgh PA Sponsored by the U.S. Department of Defense Tricia Oberndorf

© 2003 by Carnegie Mellon University page 2 Agenda COTS Background A/APCS Framework CMMI Background Implications of Using CMMI for COTS-Based Systems

© 2003 by Carnegie Mellon University page 3 COTS Background Building systems today: Few entirely custom-built to order Commercial products expected to play major role Diverse things influence the shape and function of a COTS-based system (CBS): Stakeholder needs Product characteristics and marketplace dynamics Degree of interaction with legacy systems Together, these compel many changes in system development and maintenance methods for CBSs.

© 2003 by Carnegie Mellon University page 4 What is a COTS-based system? A COTS product is one that is … Sold, leased, or licensed to the general public Offered by a vendor trying to profit from it Is supported and evolved by the vendor, who retains intellectual property rights Available in multiple, identical copies Used without modification of its internals A COTS-based system is one that … Uses one or more off-the-shelf components from one or more suppliers plus any needed custom components Is integrated to achieve new or expanded system functionality Legacy Reuse assets COTS products Custom

© 2003 by Carnegie Mellon University page 5 Conceptual Bases: Requirements 1 “Nail down the requirements first” will not work – the marketplace will not cooperate Think of it as a new sphere of influence: Constraints come from stakeholder needs AND marketplace imperatives System that is created must accommodate competing sources of gravity

© 2003 by Carnegie Mellon University page 6 Conceptual Bases: Requirements 2 Requirements (cont.) Must distinguish between negotiable and non- negotiable Keep the non-negotiable set as small as possible Understand how to prioritize and trade-off the negotiables A risk-driven spiral approach is key: Frequent use of prototypes Considerable interaction with system’s end users Gradual refinement of understanding of system

© 2003 by Carnegie Mellon University page 7 Conceptual Bases: Knowledge CBS development and maintenance is dependent on an evolving Body of Knowledge. Architecture and design constraints End-user business processes Acquisition constraints BoK Prioritized negotiablesMarketplace offerings Evolving System View Non-negotiables Legacy system context Risks

© 2003 by Carnegie Mellon University page 8 Marketplace Affects CBS Approach Traditional Engineering Approach Requirements Architecture & Design Implementation Requirements-drivenNegotiation-driven Required CBS Approach Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk

© 2003 by Carnegie Mellon University page 9 The Process Framework Based on these realizations and concepts, we have defined a framework that sets out the basics of a process that would be appropriate for the creation of a COTS- based system: A/APCS: The Assembly/Acquisition Process for COTS-based Systems

© 2003 by Carnegie Mellon University page 10 Process Overview Three classes of activities: Iterative: short-term, engineering-oriented -Discovery: gather and refine system knowledge -Assembly: construct a prototype -Assessment: determine success of iteration & plan Pervasive: long-term, organizational scope -e.g., CM, license management, vendor relationship management, contract tracking and oversight Executive: event-driven, deal with decision making -e.g., cost estimating, contract negotiation, project oversight Today we focus on the Iterative.

© 2003 by Carnegie Mellon University page 11 Discovery Gather and Refine activities occur simultaneously. Gather: Sources take many forms. -Stakeholders, business processes, legacy systems, COTS marketplace Each is independent of the others. There is no optimal order. Refine: Analysis and harmonization of the gathered knowledge Yields the technical definition of the emerging system Finds gaps, negotiates conflicts

© 2003 by Carnegie Mellon University page 12 Discovery Activities Data from: stakeholders products … Refine Gather Negotiate conflicts Gap - need more data Continue Discovery Knowledge is sufficient for constructing executable Go on? Harmonized data Mismatch Agreement Gather data z Gather data y Gather data x Analyze

© 2003 by Carnegie Mellon University page 13 Assembly Produces a prototype that reflects what’s been discovered Reflects a traditional sequential process Guided by local “ candidate requirements” -Iteration Objectives, Detailed Iteration Plan plus hypotheses and desired behaviors expect to derive from prototype -Vets “candidate requirements” Produces an executable version of full system -Likely to have less than complete functionality -Executable, not paper or specification or small piece Does not preclude prototyping “in the small” throughout

© 2003 by Carnegie Mellon University page 14 Evaluation and Assessment Occur at two levels: Evaluation of the prototype in its own context -Measures actual outcome against expected outcome -Considers degree to which prototype satisfies its local “requirements” Assessment of the iteration itself -Addresses issues at project (not iteration) level -Considers whether objectives were good ones -Determines whether iteration results indicate changes in project schedule, budget, etc.

© 2003 by Carnegie Mellon University page 15 On-going Iterations of A/APCS Prototype Deployed System Body of Knowledge Body of Knowledge BoK

© 2003 by Carnegie Mellon University page 16 Key Aspects of COTS Approach Simultaneous Definition and Trades Parallel Business Process Engineering Negotiable Requirements Flexible Architecture Current Market Knowledge Integral Programmatics and Risks Continuous Stakeholder Involvement Disciplined Spiral or Iterative Practices Frequent Executables

© 2003 by Carnegie Mellon University page 17 CMMI Guidance for improving an organization’s processes and ability to develop, acquire, and maintain products or services Provides guidance for developing processes is not processes or process descriptions will not map one to one with the processes in your organization Integrates four disciplines (bodies of knowledge) System Engineering Software Engineering Integrated Product and Process Development Supplier Sourcing Expects all practices to be interpreted using knowledge of CMMI the organization the business environment CMMI process areas must be interpreted for each targeted domain

© 2003 by Carnegie Mellon University page 18 Presentation Format Work Process Implications High-level guidance for selected CMMI Process Areas or disciplines -Highlight particular relevance -Provide unique interpretation -Suggest new process areas -Correct misconceptions Guidance for additional CMMI Process Areas for later reference Important process considerations for a COTS-based system approach Take-away message for each aspect Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk

© 2003 by Carnegie Mellon University page 19 Simultaneous Definition and Trades Selected Work Process Implications Decision Analysis and Resolution – continuous stakeholder negotiations require robust, agreed-upon decision processes Technical Solution – alternative solutions (including COTS product selection) are analyzed continuously to accommodate new information Risk Management – a key project risk is over-defining one sphere without adequate understanding of implications on other spheres Project Planning – project activities start concurrently with extensive interaction among them from project start until the system is retired Verification and Validation – continuous determination that information in each sphere is sufficient, complete, and meets operational needs Balanced consideration of four diverse spheres of influence - Simultaneous discovery of information in each - Decisions in one inform and constrain decisions in others - Continuous identification and analysis of trades - Trades negotiated as definition of the solution evolves Continuously reconcile what stakeholders want with what is available Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk

© 2003 by Carnegie Mellon University page 20 Parallel Business Process Engineering Selected Work Process Implications CMMI does not address changes to end-user business processes; expand concepts in Organizational Process Focus to plan and implement end-user business process improvement Organizational Environment for Integration – a shared vision of success among stakeholders, with suitable incentives and leadership, is critical to aligning business processes with alternative solutions Project Planning/Integrated Project Management for IPPD – implementing agreed upon business processes must be integrated in system planning End-users must be willing/able to change business processes to match those in COTS products Business process changes must be explicitly managed and coordinated as part of the project Business processes may continue to change with new COTS releases or new COTS products Align business process engineering with system engineering Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk

© 2003 by Carnegie Mellon University page 21 Negotiable Requirements Selected Work Process Implications Requirements Development – must prioritize requirements to aid tradeoffs -stakeholders agree on minimum set of critical “must-have” requirements -evolvability is a high-priority, required quality attribute Requirements Management – disciplined and controlled management of requirements begins at project start to track identified and negotiated trades Project Planning/Integrated Project Management for IPPD – managing the project to encourage and reinforce the continual discovery of requirements while establishing sufficient stability to deliver a solution is challenging Requirements must be flexible enough to leverage available and projected COTS products Committing to a requirement is premature until COTS product behavior is understood As marketplace continues to change, requirements must be renegotiated Requirements formation is a journey of discovery Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk

© 2003 by Carnegie Mellon University page 22 Flexible Architecture Selected Work Process Implications Technical Solution – each COTS product release requires evaluation and analysis of alternatives to support any incorporation decision -Describe behavior and linkage among system components for each Product Integration – composing and evaluating executable representations from project start is critical to verify and validate architecture suitability and evolvablity Project Planning/Integrated Project Management for IPPD – appropriate skills and resources are necessary to form, evaluate, and maintain system flexibility -include effort to create and maintain “wrappers” or “glue” and re-integrate solution as COTS products change Architecture is considered early; evolves and is maintained until the system retired Potential drivers of change accommodated in architecture Architecture is created and maintained as a corporate asset Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk

© 2003 by Carnegie Mellon University page 23 Current Market Knowledge Selected Work Process Implications Supplier Agreement Management – license agreements with COTS vendors must be negotiated to meet project needs Integrated Supplier Management – partnering with key vendors is critical (not just supplier management) -vendors seldom allow process monitoring; use hands-on evaluation -relationships with vendors’ other customers helps to amplify leverage Technical Solution – modification of COTS products introduces long term maintenance considerations and sizeable risk to project; avoid if possible Project Planning – significant resources may be required to monitor the marketplace and conduct COTS product evaluation; including experimentation facilities Anticipate and track changes to relevant market segments until system retired Anticipate and prototype system changes from updates to COTS products critical to the system Influence (not direct) COTS product changes, technology investments, and standards development Marketplace is proactively monitored Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk

© 2003 by Carnegie Mellon University page 24 Integral Programmatics and Risks Selected Work Process Implications Technical Solution – engineering trades should include risk, cost, schedule and other programmatic factors associated with each alternative solution Project Planning – estimates of work product and task attributes should be generated for each alternative Analysis of alternative solutions includes team skills and expertise, cost, schedule, and associated risks for -building, fielding, and supporting the system -implementing any needed changes to business processes (functional, operational, support) Programmatic factors shape technical alternatives Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk

© 2003 by Carnegie Mellon University page 25 Continuous Stakeholder Involvement Selected Work Process Implications IPPD discipline – integrated teaming among disparate stakeholders throughout the development and maintenance is essential Validation – end users must always be involved in validating solution suitability Integrated Project Management for IPPD – accommodates resources for stakeholder involvement -necessary changes to end-user operational processes must be explicitly and continuously managed and coordinated with solution development Required stakeholder commitment may be unprecedented Significant commitment from all stakeholders required -identify, evaluate, and select alternative solutions -confirm results of any and all negotiations -agree evolving system meets their needs Stakeholders must reflect full diversity of interests Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk

© 2003 by Carnegie Mellon University page 26 Disciplined Spiral or Iterative Practices Continuously determine a compatible and feasible set of: business processes, requirements, plans, design, COTS products, and other components -Enterprise business objectives drive solution definition -Risk considerations drive degree of detail -Marketplace dynamics drive development and maintenance processes Spiral development facilitates developing a viable solution Selected Work Process Implications Project Planning – if not already implemented, extensive effort may be needed to revamp planning and engineering processes for a spiral development approach Risk Management – tracking effectiveness of risk mitigation is key -highest priority remaining risks should be used to (re) direct and manage the project Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk

© 2003 by Carnegie Mellon University page 27 Frequent Executables Frequent executable representations reduce risk and reduce misunderstandings -Provide critical insight into the solution’s behavior -Explore critical system attributes -Validate end user business process -Verify technical viability Executable representations demonstrate stakeholder buy-in Selected Work Process Implications Requirements Development, Technical Solution, and Product Integration – each iteration should produce an executable representation reflecting current understanding of requirements, COTS products, business processes, and alternative designs explored and negotiated Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk

© 2003 by Carnegie Mellon University page 28 Conclusion CBS: changes many things about your approach requires even more discipline than custom development Although it needs interpretation, CMMI is a sound basis for CBS process improvement. Applying CMMI to CBSs is more than just Supplier Management: -All four disciplines -All process area categories Process areas will need to be integrated and applied differently Must add in your own activities to plan, design, and deploy changes to enterprise processes

© 2003 by Carnegie Mellon University page 29 For more information: TR coming: CMU/SEI-2003-TR Tricia OberndorfLisa Brownsword Director, Dynamic SystemsSr. MTS, CBS (ISIS)