1. © 2003 by Carnegie Mellon University page 1 Processes for COTS-Based Systems LA SPIN 3 September 2003 Software Engineering Institute Carnegie Mellon University Pittsburgh PA 15213 Sponsored by the U.S. Department of Defense Tricia Oberndorf

2. © 2003 by Carnegie Mellon University page 2 Outline Background Process Overview High-level Process Example Low-level Activities Descriptions Converting This to a Process: EPIC

3. © 2003 by Carnegie Mellon University page 3 Background Building systems today: Few custom-built to order Commercial products expected to play major role Diverse things influence the shape and function of a COTS-based system (CBS): Stakeholder needs Characteristics of products and marketplace Degree of interaction with legacy systems Together, these compel many changes in system development and maintenance methods for CBSs.

4. © 2003 by Carnegie Mellon University page 4 COTS Spheres of Influence Vendors Market segments Available COTS Products Emerging technology Marketplace Use cases Quality attributes Business model Stakeholder Needs / Business Processes Stakeholder Needs / Business Processes Patterns Interfaces Infrastructure Architecture/ Design Architecture/ Design Risk management Project management Change management License negotiation Programmatics/ Risk COSTS Programmatics/ Risk COSTS

5. © 2003 by Carnegie Mellon University page 5 Marketplace Affects CBS Approach Traditional Engineering Approach Requirements Architecture & Design Implementation Requirements-drivenNegotiation-driven Required CBS Approach Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture Design Programmatics/ Risk

6. © 2003 by Carnegie Mellon University page 6 Conceptual Bases: Requirements 1 “Nail down the requirements first” will not work – the marketplace will not cooperate Think of it as a new sphere of influence: Constraints come from stakeholder needs AND marketplace imperatives System that is created must accommodate competing sources of gravity

7. © 2003 by Carnegie Mellon University page 7 Conceptual Bases: Requirements 2 Requirements (cont.) Must distinguish between negotiable and non- negotiable Keep the non-negotiable set as small as possible Understand how to prioritize and trade-off the negotiables A risk-driven spiral approach is key: Frequent use of prototypes Considerable interaction with system’s end users Gradual refinement of understanding of system

8. © 2003 by Carnegie Mellon University page 8 Conceptual Bases: Knowledge CBS development and maintenance is dependent on an evolving Body of Knowledge. Architecture and design constraints End-user business processes Acquisition constraints BoK Prioritized negotiablesMarketplace offerings Evolving System View Non-negotiables Legacy system context Risks

9. © 2003 by Carnegie Mellon University page 9 Process Overview Three classes of activities: Iterative: short-term, engineering-oriented -Discovery: gather and refine system knowledge -Assembly: construct a prototype -Assessment: determine success of iteration & plan Pervasive: long-term, organizational scope -e.g., CM, license management, vendor relationship management, contract tracking and oversight Executive: event-driven, deal with decision making -e.g., cost estimating, contract negotiation, project oversight Today we focus on the Iterative.

10. © 2003 by Carnegie Mellon University page 10 Discovery Gather and Refine activities occur simultaneously. Gather: Sources take many forms. -Stakeholders, business processes, legacy systems, COTS marketplace Each is independent of the others. There is no optimal order. Refine: Analysis and harmonization of the gathered knowledge Yields the technical definition of the emerging system Finds gaps, negotiates conflicts

11. © 2003 by Carnegie Mellon University page 11 Discovery Activities Data from: stakeholders products … Refine Gather Negotiate conflicts Gap - need more data Continue Discovery Knowledge is sufficient for constructing executable Go on? Harmonized data Mismatch Agreement Gather data z Gather data y Gather data x Analyze

12. © 2003 by Carnegie Mellon University page 12 Assembly Produces a prototype that reflects what’s been discovered Reflects a traditional sequential process Guided by local “ candidate requirements” -Iteration Objectives, Detailed Iteration Plan plus hypotheses and desired behaviors expect to derive from prototype -Vets “candidate requirements” Produces an executable version of full system -Likely to have less than complete functionality -Executable, not paper or specification or small piece Does not preclude prototyping “in the small” throughout

13. © 2003 by Carnegie Mellon University page 13 Evaluation and Assessment Occur at two levels: Evaluation of the prototype in its own context -Measures actual outcome against expected outcome -Considers degree to which prototype satisfies its local “requirements” Assessment of the iteration itself -Addresses issues at project (not iteration) level -Considers whether objectives were good ones -Determines whether iteration results indicate changes in project schedule, budget, etc.

14. © 2003 by Carnegie Mellon University page 14 On-going Iterations of A/APCS Prototype Deployed System Body of Knowledge Body of Knowledge BoK

15. © 2003 by Carnegie Mellon University page 15 Top-Level View of A/APCS Manage Program: based on global program constraints, generates iteration objectives and fielding decisions and forwards the current System View to the next iteration Perform Iteration: based on iteration objectives, the market- place, and current System View, generates a new version of the system, evaluates it, and updates the new System View Assess Iteration: based on iteration objectives and the evaluation of the system, assess the overall iteration

16. © 2003 by Carnegie Mellon University page 16 Detail of Perform Iteration Manage Program Assess Iteration Plan Iteration 1 Construct System 2 Evaluate System 3 Perform Iteration *

17. © 2003 by Carnegie Mellon University page 17 Detail of Construct System Detailed iteration plan “Go” decision for Assembly System view (defined in this iteration) Discovery activities System (for evaluation and potential deployment) Consolidated data to analyze Description of needed knowledge Gather Data 2.1 Refine Data 2.2 Assemble Candidate System 2.3 * System view (defined in previous iteration) Replanning needed COTS products

18. © 2003 by Carnegie Mellon University page 18 Low-level Activities of Refine Data Description of needed knowledge System View (defined in previous iterations) Emerging revisions to System View Additional knowledge needed Gaps Resolutions Resolutions & Gaps Data to analyze Detailed iteration plan Agreements (to all) System view Agreements, Gaps & Conflicts Gaps Assembly “Go” Replanning needed Analyze Data Negotiate Conflicts Form System View Determine Data to be Gathered 2.2.1 2.2.2 2.2.3 2.2.4

19. © 2003 by Carnegie Mellon University page 19 Low-level Activity Descriptions Construct System (Step 2) 2.1 Gather Data: Gather Stakeholder Inputs Gather COTS Product Data Gather Business Process Data Gather External System Data 2.2 Refine Data: Analyze Data Negotiate Conflicts Determine Data to be Gathered Form System View 2.3 Assemble Candidate System: Create Detailed Design Perform Needed Modifications Write Custom Code Integrate Components Test

20. © 2003 by Carnegie Mellon University page 20 Example: Gather COTS Product Data Purpose: Gather product data by evaluating COTS products Role: Evaluators, stakeholders Inputs/Entry Conditions:  Description of the kind of product data that is needed, including the level of detail needed as well as gaps that must be filled. This description forms the requirements for the product evaluation. Outputs/Exit Conditions:  Product data to the requisite level of detail, consolidated and formatted Description: This activity consists of methodically evaluating one or more COTS products. As with all of the “gather” activities, the input is the Need for Knowledge generated by the Refine Data activity. Appropriate evaluation criteria for this product data gathering effort are established; these then provide the basis for the quantification methods that guide the data collection effort.

21. © 2003 by Carnegie Mellon University page 21 Example (cont.) Notes on Planning: … Actions: 1. Define criteria 2. Prioritize criteria 3. Prepare for evaluation 4. Collect data from the sources 5. Record results 6. Consolidate and format data (described in section 7.1.5) Notes on the Actions: 1. A criterion consists of both a capability statement … 2. It is essential to rank criteria by importance. … 3. One necessary preparation is to assemble the needed … 4. Different data collection techniques may be needed. … 5. The raw scores earned by each of the examined products … 6. Product evaluation data is often diverse and …

22. © 2003 by Carnegie Mellon University page 22 Converting This to a Process While developing this framework, we engaged with a customer who needed a process PDQ. We took these principles and, using RUP as a backbone, turned them into a process. Original name: ITSEP New name: Evolutionary Process for Integrating COTS- based systems (EPIC) A negotiation-driven process that helps identify and manage the differences between what users want and what COTS products can deliver Commercial and government EPIC pilots underway

23. © 2003 by Carnegie Mellon University page 23 EPIC Conceptual Framework Time Trade Space Simultaneous Definition and Tradeoffs Marketplace Stakeholder Needs/ Business Processes Architecture/ Design Programmatics/ Risk Trades are negotiation-driven with knowledge of marketplace Requirements formed based on knowledge of market/architecture Continuous awareness of end-user process changes Project business and contractual approaches support trades Trade Space Decisions Converge

24. © 2003 by Carnegie Mellon University page 24 Knowledge Grows Incrementally Risk-based spiral development focuses and integrates diverse information -Prioritized stakeholder needs, end-user processes -Organization and system architecture, design constraints -Identified risks, programmatic constraints -Marketplace offerings, product characteristics, other buyer usage Frequent, evolving executable representations demonstrate current understanding Executable Time

25. © 2003 by Carnegie Mellon University page 25 Continuous Stakeholder Negotiation Quick resolution to discovered mismatches -Business process owners and end users -System engineers and developers -Vendors and suppliers -Project managers -Change agents Time Stakeholder Buy-in Increases Stakeholder needs mature with increased understanding of marketplace implications Business processes change to leverage available products End users committed to system solution

26. © 2003 by Carnegie Mellon University page 26 EPIC: A Negotiation-Driven Approach Drive strategic vision to sustainable solution Increase stakeholder buy-in and reconcile end-user processes with COTS-based system Accumulate knowledge through risk-based spirals Coordinates operational change, system & software engineering, and project management to field initial capability in 6-18 months

27. © 2003 by Carnegie Mellon University page 27 Iterations Redefined for CBS Assess iteration Gather information Simultaneous Definition and Tradeoffs (1 to 8 weeks per outer cycle) Plan iteration Assemble executable representation Executable Refine into harmonized set by identifying and analyzing mismatches and negotiating among stakeholders

28. © 2003 by Carnegie Mellon University page 28 Phases Bounded by Anchor Points LifeCycle Architecture Initial Operational Capability ElaborationConstructionTransition Refine, experiment & select solution Try/select COTS Prototype business process changes Implement selected solution Apply/track COTS Prepare to change business process Field and support solution Track/update COTS Change business process ……… LifeCycle Objectives Inception Gather and define project scope Survey/try COTS Agree to business process changes … 6 to 18 months

29. © 2003 by Carnegie Mellon University page 29 Inception Phase: Goal Achieve concurrence among stakeholders on project’s life-cycle objectives Match segments of marketplace with critical use cases Identify implications of potential business process changes Establish acquisition strategy for appropriate vendor relationships Establish feasibility for the project through the business case that shows there is one or more candidate solutions Survey & Try COTS

30. © 2003 by Carnegie Mellon University page 30 Inception Phase: Activities Gather high-level information (critical use cases) -“Must have” stakeholder needs and required business processes -Market drivers, technologies, products -Architectural constraints and design alternatives -Programmatic constraints and risks -Incentives /inhibitors to business process changes Refine information to form high-level candidate solutions Assemble executable representation(s)

31. © 2003 by Carnegie Mellon University page 31 Elaboration Phase: Goal Define a high-fidelity solution with predictable cost/schedule Select and acquire COTS products Business process implementation supported by all user roles Achieve sufficient stability of the solution across architecture, requirements, and marketplace Mitigate development and rollout risks Try & Select COTS

32. © 2003 by Carnegie Mellon University page 32 Elaboration Phase: Activities Gather detailed knowledge Refine knowledge to form a stable baseline -Gaps/mismatches identified and negotiated -COTS integration mechanisms defined and validated -Business process changes implementation refined -Balanced across all 4 spheres Assemble architectural prototypes -Business process changes prototyped... Amplify each candidate solution Amplify selected solution

33. © 2003 by Carnegie Mellon University page 33 Construction Phase: Goal Achieve a product quality release ready for the user community Prepare functional units for needed business process changes Organization changes Training Manage vendor relationships Balance development stability and potential obsolescence with marketplace volatility Apply & Track COTS

34. © 2003 by Carnegie Mellon University page 34 Construction Phase: Activities Gather marketplace information continually Refine selected solution as necessary Assemble selected solution for fielding -Develop any custom components -Tailor products (non-source code customizations) -Develop production quality COTS integration code/data sets -Implement business process changes for initial fielding

35. © 2003 by Carnegie Mellon University page 35 Transition Phase: Goal Roll out and support selected solution set to the user community Initial fielding (beta testing) Full fielding On-going support until solution release retired Implement business process changes across user community Balance operational stability with marketplace volatility Manage relationships with vendors Track & Update COTS

36. © 2003 by Carnegie Mellon University page 36 Transition Phase: Activities Gather marketplace information continually Refine solution to accommodate the impact of new products, releases, or technology Assemble maintenance releases -Re-tailor products (non-source code customizations) -Re-develop COTS integration code/data sets -Re-integrate and re-test

37. © 2003 by Carnegie Mellon University page 37 Closing This presentation has discussed: a framework for COTS-based systems processes a specific process based in that framework The principles are broadly applicable, and you are invited to try them out.

38. © 2003 by Carnegie Mellon University page 38 For More Information David J. Carney SEI Pittsburgh, PA 412-268-6525 djc@sei.cmu.edu Tricia Oberndorf SEI Pittsburgh, PA 412-268-6138 po@sei.cmu.edu Patrick Place SEI Pittsburgh, PA 412-268-7746 prp@sei.cmu.edu Ceci Albert SEI Washington, DC 703-908-8215 cca@sei.cmu.edu Lisa Brownsword SEI Washington, DC 703-908-8203 llb@sei.cmu.edu

39. © 2003 by Carnegie Mellon University page 39 Acronyms A/APCSAssembly/Acquisition Process for COTS-based Systems CBSCOTS-based system CMconfiguration management COTScommercial off-the-shelf EPIC Evolutionary Process for Integrating COTS- based systems ITSEPIntegrating Technology by a Structured Evolutionary Process SEISoftware Engineering Institute

40. © 2003 by Carnegie Mellon University page 40 BACKUPS

41. © 2003 by Carnegie Mellon University page 41 Guiding Principles 1 The requirements process must celebrate flexibility: requirements definition must be delayed and negotiated, and the true requirements minimized. Use of any COTS product necessitates documenting, and probably re-engineering of existing end-user process. Any COTS–based system will be in a necessary state of continual system re-formation and evolution throughout its useful lifetime.

42. © 2003 by Carnegie Mellon University page 42 Guiding Principles 2 Hands-on product evaluations are mandatory; these must be budgeted, scheduled, and accepted as a fundamental project activity, as central as requirements and design. Prototyping of the integrated collection of COTS products from the earliest possible moment throughout system lifetime is a basic necessity. Satisfactory contractual commitments can only result from the active participation of end users, testers, and other stakeholders, continuously throughout the entire period from project inception through sustainment and maintenance.