1. July 2002 COSYSMO-IP COnstructive SYStems Engineering Cost Model – Information Processing PSM User’s Group Conference Keystone, Colorado July 24 & 25, 2002 Dr. Barry Boehm Ricardo Valerdi University of Southern California Center for Software Engineering Version 3

2. July 20022 Outline – Day 1 USC Center for Software Engineering Background & Update on COSYSMO-IP Ops Con & EIA632 Delphi Round 1 Results Updated Drivers Lessons Learned/Improvements LMCO & INCOSE Comments Q & A

3. July 20023 Outline – Day 2 Review of yesterday’s modified slides to clarify terminology A few new slides to emphasize points Review of current driver definitions Definition for two new Cost drivers – Technology Maturity – Physical system/information system tradeoff analysis complexity

4. July 20024 Objectives of the Workshop Agree on a Concept of Operation Converge on scope of COSYSMO-IP model Address definitions of model parameters Discuss data collection process

5. July 20025 8 faculty/research staff, 18 PhD students Corporate Affiliates program (TRW, Aero Galorath, Raytheon, Lockheed, Motorola, et al) 17th International Forum on COCOMO and Software Cost Modeling October 22-25, 2002, Los Angeles, CA –Theme: Software Cost Estimation and Risk Management Annual research review in March 2003

6. July 20026 COSYSMO-IP: What is it? The purpose of the COSYSMO-IP project is to develop an initial increment of a parametric model to estimate the cost of system engineering activities during system development. The focus of the initial increment is on the cost of systems engineering for information processing systems or subsystems.

7. July 20027 What Does COSYSMO-IP Cover? Includes: –System engineering in the inception, elaboration, and construction phases, including test planning –Requirements development and specification activities –Physical system/information system tradeoff analysis –Operations analysis and design activities –System architecture tasks Including allocations to hardware/software and consideration of COTS, NDI and legacy impacts –Algorithm development and validation tasks Defers: –Physical system/information system operation test & evaluation, deployment –Special-purpose hardware design and development –Structure, power and/or specialty engineering –Manufacturing and/or production analysis

8. July 20028 Candidate COSYSMO Evolution Path Inception Elaboration Construction Transition Oper Test & Eval 1. COSYSMO-IP 2. COSYSMO-C4ISR 3. COSYSMO-Machine 4. COSYSMO-SoS IP (Sub)system C4ISR System Physical Machine System System of Systems (SoS)

9. July 20029 Current COSYSMO-IP Operational Concept COSYSMO-IP Effort Duration Calibration WBS guided By EIA 632 Size Drivers Effort Multipliers # Requirements # Interfaces # Scenarios # Algorithms Volatility Factor … - Application factors - Team factors - Schedule driver

10. July 200210 EIA632/COSYSMO-IP Mapping COSYSMO-IP Category EIA632 Requirement Supplier Performance 3 Technical Management 4-12 Requirements Definition 14-16 Solution Definition 17-19 Systems Analysis 22-24 Requirements Validation 25-29 Design Solution Verification 30 End Products Validation - COTS33a EIA632 Reqs. not included in COSYSMO-IP are: 1,2,13,20,21,31,32,33b

11. July 200211 Activity Elements Covered by EIA632, COCOMOII, and COSYSMO-IP = COCOMOII = COSYSMO-IP When doing COSYSMO-IP and COCOMOII, Subtract grey areas prevent double counting.

12. July 200212 Past, Present, and Future Initial set if parameters compiled by Affiliates 200120022003 Performed First Delphi Round Working Group meeting at ARR PSM Workshop Meeting at CCII Conference

13. July 200213 Future Parameter Refinement Opportunities 200420052003 Driver definitions Data collection (Delphi) Model calibration First iteration of model

14. July 200214 Delphi Survey Survey was conducted to: –Determine the distribution of effort across effort categories –Determine the range for size driver and effort multiplier ratings –Identify the cost drivers to which effort is most sensitive to –Reach consensus from a sample of systems engineering experts Distributed Delphi surveys to Affiliates and received 28 responses 3 Sections: –Scope, Size, Cost Also helped us refine the scope of the model elements

15. July 200215 System Engineering Effort Distribution Category (EIA Requirement) Supplier Performance (3) Technical Management (4-12) Requirements Definition (14-16) Solution Definition (17-19) Systems Analysis (22-24) Requirements Validation (25-29) Design Solution Verification (30) End Products Validation (33a) Delphi 5.2% 13.1% 16.6% 18.1% 19.2% 11.3% 10.5% 6.6% Suggested 5% 15% 20% 15% 5% 3.05 4.25 4.54 4.28 5.97 4.58 6.07 3.58 Std. Dev. Delphi Round 1 Results

16. July 200216 Delphi Round 1 Highlights (cont.) Range of sensitivity for Size Drivers # Algorithms # Requirements # Interfaces # TPM’s # Scenarios # Modes # Platforms 5.57 Relative Effort 1 2.23 2.54 2.212.10 6.48 6 4 2

17. July 200217 Two Most Sensitive Size Drivers Suggested Rel. Effort Delphi Respondents EMR Rel. Effort Standard Deviation # Interfaces 45.571.80 # Algorithms 66.482.09

18. July 200218 Delphi Round 1 Highlights (cont.) Range of sensitivity for Cost Drivers (Application Factors) EMR 1.93 2.81 2.13 2.43 2.24 4 2 Requirements und. Architecture und. Level of service reqs. Legacy transition COTS Platform difficulty Bus. process reeng. 1.741.13

19. July 200219 Delphi Round 1 Highlights (cont.) Range of sensitivity for Cost Drivers (Team Factors) 1.28 2.46 1.91 2.16 1.94 1.25 Tool support Stakeholder comm. Stakeholder cohesion Personnel capability Personal experience Process maturity Multisite coord. Formality of deliv. 1.841.78 EMR 4 2

20. July 200220 Suggested EMR Delphi Respondents EMR Mean Standard Deviation Arch. Under. 1.662.240.83 Reqs. Under. 1.732.430.70 Pers. Cap. 2.152.460.66 Serv. Req. 2.52.810.67 Four Most Sensitive Cost Drivers

21. July 200221 4 Size Drivers 1. Number of System Requirements 2. Number of Major Interfaces 3. Number of Operational Scenarios 4. Number of Unique Algorithms Number of Technical Performance Measures Number of Modes of Operation Number of Different Platforms COST Driver COST Driver

22. July 200222 Size Driver Definitions (1 of 4) Number of System Requirements The number of requirements taken from the system specification. A requirement is a statement of capability or attribute containing a normative verb such as shall or will. It may be functional or system service-oriented in nature depending on the methodology used for specification. System requirements can typically be quantified by counting the number of applicable shall’s or will’s in the system or marketing specification. Note: Use this driver as the basis of comparison for the rest of the drivers.

23. July 200223 Size Driver Definitions (2 of 4) Number of Major Interfaces The number of shared major physical and logical boundaries between system components or functions (internal interfaces) and those external to the system (external interfaces). These interfaces typically can be quantified by counting the number of interfaces identified in either the system’s context diagram and/or by counting the significant interfaces in applicable Interface Control Documents.

24. July 200224 Size Driver Definitions (3 of 4) Number of Operational Scenarios* The number of operational scenarios** that a system is specified to satisfy. Such threads typically result in end-to-end test scenarios that are developed to validate the system satisfies its requirements. The number of scenarios can typically be quantified by counting the number of end-to-end tests used to validate the system functionality and performance. They can also be calculated by counting the number of high-level use cases developed as part of the operational architecture. Number of Modes of Operation (to be merged with Op Scen) The number of defined modes of operation for a system. For example, in a radar system, the operational modes could be air-to-air, air-to- ground, weather, targeting, etc. The number of modes is quantified by counting the number of operational modes specified in the Operational Requirements Document. *counting rules need to be refined **Op Scen can be derived from system modes

25. July 200225 Size Driver Definitions (4 of 4) Number of Unique Algorithms The number of newly defined or significantly altered functions that require unique mathematical algorithms to be derived in order to achieve the system performance requirements. Note: Examples could include a complex aircraft tracking algorithm like a Kalman Filter being derived using existing experience as the basis for the all aspect search function. Another Example could be a brand new discrimination algorithm being derived to identify friend or foe function in space-based applications. The number can be quantified by counting the number of unique algorithms needed to support each of the mathematical functions specified in the system specification or mode description document (for sensor-based systems).

26. July 200226 12 Cost Drivers 1. Requirements understanding 2. Architecture complexity 3. Level of service requirements 4. Migration complexity 5. COTS assessment complexity 6. Platform difficulty 7. Required business process reengineering 8. Technology Maturity 9. Physical system/information subsystem tradeoff analysis complexity Application Factors (5)

27. July 200227 Cost Driver Definitions (1,2 of 5) Requirements understanding The level of understanding of the system requirements by all stakeholders including the systems, software, hardware, customers, team members, users, etc… Architecture complexity The relative difficulty of determining and managing the system architecture in terms of IP platforms, standards, components (COTS/GOTS/NDI/new), connectors (protocols), and constraints. This includes systems analysis, tradeoff analysis, modeling, simulation, case studies, etc…

28. July 200228 Cost Driver Definitions (3,4,5 of 5) Migration complexity (formerly Legacy transition complexity) The complexity of migrating the system from previous system components, databases, workflows, etc, due to new technology introductions, planned upgrades, increased performance, business process reengineering etc… Level of service requirements The difficulty and criticality of satisfying the Key Performance Parameters (KPP). For example: security, safety, response time, the “illities”, etc… Technology Maturity The relative readiness for operational use of the key technologies.

29. July 200229 12 Cost Drivers (cont.) 1. Number and diversity of stakeholder communities 2. Stakeholder team cohesion 3. Personnel capability 4. Personal experience/continuity 5. Process maturity 6. Multisite coordination 7. Formality of deliverables 8. Tool support Team Factors (7)

30. July 200230 Cost Driver Definitions (1,2,3 of 7) Stakeholder team cohesion Leadership, frequency of meetings, shared vision, approval cycles, group dynamics (self-directed teams, project engineers/managers), IPT framework, and effective team dynamics. Personnel capability Systems Engineering’s ability to perform in their duties and the quality of human capital. Personnel experience/continuity The applicability and consistency of the staff over the life of the project with respect to the customer, user, technology, domain, etc…

31. July 200231 Cost Driver Definitions (4,5,6,7 of 7) Process maturity Maturity per EIA/IS 731, SE CMM or CMMI. Multisite coordination Location of stakeholders, team members, resources (travel). Formality of deliverables The breadth and depth of documentation required to be formally delivered. Tool support Use of tools in the System Engineering environment.

32. July 200232 Lessons Learned/Improvements Lesson 1 – Need to better define the scope and future of COSYSMO-IP via Con Ops Lesson 2 – Drivers can be interpreted in different Ways depending on the type of program Lesson 3 – COSYSMO is too software-oriented Lesson 4 – Delphi needs to take less time to fill out Lesson 5 – Need to develop examples, rating scales

33. July 200233 The current COSYSMO focus is too software oriented. This is a good point. We propose to change the scope from "software-intensive systems or subsystems" to "information processing (IP) systems or subsystems." These include not just the software but also the associated IP hardware processors; memory; networking; display or other human-computer interaction devices. System engineering of these IP systems or subsystems includes considerations of IP hardware device acquisition lead times, producibility, and logistics. Considerations on non-IP hardware acquisition, producibility, and logistics are considered as IP systems engineering cost and schedule drivers for the IOC version of COSYSMO. Perhaps we should call it COSYSMO-IP. LMCO Comments

34. July 200234 The COSYSMO project should begin by working out the general framework and WBS for the full life cycle of a general system. We agree that such a general framework and WBS will eventually be needed. However, we feel that progress toward it can be most expeditiously advanced by working on definitions of and data for a key element of the general problem first. If another group would like to concurrently work out the counterpart definitions and data considerations for the general system engineering framework, WBS, and estimation model, we will be happy to collaborate with them. LMCO Comments (cont.)

35. July 200235 Points of Contact Dr. Barry Boehm [boehm@sunset.usc.edu] (213) 740-8163 Ricardo Valerdi [rvalerdi@sunset.usc.edu] (213) 440-4378 Donald Reifer [dreifer@earthlink.net] (310) 530-4493 Websites http://valerdi.com/cosysmo http://sunset.usc.edu

36. July 200236 Backup slides

37. July 200237 COCOMOII Suite COCOMOII COQUALMO COPSEMO CORADMO COSYSMO-IP COPROMO COCOTS For more information visit http://sunset.usc.edu