July 2002 COSYSMO-IP COnstructive SYStems Engineering Cost Model – Information Processing PSM User’s Group Conference Keystone, Colorado July 24 & 25, 2002 Dr. Barry Boehm Ricardo Valerdi University of Southern California Center for Software Engineering Version 3

July Outline – Day 1 USC Center for Software Engineering Background & Update on COSYSMO-IP Ops Con & EIA632 Delphi Round 1 Results Updated Drivers Lessons Learned/Improvements LMCO & INCOSE Comments Q & A

July Outline – Day 2 Review of yesterday’s modified slides to clarify terminology A few new slides to emphasize points Review of current driver definitions Definition for two new Cost drivers – Technology Maturity – Physical system/information system tradeoff analysis complexity

July Objectives of the Workshop Agree on a Concept of Operation Converge on scope of COSYSMO-IP model Address definitions of model parameters Discuss data collection process

July faculty/research staff, 18 PhD students Corporate Affiliates program (TRW, Aero Galorath, Raytheon, Lockheed, Motorola, et al) 17th International Forum on COCOMO and Software Cost Modeling October 22-25, 2002, Los Angeles, CA –Theme: Software Cost Estimation and Risk Management Annual research review in March 2003

July COSYSMO-IP: What is it? The purpose of the COSYSMO-IP project is to develop an initial increment of a parametric model to estimate the cost of system engineering activities during system development. The focus of the initial increment is on the cost of systems engineering for information processing systems or subsystems.

July What Does COSYSMO-IP Cover? Includes: –System engineering in the inception, elaboration, and construction phases, including test planning –Requirements development and specification activities –Physical system/information system tradeoff analysis –Operations analysis and design activities –System architecture tasks Including allocations to hardware/software and consideration of COTS, NDI and legacy impacts –Algorithm development and validation tasks Defers: –Physical system/information system operation test & evaluation, deployment –Special-purpose hardware design and development –Structure, power and/or specialty engineering –Manufacturing and/or production analysis

July Candidate COSYSMO Evolution Path Inception Elaboration Construction Transition Oper Test & Eval 1. COSYSMO-IP 2. COSYSMO-C4ISR 3. COSYSMO-Machine 4. COSYSMO-SoS IP (Sub)system C4ISR System Physical Machine System System of Systems (SoS)

July Current COSYSMO-IP Operational Concept COSYSMO-IP Effort Duration Calibration WBS guided By EIA 632 Size Drivers Effort Multipliers # Requirements # Interfaces # Scenarios # Algorithms Volatility Factor … - Application factors - Team factors - Schedule driver

July EIA632/COSYSMO-IP Mapping COSYSMO-IP Category EIA632 Requirement Supplier Performance 3 Technical Management 4-12 Requirements Definition Solution Definition Systems Analysis Requirements Validation Design Solution Verification 30 End Products Validation - COTS33a EIA632 Reqs. not included in COSYSMO-IP are: 1,2,13,20,21,31,32,33b

July Activity Elements Covered by EIA632, COCOMOII, and COSYSMO-IP = COCOMOII = COSYSMO-IP When doing COSYSMO-IP and COCOMOII, Subtract grey areas prevent double counting.

July Past, Present, and Future Initial set if parameters compiled by Affiliates Performed First Delphi Round Working Group meeting at ARR PSM Workshop Meeting at CCII Conference

July Future Parameter Refinement Opportunities Driver definitions Data collection (Delphi) Model calibration First iteration of model

July Delphi Survey Survey was conducted to: –Determine the distribution of effort across effort categories –Determine the range for size driver and effort multiplier ratings –Identify the cost drivers to which effort is most sensitive to –Reach consensus from a sample of systems engineering experts Distributed Delphi surveys to Affiliates and received 28 responses 3 Sections: –Scope, Size, Cost Also helped us refine the scope of the model elements

July System Engineering Effort Distribution Category (EIA Requirement) Supplier Performance (3) Technical Management (4-12) Requirements Definition (14-16) Solution Definition (17-19) Systems Analysis (22-24) Requirements Validation (25-29) Design Solution Verification (30) End Products Validation (33a) Delphi 5.2% 13.1% 16.6% 18.1% 19.2% 11.3% 10.5% 6.6% Suggested 5% 15% 20% 15% 5% Std. Dev. Delphi Round 1 Results

July Delphi Round 1 Highlights (cont.) Range of sensitivity for Size Drivers # Algorithms # Requirements # Interfaces # TPM’s # Scenarios # Modes # Platforms 5.57 Relative Effort

July Two Most Sensitive Size Drivers Suggested Rel. Effort Delphi Respondents EMR Rel. Effort Standard Deviation # Interfaces # Algorithms

July Delphi Round 1 Highlights (cont.) Range of sensitivity for Cost Drivers (Application Factors) EMR Requirements und. Architecture und. Level of service reqs. Legacy transition COTS Platform difficulty Bus. process reeng

July Delphi Round 1 Highlights (cont.) Range of sensitivity for Cost Drivers (Team Factors) Tool support Stakeholder comm. Stakeholder cohesion Personnel capability Personal experience Process maturity Multisite coord. Formality of deliv EMR 4 2

July Suggested EMR Delphi Respondents EMR Mean Standard Deviation Arch. Under Reqs. Under Pers. Cap Serv. Req Four Most Sensitive Cost Drivers

July Size Drivers 1. Number of System Requirements 2. Number of Major Interfaces 3. Number of Operational Scenarios 4. Number of Unique Algorithms Number of Technical Performance Measures Number of Modes of Operation Number of Different Platforms COST Driver COST Driver

July Size Driver Definitions (1 of 4) Number of System Requirements The number of requirements taken from the system specification. A requirement is a statement of capability or attribute containing a normative verb such as shall or will. It may be functional or system service-oriented in nature depending on the methodology used for specification. System requirements can typically be quantified by counting the number of applicable shall’s or will’s in the system or marketing specification. Note: Use this driver as the basis of comparison for the rest of the drivers.

July Size Driver Definitions (2 of 4) Number of Major Interfaces The number of shared major physical and logical boundaries between system components or functions (internal interfaces) and those external to the system (external interfaces). These interfaces typically can be quantified by counting the number of interfaces identified in either the system’s context diagram and/or by counting the significant interfaces in applicable Interface Control Documents.

July Size Driver Definitions (3 of 4) Number of Operational Scenarios* The number of operational scenarios** that a system is specified to satisfy. Such threads typically result in end-to-end test scenarios that are developed to validate the system satisfies its requirements. The number of scenarios can typically be quantified by counting the number of end-to-end tests used to validate the system functionality and performance. They can also be calculated by counting the number of high-level use cases developed as part of the operational architecture. Number of Modes of Operation (to be merged with Op Scen) The number of defined modes of operation for a system. For example, in a radar system, the operational modes could be air-to-air, air-to- ground, weather, targeting, etc. The number of modes is quantified by counting the number of operational modes specified in the Operational Requirements Document. *counting rules need to be refined **Op Scen can be derived from system modes

July Size Driver Definitions (4 of 4) Number of Unique Algorithms The number of newly defined or significantly altered functions that require unique mathematical algorithms to be derived in order to achieve the system performance requirements. Note: Examples could include a complex aircraft tracking algorithm like a Kalman Filter being derived using existing experience as the basis for the all aspect search function. Another Example could be a brand new discrimination algorithm being derived to identify friend or foe function in space-based applications. The number can be quantified by counting the number of unique algorithms needed to support each of the mathematical functions specified in the system specification or mode description document (for sensor-based systems).

July Cost Drivers 1. Requirements understanding 2. Architecture complexity 3. Level of service requirements 4. Migration complexity 5. COTS assessment complexity 6. Platform difficulty 7. Required business process reengineering 8. Technology Maturity 9. Physical system/information subsystem tradeoff analysis complexity Application Factors (5)

July Cost Driver Definitions (1,2 of 5) Requirements understanding The level of understanding of the system requirements by all stakeholders including the systems, software, hardware, customers, team members, users, etc… Architecture complexity The relative difficulty of determining and managing the system architecture in terms of IP platforms, standards, components (COTS/GOTS/NDI/new), connectors (protocols), and constraints. This includes systems analysis, tradeoff analysis, modeling, simulation, case studies, etc…

July Cost Driver Definitions (3,4,5 of 5) Migration complexity (formerly Legacy transition complexity) The complexity of migrating the system from previous system components, databases, workflows, etc, due to new technology introductions, planned upgrades, increased performance, business process reengineering etc… Level of service requirements The difficulty and criticality of satisfying the Key Performance Parameters (KPP). For example: security, safety, response time, the “illities”, etc… Technology Maturity The relative readiness for operational use of the key technologies.

July Cost Drivers (cont.) 1. Number and diversity of stakeholder communities 2. Stakeholder team cohesion 3. Personnel capability 4. Personal experience/continuity 5. Process maturity 6. Multisite coordination 7. Formality of deliverables 8. Tool support Team Factors (7)

July Cost Driver Definitions (1,2,3 of 7) Stakeholder team cohesion Leadership, frequency of meetings, shared vision, approval cycles, group dynamics (self-directed teams, project engineers/managers), IPT framework, and effective team dynamics. Personnel capability Systems Engineering’s ability to perform in their duties and the quality of human capital. Personnel experience/continuity The applicability and consistency of the staff over the life of the project with respect to the customer, user, technology, domain, etc…

July Cost Driver Definitions (4,5,6,7 of 7) Process maturity Maturity per EIA/IS 731, SE CMM or CMMI. Multisite coordination Location of stakeholders, team members, resources (travel). Formality of deliverables The breadth and depth of documentation required to be formally delivered. Tool support Use of tools in the System Engineering environment.

July Lessons Learned/Improvements Lesson 1 – Need to better define the scope and future of COSYSMO-IP via Con Ops Lesson 2 – Drivers can be interpreted in different Ways depending on the type of program Lesson 3 – COSYSMO is too software-oriented Lesson 4 – Delphi needs to take less time to fill out Lesson 5 – Need to develop examples, rating scales

July The current COSYSMO focus is too software oriented. This is a good point. We propose to change the scope from "software-intensive systems or subsystems" to "information processing (IP) systems or subsystems." These include not just the software but also the associated IP hardware processors; memory; networking; display or other human-computer interaction devices. System engineering of these IP systems or subsystems includes considerations of IP hardware device acquisition lead times, producibility, and logistics. Considerations on non-IP hardware acquisition, producibility, and logistics are considered as IP systems engineering cost and schedule drivers for the IOC version of COSYSMO. Perhaps we should call it COSYSMO-IP. LMCO Comments

July The COSYSMO project should begin by working out the general framework and WBS for the full life cycle of a general system. We agree that such a general framework and WBS will eventually be needed. However, we feel that progress toward it can be most expeditiously advanced by working on definitions of and data for a key element of the general problem first. If another group would like to concurrently work out the counterpart definitions and data considerations for the general system engineering framework, WBS, and estimation model, we will be happy to collaborate with them. LMCO Comments (cont.)

July Points of Contact Dr. Barry Boehm (213) Ricardo Valerdi (213) Donald Reifer (310) Websites

July Backup slides

July COCOMOII Suite COCOMOII COQUALMO COPSEMO CORADMO COSYSMO-IP COPROMO COCOTS For more information visit