1. INCOSE CAB Briefing November 2002 COSYSMO COnstructive SYStems Engineering Cost MOdel November 1, 2002 Dr. Barry Boehm Ricardo Valerdi University of Southern California Center for Software Engineering (CSE) INCOSE CAB Briefing

2. November 2002 Outline Background on CSE, COSYSMO, and COCOMO II COSYSMO Overview –Operational concept and scope –Prototype demo Model Progress to Date –Front end sizing and drivers –Full life cycle sizing and drivers Calendar of activities/milestones Action items How can the CAB help?

3. INCOSE CAB Briefing November 2002 USC Center for Software Engineering (CSE) Researches, teaches, and practices CMMI-based Software engineering –Systems and software engineering fully integrated Focuses on better models to guide integrated systems and software engineering –Success models: stakeholder win-win, business cases –Product models: requirements, architectures, COTS –Process models: spiral extensions, value-based RUP extensions –Property models: cost, schedule, quality Applies and extends research on major programs (DARPA/Army, FCS, FAA ERAM, NASA Missions)

4. INCOSE CAB Briefing November 2002 Commercial Industry (15) –Daimler Chrysler, Freshwater Partners, Galorath, Group Systems.Com, Hughes, IBM, Cost Xpert Group, Microsoft, Motorola, Price Systems, Rational, Reuters Consulting, Sun, Telcordia, Xerox Aerospace Industry (6) –Boeing, Lockheed Martin, Northrop Grumman, Raytheon, SAIC, TRW Government (8) –DARPA, DISA, FAA, NASA-Ames, NSF, OSD/ARA/SIS, US Army Research Labs, US Army TACOM FFRDC’s and Consortia (4) –Aerospace, JPL, SEI, SPC International (1) –Chung-Ang U. (Korea) USC-CSE Affiliates (34)

5. INCOSE CAB Briefing November 2002 USC-CSE Cost, Schedule, and Quality Models Build on experience with COCOMO 1981, COCOMO II –Most widely used software cost models worldwide –Developed with Affiliate funding, expertise, data support Collaborative efforts between Computer Science (CS) and Industrial Systems Engineering (ISE) Depts. –3 CS PhD’s, 2 ISE PhD’s to date –Valerdi an ISE PhD student –Boehm joint appointment in CS, ISE COCOMO Suite of models –Cost, schedule: COCOMO II, CORADMO, COCOTS –Quality: COQUALMO –Systems Engineering: COSYSMO Uses mature 7-step model development methodology

6. INCOSE CAB Briefing November 2002 7-step Modeling Methodology Analyze Existing literature 1 2 3 4 5 6 7 Perform Behavioral Analysis Identify Relative Significance Perform Expert- Judgement, Delphi Assessment Gather Project Data Determine Bayesian A-Posteriori Update Gather more data; refine model A-PRIORI MODEL + SAMPLING DATA = A-POSTERIORI MODEL

7. INCOSE CAB Briefing November 2002 COSYSMO: Overview Parametric model to estimate system engineering costs Covers full system engineering lifecycle Focused on use for Investment Analysis, Concept Definition phases estimation and tradeoff analyses –Input parameters can be determined in early phases

8. INCOSE CAB Briefing November 2002 Key Members of the COSYSMO Working Group Karen Owens, Marilee Wheaton Evin Stump Garry Roedler, Gary Hafen Gary Thomas, John Rieff Tony Jordano, Don Greenlee Chris Miller Marilee Wheaton Cheryl Jones Barry Boehm, Elliot Axelband, Don Reifer, Ricardo Valerdi Aerospace Corp. Galorath LMCO Raytheon SAIC SPC TRW US Army/PSSM USC

9. INCOSE CAB Briefing November 2002 COSYSMO Size Drivers Effort Multipliers Effort Duration Calibration # Requirements # Interfaces # Scenarios # Algorithms + Volatility Factor - Application factors -5 factors - Team factors -7 factors - Schedule driver WBS guided by EIA/ANSI 632 & ISO/IEC 15288 COSYSMO Operational Concept

10. INCOSE CAB Briefing November 2002 Previous COSYSMO Evolution Path Inception Elaboration Construction Transition Oper Test & Eval 1. COSYSMO-IP 2. COSYSMO-C4ISR 3. COSYSMO-Machine 4. COSYSMO-SoS IP (Sub)system C4ISR System Physical Machine System System of Systems (SoS)

11. INCOSE CAB Briefing November 2002 Revised View of COSYSMO Evolution Path (Results from last week’s meeting) Oper Test & Eval 1. COSYSMO-IP 2. COSYSMO-C4ISR 3. COSYSMO-Machine 4. COSYSMO-SoS Global Command and Control System Satellite Ground Station Joint Strike Fighter Future Combat Systems Initiate data collection for all and let the amount of data received determine what is included. Include ISO/IEC 15288 Stages DevelopConceptualize Transition to Operation Operate, Maintain, or Enhance Replace or Dismantle

12. INCOSE CAB Briefing November 2002 EIA/ANSI 632 EIA/ANSI 632 - Provide an integrated set of fundamental processes to aid a developer in the engineering or re-engineering of a system Breadth and Depth of Key SE Standards System life ISO/IEC 15288 Level of detail ConceptualizeDevelop Transition to Operation Operate, Maintain, or Enhance Replace or Dismantle Process description High level practices Detailed practices ISO/IEC 15288 - Establish a common framework for describing the life cycle of systems Purpose of the Standards: IEEE 1220 IEEE 1220 - Provide a standard for managing systems engineering Input to 632/1220 Source : Draft Report ISO Study Group May 2, 2000

13. INCOSE CAB Briefing November 2002 ISO/IEC 15288 Key Terms System –a combination of interacting elements organized to achieve one or more stated purposes System-of-Interest –the system whose life cycle is under consideration in the context of this International Standard System Element –a member of a set of elements that constitutes a system –NOTE: A system element is a discrete part of a system that can be implemented to fulfill specified requirements Enabling System –a system that complements a system-of-interest during its life cycle stages but does not necessarily contribute directly to its function during operation –NOTE: For example, when a system-of-interest enters the production stage, an enabling production system is required Source: ISO/IEC 15288.

14. INCOSE CAB Briefing November 2002 ISO/IEC 15288 System of Interest Structure Make or buy Source: ISO/IEC 15288.

15. INCOSE CAB Briefing November 2002 Raytheon myCOSYSMO* Demo *Developed by Gary Thomas at Raytheon Garland

16. INCOSE CAB Briefing November 2002 Outline Background on CSE, COSYSMO, and COCOMO II COSYSMO Overview –Operational concept and scope –Prototype demo Model Progress to Date –Front end sizing and drivers –Full life cycle sizing and drivers Calendar of activities/milestones Action items How can the CAB help?

17. INCOSE CAB Briefing November 2002 4 Size Drivers 1. Number of System Requirements 2. Number of Major Interfaces 3. Number of Operational Scenarios 4. Number of Unique Algorithms Each weighted by complexity, volatility, and degree of reuse

18. INCOSE CAB Briefing November 2002 Size Driver Definitions (1 of 4) Number of System Requirements The number of requirements taken from the system specification. A requirement is a statement of capability or attribute containing a normative verb such as shall or will. It may be functional or system service-oriented in nature depending on the methodology used for specification. System requirements can typically be quantified by counting the number of applicable shall’s or will’s in the system or marketing specification. Note 1: Use this driver as the basis of comparison for the rest of the drivers. Note 2: Use equivalent size weighted by complexity, volatility, and degree of reuse.

19. INCOSE CAB Briefing November 2002 Size Driver Definitions (2 of 4) Number of Major Interfaces The number of shared major physical and logical boundaries between system components or functions (internal interfaces) and those external to the system (external interfaces). These interfaces typically can be quantified by counting the number of interfaces identified in either the system’s context diagram and/or by counting the significant interfaces in applicable Interface Control Documents.

20. INCOSE CAB Briefing November 2002 Size Driver Definitions (3 of 4) Number of Operational Scenarios* The number of operational scenarios** that a system is specified to satisfy. Such threads typically result in end-to-end test scenarios that are developed to validate the system satisfies its requirements. The number of scenarios can typically be quantified by counting the number of end-to-end tests used to validate the system functionality and performance. They can also be calculated by counting the number of high-level use cases developed as part of the operational architecture. Number of Modes of Operation (to be merged with Op Scen) The number of defined modes of operation for a system. For example, in a radar system, the operational modes could be air-to-air, air-to- ground, weather, targeting, etc. The number of modes is quantified by counting the number of operational modes specified in the Operational Requirements Document. *counting rules need to be refined **Op Scen can be derived from system modes

21. INCOSE CAB Briefing November 2002 Size Driver Definitions (4 of 4) Number of Unique Algorithms The number of newly defined or significantly altered functions that require unique mathematical algorithms to be derived in order to achieve the system performance requirements. Note: Examples could include a complex aircraft tracking algorithm like a Kalman Filter being derived using existing experience as the basis for the all aspect search function. Another Example could be a brand new discrimination algorithm being derived to identify friend or foe function in space-based applications. The number can be quantified by counting the number of unique algorithms needed to support each of the mathematical functions specified in the system specification or mode description document (for sensor-based systems).

22. INCOSE CAB Briefing November 2002 12 Cost Drivers 1. Requirements understanding 2. Architecture complexity 3. Level of service requirements 4. Migration complexity 5. Technology Maturity Application Factors (5)

23. INCOSE CAB Briefing November 2002 Cost Driver Definitions (1,2 of 5) Requirements understanding The level of understanding of the system requirements by all stakeholders including the systems, software, hardware, customers, team members, users, etc… Architecture complexity The relative difficulty of determining and managing the system architecture in terms of IP platforms, standards, components (COTS/GOTS/NDI/new), connectors (protocols), and constraints. This includes systems analysis, tradeoff analysis, modeling, simulation, case studies, etc…

24. INCOSE CAB Briefing November 2002 Cost Driver Definitions (3,4,5 of 5) Migration complexity (formerly Legacy transition complexity) The complexity of migrating the system from previous system components, databases, workflows, etc, due to new technology introductions, planned upgrades, increased performance, business process reengineering etc… Level of service requirements The difficulty and criticality of satisfying the Key Performance Parameters (KPP). For example: security, safety, response time, the “illities”, etc… Technology Maturity The relative readiness for operational use of the key technologies.

25. INCOSE CAB Briefing November 2002 12 Cost Drivers (cont.) 1. Stakeholder team cohesion 2. Personnel capability 3. Personal experience/continuity 4. Process maturity 5. Multisite coordination 6. Formality of deliverables 7. Tool support Team Factors (7)

26. INCOSE CAB Briefing November 2002 Cost Driver Definitions (1,2,3 of 7) Stakeholder team cohesion Leadership, frequency of meetings, shared vision, approval cycles, group dynamics (self-directed teams, project engineers/managers), IPT framework, and effective team dynamics. Personnel capability Systems Engineering’s ability to perform in their duties and the quality of human capital. Personnel experience/continuity The applicability and consistency of the staff over the life of the project with respect to the customer, user, technology, domain, etc…

27. INCOSE CAB Briefing November 2002 Cost Driver Definitions (4,5,6,7 of 7) Process maturity Maturity per EIA/IS 731, SE CMM or CMMI. Multisite coordination Location of stakeholders, team members, resources (travel). Formality of deliverables The breadth and depth of documentation required to be formally delivered. Tool support Use of tools in the System Engineering environment.

28. INCOSE CAB Briefing November 2002 Mapping of Old to New COSYSMO-IP Drivers Number of System Requirements Number of Major Interfaces Number of Technical Performance Measures Number of Operational Scenarios Number of Modes of Operation Number of Different Platforms Number of Unique Algorithms Old (7) New (4) Number of System Requirements Number of Major Interfaces Number of Operational Scenarios Number of Unique Algorithms Size Factors Level of Service Requirements Architecture complexity

29. INCOSE CAB Briefing November 2002 Delphi Round 1 Highlights Range of sensitivity for Size Drivers # Algorithms # Requirements # Interfaces # TPM’s # Scenarios # Modes # Platforms 5.57 Relative Effort 1 2.23 2.54 2.212.10 6.48 6 4 2

30. INCOSE CAB Briefing November 2002 Requirements understanding Architecture complexity Level of service requirements Legacy Transition complexity COTS assessment complexity Platform difficulty Required business process reengineering Technology Maturity Physical system/information subsystem tradeoff analysis complexity Requirements understanding Architecture complexity Level of service requirements Migration complexity Technology Maturity Application Cost Factors Mapping of Old to New COSYSMO-IP Drivers # of TPMs # of Platforms Old (9) New (5)

31. INCOSE CAB Briefing November 2002 Delphi Round 1 Highlights (cont.) Range of sensitivity for Cost Drivers (Application Factors) EMR 1.93 2.81 2.13 2.43 2.24 4 2 Requirements und. Architecture und. Level of service reqs. Legacy transition COTS Platform difficulty Bus. process reeng. 1.741.13

32. INCOSE CAB Briefing November 2002 Number and diversity of stakeholder communities Stakeholder team cohesion Personnel capability Personnel experience/continuity Process maturity Multisite coordination Formality of deliverables Tool support Old (8) New (7) Stakeholder team cohesion Personnel capability Personal experience/continuity Process maturity Multisite coordination Formality of deliverables Tool support Reqs Und Mapping of Old to New COSYSMO-IP Drivers Team Cost Factors

33. INCOSE CAB Briefing November 2002 Delphi Round 1 Highlights (cont.) Range of sensitivity for Cost Drivers (Team Factors) 1.28 2.46 1.91 2.16 1.94 1.25 Tool support Stakeholder comm. Stakeholder cohesion Personnel capability Personal experience Process maturity Multisite coord. Formality of deliv. 1.841.78 EMR 4 2

34. INCOSE CAB Briefing November 2002 Outline Background on CSE, COSYSMO, and COCOMO II COSYSMO Overview –Operational concept and scope –Prototype demo Model Progress to Date –Front end sizing and drivers –Full life cycle sizing and drivers Calendar of activities/milestones Action items How can the CAB help?

35. INCOSE CAB Briefing November 2002 INCOSE Issues and Answers Issue Application scope Life Cycle scope Too many size drivers Conflicting cost drivers Overlap between COSYSMO and CII Answer Framework covers all systems; initial model scope TBD by data Full 15288 lifecycle Reduced from 7 to 4 Reduced from 17 to 12 Candidate starting point identified

36. INCOSE CAB Briefing November 2002 COSYSMO/COCOMO II Mapping Previous candidate starting point = COCOMOII = COSYSMO-IP When doing COSYSMO-IP and COCOMOII, Subtract grey areas prevent double counting. TBD

37. INCOSE CAB Briefing November 2002 Size Drivers vs. EIA/ANSI 632 & ISO/IEC 15288 Stages Late in the Life Cycle Legend Bold = existing driver Italics = proposed addition

38. INCOSE CAB Briefing November 2002 Cost Drivers vs. EIA/ANSI 632 & ISO/IEC 15288 Stages Late in the Life Cycle Legend Bold = existing driver Italics = proposed addition

39. INCOSE CAB Briefing November 2002 Cost Drivers vs. EIA/ANSI 632 & ISO/IEC 15288 Stages Late in the Life Cycle Legend Bold = existing driver Italics = proposed addition

40. INCOSE CAB Briefing November 2002 Outline Background on CSE, COSYSMO, and COCOMO II COSYSMO Overview –Operational concept and scope –Prototype demo Model Progress to Date –Front end sizing and drivers –Full life cycle sizing and drivers Calendar of activities/milestones Action items How can the CAB help?

41. INCOSE CAB Briefing November 2002 Parametric Cost Model Critical Path Usual # Months* 6Converge on cost drivers, WBS 6Converge on detailed definitions and rating scales 12Obtain initial exploratory dataset (5-10 projects) 6Refine model based on data collection & analysis experience 12+Obtain IOC calibration dataset (30 projects) 9Refine IOC model and tool Critical Path Task *Can be shortened and selectively overlapped

42. INCOSE CAB Briefing November 2002 Calendar of Activities: 2003 (opportunities to accelerate tasks) Telecon 20032004 INCOSE 2003 USC CSE Annual Research Review INCOSE Fall Workshop COCOMO Forum Practical Software & Systems Measurement Workshop Conference on Systems Integration DJFMAMJJASOND Paper & tutorial submitted Practical Software & Systems Measurement Workshop

43. INCOSE CAB Briefing November 2002 Action Items from Last Week 1.Develop a project Plan 2.Address technology maturity/obsolescence 3.Refine driver definitions to incorporate ISO/IEC 15288 definitions 4.Incorporate System and People idea 5.Refine drivers applicability matrix 6.Develop data collection strategy 7.Generate Data Collection Form 8.Update Stakeholder Cohesion to include diversity, identification and trust

44. INCOSE CAB Briefing November 2002 Outcomes From Last Week’s Workshop Reach consensus on resolving the issues Converge on scope of COSYSMO-IP model Address INCOSE issues Address definitions of model parameters Discuss data collection process Promote involvement by Affiliates Define next steps for CSI and INCOSE conferences

45. INCOSE CAB Briefing November 2002 How can the CAB help? Model Calibration Data –COSYSMO IOC will be delivered within 9 months of having 30 “clean” data points Commitment of resources to assist with model and data definition and collection Your support for our proposal to INCOSE SECOE Help in obtaining lead participants from other INCOSE Corporate Members Establish COSYSMO “owner” within INCOSE –Measurement Working Group willing Data, Data, Data

46. INCOSE CAB Briefing November 2002 Key Members of the COSYSMO Working Group Karen Owens, Marilee Wheaton Evin Stump Garry Roedler, Gary Hafen Gary Thomas, John Rieff Tony Jordano, Don Greenlee Chris Miller Marilee Wheaton Cheryl Jones Barry Boehm, Elliot Axelband, Don Reifer, Ricardo Valerdi Aerospace Corp. Galorath LMCO Raytheon SAIC SPC TRW US Army/PSSM USC

47. INCOSE CAB Briefing November 2002 Points of Contact Dr. Barry Boehm [boehm@sunset.usc.edu] Dr. Elliot Axelband [axelband@usc.edu] Don Reifer [dreifer@earthlink.net] Ricardo Valerdi [rvalerdi@sunset.usc.edu] Website http://sunset.usc.edu

48. INCOSE CAB Briefing November 2002 Backup Charts

49. INCOSE CAB Briefing November 2002 USC-CSE Research ($10M backlog) DARPA/Army: Model applications and extensions for Future Combat Systems DARPA: Architectures for mobile distributed systems (DASADA) FAA: Acquisition processes; COCOMO security extensions NASA: Empirical methods for High Dependability Computing NSF: Center for Empirically-Based Software Engineering (with U. of Maryland) NSF: Strategic Design (with CMU, Virginia, Washington) Industry Affiliates’ program

50. INCOSE CAB Briefing November 2002 General Affiliate Benefits Affiliates-only Web portal –Early access to tools, methods, papers, talks, student resumes Tools: COCOMO Suite, Architecture tools, WinWin Technical Report series Workshops on Affiliate-prioritized topics Annual Research Review and Steering Group meeting Annual one-day professor-visit Bilateral visit arrangements; internships Conferences and special workshops Monthly LA SPIN meetings Tutorials and eWorkshops

51. INCOSE CAB Briefing November 2002 Collaboration Modes and Special Benefits Software architecting assistance -Aerospace, Hughes, JPL, Northrop Grumman, TACOM, TRW, Xerox Software process/cost/quality/cycle time assistance -Aerospace, Litton, Microsoft, Northrop Grumman, Raytheon, SAIC, Sun, TACOM, TRW, Xerox Management reviews of critical projects -Litton, Motorola, SAIC, SEI, TRW Reviews of corporate research programs -Daimler Chrysler, Draper Labs, Lockheed Martin, SAIC, SEI, SPC, Telcordia, TRW Joint research contracts -Aerospace, Lockheed Martin, Northrop Grumman, SEI, SPC, TRW Aid in commercializing USC-CSE research -C-Bridge, Galorath, Group Systems.com, Marotz, Price Systems, Rational

52. INCOSE CAB Briefing November 2002 Collaboration Modes and Special Benefits - II Special Projects -Aerospace, Auto Club, FAA, Fidelity, IBM, JPL, Litton, Northrop Grumman, Telcordia Joint workshops on key topics -Aerospace, Motorola, Rational, DOD/SIS, SEI, SPC Focused working groups (COSYSMO) -Aerospace, Galorath, Lockheed Martin, Raytheon, SAIC, SPC, TRW Visiting collaborators -Aerospace, Chung-Ang, C-Bridge, IBM, JPL, Litton, Northrop Grumman, SEI, TRW Corporate State-of-the-art tutorials -Boeing, Chung-Ang, Daimler Chrysler, Draper, EDS, FAA, Fidelity, IBM, JPL, Litton, Lockheed Martin, Lucent, Motorola, Microsoft, Raytheon, SAIC, SEI, SPC, Sun, TRW, Xerox