LASPIN-INCOSE – 7/30/031 LASPIN-INCOSE Meeting Los Angeles, CA July 30, 2003 Ricardo Valerdi The Aerospace Corporation & University of Southern California Center for Software Engineering

LASPIN-INCOSE – 7/30/032 Outline Goals of this talk Background, key ideas, and definitions Overview of COSYSMO Delphi results Estimation example Data collection process Demo Next steps

LASPIN-INCOSE – 7/30/033 Goals of this talk 1.Provide an overview of COSYSMO 2.Demonstrate the prototype 3.Share future plans for data collection 4.Get feedback/suggestions on approach and data collection

LASPIN-INCOSE – 7/30/034 “All models are wrong, but some of them are useful” - W. E. Deming Source:

LASPIN-INCOSE – 7/30/035 Key Definitions & Concepts Calibration: the tuning of parameters based on project data CER: a model that represents the cost estimating relationships of factors Cost Estimation: prediction of both the person-effort and elapsed time of a project Driver: A factor that is highly correlated to the amount of Systems Engineering effort Parametric: an equation or model that is approximated by a set of parameters Rating Scale: a range of values and definitions for a particular driver Understanding: an individual’s subjective judgment of their level of comprehension

LASPIN-INCOSE – 7/30/036 COCOMO II COCOMO is the most widely used, thoroughly documented and calibrated software cost model COCOMO - the “COnstructive COst MOdel” –COCOMO II is the update to COCOMO 1981 –ongoing research with annual calibrations made available Originally developed by Dr. Barry Boehm and published in 1981 book Software Engineering Economics COCOMO II described in Software Cost Estimation with COCOMO II (Prentice Hall 2000)

LASPIN-INCOSE – 7/30/037 USC Center for Software Engineering (CSE) Researches, teaches, and practices CMMI-based Software engineering –Systems and software engineering fully integrated Collaborative efforts between Computer Science (CS) and Industrial Systems Engineering (ISE) Departments COCOMO Suite of models –Cost, schedule: COCOMO II, CORADMO, COCOTS –Quality: COQUALMO –Systems Engineering: COSYSMO Applies and extends research on major programs (DARPA/Army, FCS, FAA ERAM, NASA Missions) Uses mature 7-step model development methodology

LASPIN-INCOSE – 7/30/038 Marilee Wheaton, Aerospace/INCOSE Corporate Advisory Board Dave Hickman, Aerospace/INCOSE Corporate Advisory Board Dr. Barry Boehm, USC/INCOSE Fellow Dr. Elliot Axelband, USC/INCOSE Don Greenlee, SAIC/INCOSE V&V Eric Honour, INCOSE SECOE Paul Robitaille, LMCO/INCOSE Corporate Advisory Board Garry Roedler, LMCO/ISO-IEC Chris Miller, SPC/INCOSE Measurement Working Group Dr. John E. Rieff, Raytheon/INCOSE Corporate Advisory Board Jim VanGaasbeek, Northrop/INCOSE Corporate Advisory Board INCOSE Involvement

LASPIN-INCOSE – 7/30/039 Commercial Industry (15) –Daimler Chrysler, Freshwater Partners, Galorath, Group Systems.Com, Hughes, IBM, Cost Xpert Group, Microsoft, Motorola, Price Systems, Rational, Reuters Consulting, Sun, Telcordia, Xerox Aerospace Industry (6) –BAE, Boeing, Lockheed Martin, Northrop Grumman, Raytheon, SAIC Government (8) –DARPA, DISA, FAA, NASA-Ames, NSF, OSD/ARA/SIS, US Army Research Labs, US Army TACOM FFRDC’s and Consortia (4) –Aerospace, JPL, SEI, SPC International (1) –Chung-Ang U. (Korea) USC-CSE Affiliates (34) *COSYSMO Contributors

LASPIN-INCOSE – 7/30/ step Modeling Methodology Analyze Existing literature Perform Behavioral Analysis Identify Relative Significance Perform Expert- Judgement, Delphi Assessment Gather Project Data Determine Bayesian A-Posteriori Update Gather more data; refine model Determine statistical significance

LASPIN-INCOSE – 7/30/0311 COCOMO II Software Development phases 20+ years old 200+ calibration points 23 Drivers Variable granularity 3 anchor points Size is driven by SLOC COSYSMO Systems Engineering Entire Life Cycle 2 years old ~3 calibration points 18 drivers Fixed granularity No anchor points Size is driven by requirements, I/F, etc. Model Differences

LASPIN-INCOSE – 7/30/0312 COSYSMO: Overview Parametric model to estimate system engineering costs Includes 4 size & 14 cost drivers Covers full system engineering lifecycle Focused on use for Investment Analysis, Concept Definition phases estimation, tradeoff & risk analyses –Input parameters can be determined in early phases

LASPIN-INCOSE – 7/30/0313 COSYSMO Size Drivers Effort Multipliers Effort Calibration # Requirements # Interfaces # Scenarios # Algorithms + Volatility Factor - Application factors -8 factors - Team factors -6 factors - Schedule driver WBS guided by ISO/IEC COSYSMO Operational Concept

LASPIN-INCOSE – 7/30/0314 EIA/ANSI 632 EIA/ANSI Provide an integrated set of fundamental processes to aid a developer in the engineering or re-engineering of a system Breadth and Depth of Key SE Standards System life ISO/IEC Level of detail ConceptualizeDevelop Transition to Operation Operate, Maintain, or Enhance Replace or Dismantle Process description High level practices Detailed practices ISO/IEC Establish a common framework for describing the life cycle of systems Purpose of the Standards: IEEE 1220 IEEE Provide a standard for managing systems engineering Source : Draft Report ISO Study Group May 2, 2000

LASPIN-INCOSE – 7/30/0315 Data will drive the Evolution Path & Scope of the Model Oper Test & Eval 1. COSYSMO-IP 2. COSYSMO-C4ISR 3. COSYSMO-Machine 4. COSYSMO-SoS Global Command and Control System Satellite Ground Station Joint Strike Fighter Future Combat Systems DevelopConceptualize Transition to Operation Operate, Maintain, or Enhance Replace or Dismantle

LASPIN-INCOSE – 7/30/0316 COCOMO-based Parametric Cost Estimating Relationship Where: PM NS = effort in Person Months (Nominal Schedule) A = constant derived from historical project data Size = determined by computing the weighted average of the size drivers E = exponent representing the economy/diseconomy of scale dependent on size drivers (4), currently set to 1 n = number of cost drivers (14) EM = effort multiplier for the i th cost driver. The geometric product results in an overall effort adjustment factor to the nominal effort.

LASPIN-INCOSE – 7/30/ Size Drivers 1. Number of System Requirements 2. Number of Major Interfaces 3. Number of Operational Scenarios 4. Number of Critical Algorithms Each weighted by complexity, volatility, and degree of reuse

LASPIN-INCOSE – 7/30/0318 Number of System Requirements This driver represents the number of requirements for the system-of-interest at a specific level of design. Requirements may be functional, performance, feature, or service-oriented in nature depending on the methodology used for specification. They may also be defined by the customer or contractor. System requirements can typically be quantified by counting the number of applicable “shall’s” or “will’s” in the system or marketing specification. Do not include a requirements expansion ratio – only provide a count for the requirements of the system-of-interest as defined by the system or marketing specification. EasyNominalDifficult - Well specified- Loosely specified- Poorly specified - Traceable to source- Can be traced to source with some effort - Hard to trace to source - Simple to understand- Takes some effort to understand- Hard to understand - Little requirements overlap- Some overlap- High degree of requirements overlap - Familiar- Generally familiar- Unfamiliar - Good understanding of what’s needed to satisfy and verify requirements - General understanding of what’s needed to satisfy and verify requirements - Poor understanding of what’s needed to satisfy and verify requirements

LASPIN-INCOSE – 7/30/ Cost Drivers 1. Requirements understanding 2. Architecture complexity 3. Level of service requirements 4. Migration complexity 5. Technology Maturity 6. Documentation Match to Life Cycle Needs 7. # and Diversity of Installations/Platforms 8. # of Recursive Levels in the Design Application Factors (8)

LASPIN-INCOSE – 7/30/0320 Requirements understanding This cost driver rates the level of understanding of the system requirements by all stakeholders including the systems, software, hardware, customers, team members, users, etc. Very lowLowNominalHighVery High Poor, unprecedented system Minimal, many undefined areas Reasonable, some undefined areas Strong, few undefined areas Full understanding of requirements, familiar system

LASPIN-INCOSE – 7/30/ Cost Drivers (cont.) 1. Stakeholder team cohesion 2. Personnel/team capability 3. Personnel experience/continuity 4. Process maturity 5. Multisite coordination 6. Tool support Team Factors (6)

LASPIN-INCOSE – 7/30/0322 Stakeholder team cohesion Represents a multi-attribute parameter which includes leadership, shared vision, diversity of stakeholders, approval cycles, group dynamics, IPT framework, team dynamics, trust, and amount of change in responsibilities. It further represents the heterogeneity in stakeholder community of the end users, customers, implementers, and development team. Very LowLowNominalHighVery High Culture  Stakeholders with diverse expertise, task nature, language, culture, infrastructure  Highly heterogeneous stakeholder communities  Heterogeneous stakeholder community  Some similarities in language and culture  Shared project culture  Strong team cohesion and project culture  Multiple similarities in language and expertise  Virtually homogeneous stakeholder communities  Institutionalized project culture Communication  Diverse organizational objectives  Converging organizational objectives  Common shared organizational objectives  Clear roles & responsibilities  High stakeholder trust level

LASPIN-INCOSE – 7/30/0323 Additional Proposed Drivers # and diversity of installations/platforms phased out # of years in operational life cycle Quality Attributes Manufacturability/Producibility Degree of Distribution

LASPIN-INCOSE – 7/30/0324 Delphi Results From last COSYSMO Working Group Meeting in Keystone, CO 2 rounds Avg. increased for all drivers except TOOL About 1/3 of the responses were adjusted as a result of the discussion Goal is to get 30+ Delphi surveys filled out by October Gary Hafen, LMCO Barry Boehm, USC Don Reifer, USC Paul Mohlman, Aero Rob Flowe, DAU Rick Edison, LMCO Gary Thomas, Raytheon Garry Roedler, LMCO Carl Newman, LMCO

LASPIN-INCOSE – 7/30/0325

LASPIN-INCOSE – 7/30/0326

LASPIN-INCOSE – 7/30/0327

LASPIN-INCOSE – 7/30/0328 Estimation Example You are the SE working on an upgrade of a legacy system with a pretty good understanding of requirements (“Nominal” rating)… Very lowLowNominalHighVery High Poor, unprecedented system Minimal, many undefined areas Reasonable, some undefined areas Strong, few undefined areas Full understanding of requirements, familiar system 1.0 Requirements Understanding rating scale: You estimate that the job will require 12 Person Months. This driver will have no additional impact on the cost of the job (effort is multiplied by 1.0).

LASPIN-INCOSE – 7/30/0329 Estimation Example (cont) …all of a sudden…the customer adds a requirement to make the new system backwards compatible with the old one. Your overall understanding of the requirements has decreased, your schedule and resources are fixed, and the amount of work has increased. Mayday!

LASPIN-INCOSE – 7/30/0330 Estimation Example (cont) Justify to Program Manager/Director your request to increase resources in this area. Very lowLowNominalHighVery High Poor, unprecedented system Minimal, many undefined areas Reasonable, some undefined areas Strong, few undefined areas Full understanding of requirements, familiar system 1.0 Requirements Understanding rating scale: The effort is multiplied by 1.2! Instead of 12 PM, this job will require 14.4 PM of Systems Engineering work (additional $19.2k*) *assumes $8k/PM

LASPIN-INCOSE – 7/30/0331 Data Collection Process Project & people are identified Systems engineer Cost estimator/data base manager Job/task codes in accounting system are mapped to COSYSMO Meta data is collected System scope Life cycle Application domain Cost drivers are rated Interaction between SE, Cost, USC Data is entered into secure repository at USC

LASPIN-INCOSE – 7/30/0332 Safeguarding Procedures Data identification –Only affiliate & Dr. Boehm know the OID (XXX) and only affiliate knows PID (YYY) Data storage –Stand-alone computer at USC with one-way access to the network –In a room with cypher lock & limited access Data access –Non-disclosure agreements signed –Controlled access to data by researchers (US Citizens only)

LASPIN-INCOSE – 7/30/0333 USC/Raytheon myCOSYSMO* Demo *Developed by Gary Thomas at Raytheon Garland

LASPIN-INCOSE – 7/30/0334 Parametric Cost Model Critical Path Usual # Months* 6Converge on cost drivers, WBS 6Converge on detailed definitions and rating scales 12Obtain initial exploratory dataset (5-10 projects) 6Refine model based on data collection & analysis experience 12+Obtain IOC calibration dataset (30 projects) 9Refine IOC model and tool Critical Path Task *Can be shortened and selectively overlapped

LASPIN-INCOSE – 7/30/0335 Calendar of Activities: INCOSE 2003 (Washington, DC) USC CSE Annual Research Review (Los Angeles, CA) INCOSE IW (Tampa, FL) COCOMO Forum (Los Angeles, CA) Conference on Systems Integration (Hoboken, NJ) JFMAMJJASOND Practical Software & Systems Measurement Workshop (Keystone, CO) Working Group Meeting INCOSE/SCEA Meeting (Chantilly, VA)

LASPIN-INCOSE – 7/30/0336 Don’t be left out!

LASPIN-INCOSE – 7/30/0337 Questions or Comments? Ricardo Valerdi Websites Books Boehm, B., et al, Software Cost Estimation with COCOMOII, 1 st Ed, Prentice Hall, 2000 Blanchard, B., Fabrycky, W., Systems Engineering and Analysis, 3 rd Ed, Prentice Hall, 1998 Boehm, B., Software Engineering Economics, 1 st Ed, Prentice Hall, 1981