1 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company 21 st International Forum on COCOMO and Software Cost Modeling Systems Engineering Cost Estimation: System-of-Systems Jon K. Ilseng Principal Systems Engineer Raytheon Network Centric Systems November 8, 2006

2 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Introduction Systems Engineering Definitions SECOST History SECOST Capabilities and Functionality SECOST Cost Estimation Mode: System-of-Interest System-of-Systems Definition System-of-Systems Examples SECOST Cost Estimation Mode: System-of-Systems SECOST Recommended Modifications Summary and Conclusions Agenda

3 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Introduction Systems Engineering Cost Estimates -Our customers require believable and accurate estimates -OSD AT&L encouraging defense contractors to find “most accurate and consistent systems engineering cost estimation method” -Critical that all 5 Raytheon Business Units (IIS, NCS, SAS, IDS, RMS) submit accurate, consistent, and believable cost estimates

4 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Introduction Systems Engineering Cost Estimates -Various cost estimates used past 20 years for a “system-of-interest” > Heuristic and rule of thumb – Systems Engineer use knowledge & experience to prepare cost estimates; no documented written evidence > Expert Opinion – From SE Domain Expert; no scientific or historical basis > Case Studies – Provide vital information; no scientific basis for how cost estimates actually prepared > Top down and design-to-cost – Top-down approach starts at defined system level; tries to capture SE Tasks but not scientific or historical basis; DTC is designing a system to meet cost targets > Bottoms-up – Common approach beginning with lowest level cost component and rolls up to highest level for total estimate; resource intensive effort which is time-consuming and many times no actual historical data to justify estimate > Parametric – Employs cost estimating relationships (CERs); most accurate cost estimation method; provides repeatable and most credible estimation framework; less time-consuming than bottoms-up

5 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Introduction Parametric Cost Estimation Method -October 2003 OSD SE Summit > OSD’s position is parametric-based estimates are recommended technique for preparing SE Cost Estimates -Why Parametric Cost Estimation Method > Provides a credible source > Shortens cost estimating cycle times > Creates more easily defended negotiation position with customer > Reduces customer-approval cycle times > Uses historical data to improve quality of cost estimates > Establishes greater consistency in cost estimating process -Consortium developed Constructive Systems Engineering Cost Model (COSYSMO) parametric model > U.S. Defense Contractors including Raytheon > International Council of Systems Engineering > University of Southern California Center for Software Engineering

6 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Introduction COSYSMO Cost Model -Accurately estimates time & effort for SE tasks -Parametric-based cost model -Successfully defended August 2005 doctoral disseration -Open public domain model SECOST Tool -Developed by Raytheon IIS-Garland for SE Cost Estimates > Uses COSYSMO as its embedded estimation engine > Prepares SE Cost Estimates for System-of-Interest > Raytheon proprietary -Could SECOST Tool prepare SE Cost Estimates for System-of-System > Yes, if SECOST additions & modifications are implemented

7 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Systems Engineering Definitions Systems Engineering -Various definitions across SE Domain > Applying scientific & engineering efforts to integrate related technical parameters > System solution which satisfies customers’ expectations > Interdisciplinary approach which involves integrating various engineering disciplines (i.e., electrical design, software design, hardware design -Notice something missing from these various definitions! > No mention of cost estimating, however, does not diminish importance of it -Pro-active aggressive SE process during program life-cycle > Lower life cycle costs > High system quality & enhanced technical solution > Minimizes cost & schedule overruns

8 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Systems Engineering Definitions System-of-Interest -Defined by ISO/IEC architectural structures -Comprised of interacting system elements -Each system element independent of each other; can operate on their own -Only provides real value when connected together to provide system functions > System > A system > System Element > is completely composed of > a set of interacting > system elements System Structure

9 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST History Suite of MS Excel spreadsheets Generates, documents, and archives SE cost estimates within single process-focused framework Uses open COSYSMO parametric model Developed in early 2004 from MyCOSYSMO -MyCOSYSMO leveraged off SWCOST software engineering estimation model Proprietary version of MyCOSYSMO Raytheon IIS and NCS Business Units currently collecting historical program data for local calibration

10 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Capabilities and Functionality Supports ROMs, budgetary estimates, formal proposal bids Supports multiple levels of estimate formality & complexity Consists of SECOST Framework USC COSYSMO is embedded engine Interfaces with standard Raytheon Pricing Systems Supports Cost Volume & generates Basis-of-Estimates In-Process & Historical Data Collection -Results used for local COSYSMO model calibrations -Local COSYSMO model calibrations feeds USC COSYSMO Current Raytheon NCS Systems Engineering Cost Enabler -Using SECOST Size Drivers (Requirements, Interfaces, Algorithms, Operational Scenarios) -Using SECOST Size Drivers Complexity Criteria -Using SECOST EREQ Conversion and Reuse Factors

11 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Estimation Mode: System-of- Interest Cost estimation mode prepare SE cost estimates for future pursuits Data collection mode collects SE labor hours expended during program execution Cost Estimation Mode 15 steps -1) Initialize Project Parameters (e.g. project name, period of performance, type of estimate) -2) Enter SE Contractor Work Breakdown Structure (e.g. Technical Management, IV&V, Requirements Definition & Validation) -3) Document Project Assumptions > Assumptions always associated with SE cost estimates -4) Document and Register Project Risks > Program risks always associated with SE cost estimates

12 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Estimation Mode: System-of- Interest (continued) Cost Estimation Mode 15 steps (continued) -5) Set COSYSMO Effort Multipliers (continued) > Application Effort Multiplier – evaluates specific COSYSMO application factors on scale from Very Low to Extremely High –Requirements Understanding –Architecture Understanding –Level of Service Requirements –Migration Complexity –Number & Diversity of Installations/Platforms –Number of Recursive Levels in Design –Documentation to match lifecycle needs –Technology Risk

13 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Estimation Mode: System-of- Interest (continued) Cost Estimation Mode 15 steps (continued) -5) Set COSYSMO Effort Multipliers (continued) > Team Effort Multiplier – evaluates specific COSYSMO team factors on scale from Very Low to Extremely High –Stakeholder Team Cohesion –Personnel/Team Capability –Personnel Experience/Continuity –Process Capability –Multisite Coordination –Tool Support -6) Determine Labor Distributions among Raytheon Salary Labor Grades -7) Estimate Four SE Size Drivers > System-Level Requirements –Decompose system-of-interest objectives & capabilities into requirements that can be tested, verified, or designed –Count number of requirements (“shalls”) in system specification; only requirements managed by SE – not HW or SW

14 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Estimation Mode: System-of- Interest (continued) Cost Estimation Mode 15 steps (continued) -7) Estimate Four SE Size Drivers (continued) > System-Level External & Internal Interfaces –Functional interfaces (e.g. protocols or timing requirements) not physical interfaces (e.g. number of wires) –Interfaces that involve SE for your defined system-of-interest –Only count number of unique interface types – not every interface > System-Level Algorithms –Algorithm sources are functional block diagram, mode description document, system specification, etc. > System-Level Operational Scenarios –Typically quantified by number of system test thread packages, unique end-to- end tests, number of use cases -8) Determine Effort Hours > Outputs total SE hours and equivalent requirements (EREQs)

15 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Estimation Mode: System-of- Interest (continued) Cost Estimation Mode 15 steps (continued) -9) Time Phase SE Estimate > Spread total SE hours among CWBS -10) Submit to Pricing Group > Pricing analyst processes SECOST file (e.g. adds appropriate Raytheon Business Unit, correct CLIN, other pricing variables) -11) Process Pricing Group Data > After pricing analyst processes SECOST file, sent back to SE estimator to copy and paste process SECOST file into SECOST Worksheet -12) Conduct Internal Estimate Review > Internal review among SE Estimator, Lead SE, Program Manager -13) Determine and Signoff Final Bid > After internal SE Review completed & approved by SE Center Director, final cost estimate presented to Raytheon Senior Management

16 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Estimation Mode: System-of- Interest (continued) Cost Estimation Mode 15 steps (continued) -14) Finalize Management Bid Review Charts > SECOST provides four management review package charts –SE Labor Cost Summary Chart –Past Program SE Sizing and Unit Cost –Monte Carlo Output Sample Distribution –Monte Carlo Output Cost “Probability of Success” -15) Archive the Estimate > Most important step; provides rationale and data if questions or issues are raised during cost estimation phase

17 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Definition System-of-Systems (SoS) is not the same as a Family-of-Systems (FoS) -FoS do not create capability beyond additive sum of member systems’ individual capabilities -FoS belong to domain or product lines (e.g. family of missiles, family of aircraft) -FoS lacks synergy of a SoS -FoS do not acquire qualitatively new properties as result of its grouping U.S Department of Defense (DoD) SoS Definition -“A SoS is a set or arrangement of interdependent systems that are related or connected to provide a given capability. The loss or any part of the system will significantly degrade the performance or capabilities of the whole. The development of a system of systems solution will involve trade space between the systems as well within an individual system’s performance.”

18 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Definition My thesis used the U.S. DoD SoS definition SoS Characteristics -Researched five main sources which truly defined SoS > Addressing the System of Systems Challenge Paper > Purdue University School of Aeronautics & Astronautics > “Systems of System Approaches in U.S. Department of Defense” presentation at 1 st Annual SoS Engineering Conference > SoS Engineering Center of Excellence > “System of Systems Engineering” presentation at 1 st Annual SoS Engineering Conference -Four SoS common characteristics shared by five SoS definitions > Emergence – Whole is greater than the sum of its parts; SoS behave as collective whole, interacting with its environment to adapt and respond > Independence – Each system within SoS can operate on their own > Lack of Ownership – SoS does not have an identified owner at SoS level > Evolutionary – SoS is never completely formed; continues to be a living system

19 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples U.S. DoD Programs SoS examples in commercial world (e.g. internet) Focus on U.S. DoD Programs -Raytheon’s primary customer is U.S. DoD -DoD driving towards mandating “jointness for services (i.e Air Force, Army, Navy, Marine Corps) -Current GWOT & OIF campaigns have Airmen, Sailors, Soldiers, Marines fighting together as joint units -Emphasis on jointness forced services to develop warfighting strategies to support joint warfighting -Current C2ISR, communications & computers, COE capabilities need integration to support joint warfighting -Integration of these capabilities provides integrated capability-centric jointness system; in other words a SoS

20 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples Future Combat Systems SoS -SoSCOE > Software that allows various systems to operate seamlessly > Approximately 35 million lines of code -Battle Command Software > Consists of four software packages –Mission Planning & Preparation, Situation Understanding, Battle Command & Mission Execution, Warfighter-Machine Interface -Communications & Computers > FCS SoS connected to C4ISR network by multilayered Communications & Computer network > Network provides secure access to information sources over extended distances & complex terrain -ISR > Distributed & networked array of ISR Sensors -Networked Logistics Systems > Integrates logistics into C4ISR network

21 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples

22 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples DoD Distributed Common Ground System SoS -Combination of U.S. Air Force, Army, Navy, and Marine Corps ground and surface systems -Each service’s DCGS consists of elements and processes, exploits, and posts ISR sensor data -Each service’s DCGS consists of legacy systems -DoD currently preparing migration plans to integrate all service DCGS elements > Achieves a net-centric DCGS > Integration of services’ DCGS is referred to as DoD DCGS SoS -DoD mandated DCGS SoS migrate to net-centric warfare & net-centric DCGS Enterprise

23 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples Improves accuracy and timeliness of intelligence provided to warfighter Promotes standards-based ISR infrastructure

24 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples Land Warrior SoS -High-tech SoS which provides U.S. Army soldier enhanced capabilities -Integrated fighting system which helps increase soldier’s > Lethality > Battle Command Compatibility > Survivability > Mobility > Awareness > Situational Awareness > Combat effectiveness

25 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples Land Warrior SoS -Consists of following subsystems > Weapon Subsystem –Integrates weapon-mounted sensors (multifunction laser, daylight video sight, thermal weapon sight) > Soldier Control Unit –Provides primary user interface to system functions > Personal Area Network Cables –Distributes power & data through the system > Personal Clothing & Individual Equipment –Consists of utility belt & subsystem pouches > Computer/Master Hub Subsystem –Provides control of system functions

26 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples Land Warrior SoS -Consists of following subsystems > Power Source Subsystem –Provides centralized power from dual disposable or rechargeable batteries > CommsNet Radio Subsystem –Provides transmit/receive voice & data capability > Navigation Subsystem –Provides position location data to the soldier & time reference to system > Helmet Subsystem –Provides full-color display for computer interface

27 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples NASA Exploration SoS -Represents U.S. President’s vision for U.S. space exploration -New capabilities & systems enabling safe & successful human & robotic missions

28 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples NASA Exploration SoS -Consists of following subsystems > Crew Transportation System –Flight elements which deliver human crew from Earth to mission destination & return crew safely to Earth > Cargo Delivery System –Delivers all non-crew exploration vehicle flight elements to accomplish human exploration objectives > Ground Support System –Provides all common ground-based capabilities needed to execute exploration missions > Robotic Precursor System –Provides measurements, technology, & demonstrations in advance of human missions > In-Space Support System –Encompasses capabilities provided by space-based infrastructure elements (e.g. communications, navigation, surveillance) > Destination Surface System –Encompasses all elements necessary to enable long-duration human exploration mission

29 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Estimation Mode: System-of- Systems Can SECOST be used for SoS -Yes, with recommended modifications & additions -SECOST does not account for SoS characteristics > Emergence, Independence, Lack of Ownership, Evolutionary > Complex Integration Efforts –Many domains involved > Design Optimization –Approach not feasible for SoS; have to evaluate each system within SoS to determine best optimal design > Complex interface design & management –Management of SoS interfaces more difficult than for system-of-interest > Decomposing FoS functional requirements & allocating to SoS –No SoS hierarchy; results in stovepipe solutions > SoS IV&V –More complex than system-of-interest IV&V

30 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Recommended Modifications Add Software Requirements Size Driver -SoS have significant costs associated with software integration & numerous software requirements > DoD DCGS SoS has over 2,000 unique software requirements -Previously mentioned SoS programs have extensive software-level requirements

31 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Recommended Modifications Add Software Modules Size Driver -Software module defined as CSCI > Newly developed, COTS, GFE -Previous mentioned SoS Programs contained numerous CSCIs > DoD DCGS SoS has over 3,000 unique CSCIs

32 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Recommended Modifications Modify Interfaces Complexity Criteria -Managing SoS interfaces complex (e.g. FCS SoS has over 2,000 unique interfaces)

33 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Recommended Modifications Modify COSYSMO Application Effort Multiplier -Requirements Understanding > Rates the level of understanding of the SoS system and software requirements -Migration Complexity, Number of Diversity of Installations/Platforms, Number of Recursive Levels in the Design, Documentation to Match Lifecycle Needs > Add word “SoS” to each of these COSYSMO Application Factors -Add IV&V COSYSMO Application Factor > Rates maturity & experience of performing SoS IV&V tasks -Add SoS Integration COSYSMO Application Factor > Rates maturity & experience of performing as a SoS integrator on previous SoS programs

34 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Recommended Modifications Modify COSYSMO Team Effort Multiplier -Stakeholder Team Cohesion > Add the following viewpoints “What is the number of stakeholders involved?” and “Is there a defined and agreed to list of stakeholder responsibilities?” -Add word “SoS” to Personnel Experience/Continuity COSYSMO Team Factors > Add words “and experience of systems engineers who have worked on SoS programs”

35 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Summary and Conclusions 2003 OSD SE Summit -OSD’s position statement is parametric-based estimates recommended technique for preparing SE cost estimates Raytheon IIS Garland developed SECOST -Uses COSYSMO parametric model -Currently predicts SE cost estimates for system-of-interest DoD SoS Programs are increasing in importance and come with challenges No credible method for performing SoS SE cost estimates SECOST can predict SoS SE cost estimates -With additions & modifications to SECOST SECOST can provide accurate, credible & believable SoS SE cost estimates