1. 1 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company 21 st International Forum on Systems, Software, and COCOMO Cost Modeling Systems Engineering Cost Estimation: System-of-Systems Jon K. Ilseng November 8, 2006

2. 2 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Introduction Systems Engineering Definitions COSYSMO SECOST History SECOST Capabilities and Functionality SECOST Cost Estimation Mode: System-of-Interest System-of-Systems Definition System-of-Systems Examples SECOST Cost Estimation Mode: System-of-Systems SECOST Recommended Modifications Summary and Conclusions Agenda

3. 3 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Introduction Systems Engineering Cost Estimates -Our customers require believable and accurate estimates -OSD AT&L encouraging defense contractors to find “most accurate and consistent systems engineering cost estimation method” -Critical that all 5 Raytheon Business Units (IIS, NCS, SAS, IDS, RMS) submit accurate, consistent, and believable cost estimates

4. 4 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Introduction Systems Engineering Cost Estimates -Various cost estimates used past 20 years for a “system-of-interest” > Heuristic and rule of thumb – Systems Engineer use knowledge & experience to prepare cost estimates; no documented written evidence > Expert Opinion – From SE Domain Expert; no scientific or historical basis > Case Studies – Provide vital information; no scientific basis for how cost estimates actually prepared > Top down and design-to-cost – Top-down approach starts at defined system level; tries to capture SE Tasks but not scientific or historical basis; DTC is designing a system to meet cost targets > Bottoms-up – Common approach beginning with lowest level cost component and rolls up to highest level for total estimate; resource intensive effort which is time-consuming and many times no actual historical data to justify estimate > Parametric – Employs cost estimating relationships (CERs); most accurate cost estimation method; provides repeatable and most credible estimation framework; less time-consuming than bottoms-up

5. 5 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Introduction Systems Engineering Cost Estimates -Motivation for improving SE Cost Estimates > Improved estimation process consistency > Improved accuracy, stakeholder confidence due to historical basis –Establish cost directly correlated with customer supplied requirements & sizing artifacts –Parametric Cost Initiative software & hardware study performed a few years back Parametric estimates 25-35% less expensive to develop than “bottoms-up” Parametric estimates every bit as accurate as “bottoms-up” > Improved cost realism, more quantifiable risks –Leaner, more supportable cost bids > Increased Trade Space/Reduced Cycle Time –Some cost estimates require short turn-around time > Better CMMI Support –CMMI model reveals several requirements for cost estimation model as organization moves from Level 2 through Levels 4 & 5

6. 6 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Formal SE Cost Estimation Supports CMMI Level 2 Project Planning Project Monitoring and Control Consistent WBS Size and Complexity Drivers Attribute-Based Estimates Level 3 Integrated Project Monitoring Measurement Repository Monitor Attributes Actuals vs Plan Level 4 Organizational Process Performance Quantitative Project Management Data Collection Parametric Cost Estimation Model Local Model Calibration Level 5 Organizational Innovation and Deployment SE Parametric Modeling Represents an Innovative Approach Introduction

7. 7 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Introduction Parametric Cost Estimation Method -October 2003 OSD SE Summit > OSD’s position is parametric-based estimates are recommended technique for preparing SE Cost Estimates -Why Parametric Cost Estimation Method > Provides a credible source > Shortens cost estimating cycle times > Creates more easily defended negotiation position with customer > Reduces customer-approval cycle times > Uses historical data to improve quality of cost estimates > Establishes greater consistency in cost estimating process > DCAA/DCMA Government agencies very supportive of parametrics

8. 8 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Systems Engineering Definitions Systems Engineering -Various definitions across SE Domain > Applying scientific & engineering efforts to integrate related technical parameters > System solution which satisfies customers’ expectations > Interdisciplinary approach which involves integrating various engineering disciplines (i.e., electrical design, software design, hardware design -Notice something missing from these various definitions! > No mention of cost estimating, however, does not diminish importance of it -Pro-active aggressive SE process during program life-cycle > Lower life cycle costs > High system quality & enhanced technical solution > Minimizes cost & schedule overruns

9. 9 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Systems Engineering Definitions System-of-Interest -Defined by ISO/IEC 15288 architectural structures -Comprised of interacting system elements -Each system element independent of each other; can operate on their own -Only provides real value when connected together to provide system functions > System > A system > System Element > is completely composed of > a set of interacting > system elements System Structure

10. 10 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company COSYSMO Consortium developed COnstructive SYStems Engineering Cost MOdel (COSYSMO) parametric model > University of Southern California Center for Software Engineering (USC/CSE) > USC/CSE Industrial Affiliates (including Raytheon) COSYSMO Cost Model -Accurately estimates time & effort for SE tasks -Parametric-based cost model -Successfully defended August 2005 doctoral dissertation -Open public domain model -COSYSMO led by Dr. Barry Boehm, USC/CSE, developer of COCOMO family of software cost models -COSYSMO drawn significant interest & support from INCOSE

11. 11 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST - History 2.Raytheon Affiliate builds initial version of USC’s COSYSMO -Early 2001 -“MyCOSYSMO” leveraged off Garland’s SWCOST > COCOMO II based > Proven tool - used at Garland and other IIS sites for 8+ years 3.SECOST developed as Raytheon’s Proprietary version of MyCOSYSMO -Early 2004 -Added back Company Proprietary functionality 4.SECOST deployed at several Raytheon business units as a “second opinion” for proposals 1.Beginning development of COSYSMO ­Initiated by paper walked-on in 2000 by a Raytheon Affiliate to USC/CSE -Joint effort by USC/CSE and INCOSE

12. 12 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Capabilities and Functionality Suite of MS Excel spreadsheets Generates, documents, and archives SE cost estimates within single process-focused framework Supports ROMs, budgetary estimates, formal proposal bids Supports multiple levels of estimate formality & complexity Consists of SECOST Framework USC COSYSMO is embedded engine Interfaces with standard Raytheon Pricing Systems Supports Cost Volume & generates Basis-of-Estimates In-Process & Historical Data Collection -Results used for local COSYSMO model calibrations -Local COSYSMO model calibrations feeds USC COSYSMO

13. 13 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Estimation Mode: System-of- Interest Cost estimation mode prepare SE cost estimates for future pursuits Data collection mode collects SE labor hours expended during program execution Current Raytheon NCS Systems Engineering Cost Estimation Enabler -Using SECOST Size Drivers (Requirements, Interfaces, Algorithms, Operational Scenarios) -Using SECOST Size Drivers Complexity Criteria -Using SECOST EREQ Conversion and Reuse Factors

14. 14 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Estimation Mode: System-of- Interest Focus on Cost Estimation Mode Cost Estimation Mode 15 steps -1) Initialize Project Parameters (e.g. project name, period of performance, type of estimate) -2) Enter SE Contractor Work Breakdown Structure (e.g. Technical Management, IV&V, Requirements Definition & Validation) -3) Document Project Assumptions > Assumptions always associated with SE cost estimates -4) Document and Register Project Risks > Program risks always associated with SE cost estimates

15. 15 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Estimation Mode: System-of- Interest (continued) Cost Estimation Mode 15 steps (continued) -5) Set COSYSMO Effort Multipliers (continued) > Application Effort Multiplier – evaluates specific COSYSMO application factors on scale from Very Low to Extremely High –Requirements Understanding –Architecture Understanding –Level of Service Requirements –Migration Complexity –Number & Diversity of Installations/Platforms –Number of Recursive Levels in Design –Documentation to match lifecycle needs –Technology Risk

16. 16 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Estimation Mode: System-of- Interest (continued) Cost Estimation Mode 15 steps (continued) -5) Set COSYSMO Effort Multipliers (continued) > Team Effort Multiplier – evaluates specific COSYSMO team factors on scale from Very Low to Extremely High –Stakeholder Team Cohesion –Personnel/Team Capability –Personnel Experience/Continuity –Process Capability –Multisite Coordination –Tool Support -6) Determine Labor Distributions among Raytheon Salary Labor Grades -7) Estimate Four SE Size Drivers > System-Level Requirements –Decompose system-of-interest objectives & capabilities into requirements that can be tested, verified, or designed –Count number of requirements (“shalls”) in system specification; only requirements managed by SE – not HW or SW

17. 17 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Estimation Mode: System-of- Interest (continued) Cost Estimation Mode 15 steps (continued) -7) Estimate Four SE Size Drivers (continued) > System-Level External & Internal Interfaces –Functional interfaces (e.g. protocols or timing requirements) not physical interfaces (e.g. number of wires) –Interfaces that involve SE for your defined system-of-interest –Only count number of unique interface types – not every interface > System-Level Algorithms –Algorithm sources are functional block diagram, mode description document, system specification, etc. > System-Level Operational Scenarios –Typically quantified by number of system test thread packages, unique end-to- end tests, number of use cases -8) Determine Effort Hours > Outputs total SE hours and equivalent requirements (EREQs)

18. 18 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Estimation Mode: System-of- Interest (continued) Cost Estimation Mode 15 steps (continued) -9) Time Phase SE Estimate > Spread total SE hours among CWBS -10) Submit to Pricing Group > Pricing analyst processes SECOST file (e.g. adds appropriate Raytheon Business Unit, correct CLIN, other pricing variables) -11) Process Pricing Group Data > After pricing analyst processes SECOST file, sent back to SE estimator to copy and paste process SECOST file into SECOST Worksheet -12) Conduct Internal Estimate Review > Internal review among SE Estimator, Lead SE, Program Manager -13) Determine and Signoff Final Bid > After internal SE Review completed & approved by SE Center Director, final cost estimate presented to Raytheon Senior Management

19. 19 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Estimation Mode: System-of- Interest (continued) Cost Estimation Mode 15 steps (continued) -14) Finalize Management Bid Review Charts > SECOST provides four management review package charts –SE Labor Cost Summary Chart –Past Program SE Sizing and Unit Cost –Monte Carlo Output Sample Distribution –Monte Carlo Output Cost “Probability of Success” -15) Archive the Estimate > Most important step; provides rationale and data if questions or issues are raised during cost estimation phase

20. 20 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Definition System-of-Systems (SoS) is not the same as a Family-of-Systems (FoS) -FoS do not create capability beyond additive sum of member systems’ individual capabilities -FoS belong to domain or product lines (e.g. family of missiles, family of aircraft) -FoS lacks synergy of a SoS -FoS do not acquire qualitatively new properties as result of its grouping U.S Department of Defense (DoD) SoS Definition -“A SoS is a set or arrangement of interdependent systems that are related or connected to provide a given capability. The loss or any part of the system will significantly degrade the performance or capabilities of the whole. The development of a system of systems solution will involve trade space between the systems as well within an individual system’s performance.”

21. 21 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Definition My thesis used the U.S. DoD SoS definition SoS Characteristics -Researched five main sources which truly defined SoS > Addressing the System of Systems Challenge Paper > Purdue University School of Aeronautics & Astronautics > “Systems of System Approaches in U.S. Department of Defense” presentation at 1 st Annual SoS Engineering Conference > SoS Engineering Center of Excellence > “System of Systems Engineering” presentation at 1 st Annual SoS Engineering Conference -Four SoS common characteristics shared by five SoS definitions > Emergence – Whole is greater than the sum of its parts; SoS behave as collective whole, interacting with its environment to adapt and respond > Independence – Each system within SoS can operate on their own > Lack of Ownership – SoS does not have an identified owner at SoS level > Evolutionary – SoS is never completely formed; continues to be a living system

22. 22 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples U.S. DoD Programs SoS examples in commercial world (e.g. internet) Focus on U.S. DoD Programs -Raytheon’s primary customer is U.S. DoD -DoD driving towards mandating “jointness for services (i.e. Air Force, Army, Navy, Marine Corps) -Current GWOT & OIF campaigns have Airmen, Sailors, Soldiers, Marines fighting together as joint units -Emphasis on jointness forced services to develop warfighting strategies to support joint warfighting -Current C2ISR, communications & computers, COE capabilities need integration to support joint warfighting -Integration of these capabilities provides integrated capability-centric jointness system; in other words a SoS

23. 23 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples Future Combat Systems SoS -SoSCOE > Software that allows various systems to operate seamlessly > Approximately 35 million lines of code -Battle Command Software > Consists of four software packages –Mission Planning & Preparation, Situation Understanding, Battle Command & Mission Execution, Warfighter-Machine Interface -Communications & Computers > FCS SoS connected to C4ISR network by multilayered Communications & Computer network > Network provides secure access to information sources over extended distances & complex terrain -ISR > Distributed & networked array of ISR Sensors -Networked Logistics Systems > Integrates logistics into C4ISR network

24. 24 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples Networked SoS which develops combat power, sustainability, agility, and versatility for full spectrum military operations Networked SoS manned by soldiers & fighting team of teams

25. 25 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples DoD Distributed Common Ground System SoS -Combination of U.S. Air Force, Army, Navy, and Marine Corps ground and surface systems -Each service’s DCGS consists of elements and processes, exploits, and posts ISR sensor data -Each service’s DCGS consists of legacy systems -DoD currently preparing migration plans to integrate all service DCGS elements > Achieves a net-centric DCGS > Integration of services’ DCGS is referred to as DoD DCGS SoS -DoD mandated DCGS SoS migrate to net-centric warfare & net-centric DCGS Enterprise

26. 26 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples Improves accuracy and timeliness of intelligence provided to warfighter Promotes standards-based ISR infrastructure

27. 27 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples Land Warrior SoS -High-tech SoS which provides U.S. Army soldier enhanced capabilities -Integrated fighting system which helps increase soldier’s > Lethality > Battle Command Compatibility > Survivability > Mobility > Awareness > Situational Awareness > Combat effectiveness

28. 28 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples Land Warrior SoS -Consists of following subsystems > Weapon Subsystem –Integrates weapon-mounted sensors (multifunction laser, daylight video sight, thermal weapon sight) > Soldier Control Unit –Provides primary user interface to system functions > Personal Area Network Cables –Distributes power & data through the system > Personal Clothing & Individual Equipment –Consists of utility belt & subsystem pouches > Computer/Master Hub Subsystem –Provides control of system functions

29. 29 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples Land Warrior SoS -Consists of following subsystems > Power Source Subsystem –Provides centralized power from dual disposable or rechargeable batteries > CommsNet Radio Subsystem –Provides transmit/receive voice & data capability > Navigation Subsystem –Provides position location data to the soldier & time reference to system > Helmet Subsystem –Provides full-color display for computer interface

30. 30 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples NASA Exploration SoS -Represents U.S. President’s vision for U.S. space exploration -New capabilities & systems enabling safe & successful human & robotic missions

31. 31 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company System-of-Systems Examples NASA Exploration SoS -Consists of following subsystems > Crew Transportation System –Flight elements which deliver human crew from Earth to mission destination & return crew safely to Earth > Cargo Delivery System –Delivers all non-crew exploration vehicle flight elements to accomplish human exploration objectives > Ground Support System –Provides all common ground-based capabilities needed to execute exploration missions > Robotic Precursor System –Provides measurements, technology, & demonstrations in advance of human missions > In-Space Support System –Encompasses capabilities provided by space-based infrastructure elements (e.g. communications, navigation, surveillance) > Destination Surface System –Encompasses all elements necessary to enable long-duration human exploration mission

32. 32 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Estimation Mode: System-of- Systems Can SECOST be used for SoS -Yes, with recommended modifications & additions to COSYSMO Size Drivers & Application and Team Effort Multipliers -SECOST does not account for SoS characteristics > Emergence, Independence, Lack of Ownership, Evolutionary > Complex Integration Efforts –Many domains involved > Design Optimization –Approach not feasible for SoS; have to evaluate each system within SoS to determine best optimal design > Complex interface design & management –Management of SoS interfaces more difficult than for system-of-interest > Decomposing FoS functional requirements & allocating to SoS –No SoS hierarchy; results in stovepipe solutions > SoS IV&V –More complex than system-of-interest IV&V

33. 33 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Recommended Modifications Add Software Requirements Size Driver -Previously mentioned SoS programs have extensive software-level requirements > DoD DCGS SoS has over 2,000 unique software requirement -Significant costs associated with numerous software requirements & software integration

34. 34 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Recommended Modifications Add Software Modules Size Driver -Software module defined as CSCI > Newly developed, COTS, GFE -Previous mentioned SoS Programs contained numerous CSCIs > DoD DCGS SoS has over 3,000 CSCIs

35. 35 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Recommended Modifications Modify Interfaces Complexity Criteria -Managing SoS interfaces complex (e.g. FCS SoS has over 2,000 unique interfaces)

36. 36 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Recommended Modifications Modify COSYSMO Application Effort Multiplier -Requirements Understanding > Rates the level of understanding of the SoS system and software requirements -Migration Complexity, Number of Diversity of Installations/Platforms, Number of Recursive Levels in the Design, Documentation to Match Lifecycle Needs > Add word “SoS” to each of these COSYSMO Application Factors -Add IV&V COSYSMO Application Factor > Rates maturity & experience of performing SoS IV&V tasks -Add SoS Integration COSYSMO Application Factor > Rates maturity & experience of performing as a SoS integrator on previous SoS programs

37. 37 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company SECOST Cost Recommended Modifications Modify COSYSMO Team Effort Multiplier -Stakeholder Team Cohesion > Add the following viewpoints “What is the number of stakeholders involved?” and “Is there a defined and agreed to list of stakeholder responsibilities?” -Add word “SoS” to Personnel Experience/Continuity COSYSMO Team Factors > Add words “and experience of systems engineers who have worked on SoS programs”

38. 38 SECOST is a tool developed by Raytheon and used with permission 10/21/06 Copyright © 2006 Raytheon Company Summary and Conclusions October 2003 OSD SE Summit > OSD’s position is parametric-based estimates are recommended technique for preparing SE Cost Estimates Current SE Cost Estimation Method Raytheon parametric-based SECOST Tool -Provides SE cost estimates for system-of-interest DoD SoS Programs are increasing in importance and come with challenges No credible method for performing SoS SE cost estimates Parametric-based SECOST tool can predict SoS SE cost estimates -With additions & modifications to SECOST Parametric-based SECOST tool can provide accurate, credible & believable SoS SE cost estimates