1. System of Systems Engineering (SoSE) Cost Estimation Jo Ann Lane jolane at usc.edu Presented by Marilee Wheaton November 2010

2. 2 Overview Key definitions Current SE cost model capabilities Extensions for complex systems Extensions for SoSs Alternatives for “# of requirements” size driver Using SoSE cost model to evaluate alternatives Summary

3. 3 Net-Centric SoS Net-Centric Connectivit y What is a “System of Systems”? Very large systems using a framework or architecture to integrate constituent systems Exhibits emergent behavior not otherwise achievable by constituent systems SoS constituent systems (CS) Independently developed and managed New or existing systems in various stages May include multiple COTS products Have their own purpose Can dynamically come and go from SoS Typical domains Business: Enterprise-wide and cross-enterprise integrations Military/Crisis Response: Dynamic communications infrastructure Based on Mark Maier’s SoS definition [Maier, 1998] Laboratory System Imaging Management System Pharmacy System Patient Management System Telemetry System Health Care Network

4. SoS SE Taxonomy in Order of Increasing Authority and Responsibility Virtual [Maier, 1998] Ad hoc Collaborative [Maier, 1998] No formal management at SoS level Acknowledged [Dahmann, 2008] SoS SE team to guide SoS SE efforts Directed [Maier, 2008] SoS SE efforts managed through formal mechanisms Net-Centric SoS Net-Centric Connectivit y Laboratory System Imaging Management System Pharmacy System Patient Management System Telemetry System Health Care Network Internet Future Combat Systems

5. 5 Example: SoSE (Directed)

6. 6 Example: SoSE (Acknowledged)

7. 7 ● ● ● ValuationExploration Architecting Develop Operation Valuation Exploration Architecting Develop Operation ValuationExploration Architecting Develop Operation Develop Operation System A System B System C System x SoS-Level ValuationExploration Architecting Develop FCR 1 DCR 1 Operation OCR 1 Rebaseline/ Adjustment FCR 1 OCR 2                                              OCR x1 FCR B DCR B OCR B1 FCR A DCR A FCR C DCR C OCR C1 OCR x2 OCR x3 OCR x4 OCR x5 OCR C2 OCR B2 OCR A1 Example: SoSE (Collaborative) X

8. 8 Translating capability objectives Translating capability objectives Translating capability objectives Addressing new requirements & options Addressing new requirements & options Addressing requirements & solution options Understanding systems & relationships (includes plans) Understanding systems & relationships (includes plans) Understanding systems & relationships External Environment Developing, evolving and maintaining SoS design/arch Developing, evolving and maintaining SoS design/arch Developing & evolving SoS architecture Assessing (actual) performance to capability objectives Assessing (actual) performance to capability objectives Assessing performance to capability objectives Orchestrating upgrades to SoS Orchestrating upgrades to SoS Orchestrating upgrades to SoS Monitoring & assessing changes Monitoring & assessing changes Monitoring & assessing changes Traditional SE and SoSE Activities Traditional SE (Defense Acquisition Guide [DoD, 2006] View) SoSE (SoS SE Guidebook View Based on Interviews and Analysis of 18 DoD SoSs in Various Stages)

9. 9 SoSE Compared to Traditional SE Activities: Key Challenges for SoSE People Challenges Business model and incentives to encourage working together at the SoS level Removing multiple decision making layers Requiring accountability at the enterprise level Process Challenges Determining what to manage and what to leave to the CSs Doing the necessary tradeoffs at the SoS level Human-system integration Technical Challenges Commonality of data, architecture, and business strategies at the SoS level Evolution management Maturity of technology

10. 10 COCOMO Cost Model Suite Overview* * Barry Boehm, Ricardo Valerdi, Jo Ann Lane, and Winsor Brown, “COCOMO Suite Methodology and Evolution”, CrossTalk, April 2005.

11. 11 SoSE Cost Model Background Early attempts to develop a “directed” SoS cost model were not successful Seldom start with greenfield development Not enough directed SoSs to calibrate a cost model SoS cost model needs Cost estimation associated with a new capability Cost tradeoffs to support decisions Example: Migrate collaborative SoS to an acknowledged SoS

12. 12 Size Drivers Cost Drivers SE Effort Calibration Number of requirements Number of interfaces Number of algorithms Number of operational scenarios 8 Application factors 6 Team factors Schedule driver COSYSMO Current Systems Engineering Cost Model * Capabilities Prediction Accuracy Academic version Single system cost model calibrated with data from multiple organizations: PRED(30)=75% Local calibration versions Anecdotal evidence: PRED(30)=85% * COSYSMO [Valerdi, 2005] General Form of academicCOSYSMO Equation Effort (person months) = [38.55 * EM * (size) 1.06 ] / 152 where 38.55 and 1.06 are the academicCOSYSMO calibration factors EM is computed from cost drivers

13. 13 COSYSMO Limitations for Complex Systems and SoSs Limitations for complex systems and SoS Single set of cost drivers for system does not support definition of multiple components with different characterizations Additional limitations for SoS Does not address constituent system oversight effort at SoS level Does not address constituent system engineering contributions to SoSE

14. 14 Modifications for Complex Systems Additional modifications The academic calibration constants can be adjusted to provide more accurate estimates by performing a local calibration Reuse factors [Wang et al., 2008] can be added for each component Effort (person months) = 38.55*∑EM i *(part i size/total size)*(total size) 1.06 /152 where 38.55 and 1.06 are the academicCOSYSMO calibration factors i ranges from 1 to the number of components within the complex system

15. 15 Extensions for SoSs

16. 16 Systems Engineering Requirements Categories for SoSE in an Acknowledged SoS Requirements related to SoS capabilities Initially engineered at SoS level by SoSE team with support from constituent system engineers for those systems impacted by the SoS capability, then allocated to constituent systems for further SE Non-SoS requirements related to constituent system stakeholder needs Must be monitored by SoSE team to identify changes that might adversely impact SoS Represents on-going volatility at the constituent system level that is occurring in parallel with SoS capability changes

17. 17 Key SoSE Characteristics Used to Develop COSYSMO SoS Extensions SoSE sub-model SoSE oversight of constituents can be characterized by using the appropriate COSYSMO reuse factor Other non-traditional SE activities performed by SoSE team can be handled through COSYSMO cost factors Two types of requirements (SoS and constituent system non-SoS requirements) modeled together using different effort multipliers for each set* Constituent system sub-model Each constituent system within the SoS is independently owned and managed Constituent system SE effort to support the SoSE team can be characterized by including extra design effort for the SoS requirements Two types of requirements (SoS and constituent system non-SoS requirements) modeled together using different effort multipliers for each set or component* * Use of multiple effort multipliers allows one to model the diseconomy of scale as the SoS becomes larger through the integration of components with different characteristics....

18. 18 SoS Effort Calculations SoSE Effort SoSE Effort = 38.55*[((SoS CR /SoS Treq )*(SoS Treq ) 1.06 *EM SoS-CR )+ ((SoS MR /SoS Treq )*(SoS Treq ) 1.06 * EM SoS-MR )/152] Where: Total SoSE requirements = SoS Capability Requirements + SoS “Monitored” Requirements SoS “monitored” reqs = [∑SE non-SoS requirements being addressed current upgrade cycles for all SoS constituent systems] * “Oversight Factor” “Oversight Factor” = 5%, 10%, 15% (these values are based on the COSYSMO reuse work and expert judgment from various CSSE affiliates and the SoS SE Guidebook team) SoS capability effort Oversight of CSs

19. 19 SoSE Effort Multiplier Example 2.50

20. 20 Example Effort Multiplier for SoSE Monitoring of CS Requirements 0.47

21. 21 SoS Effort Calculations (continued) Single Constituent System Effort Total single system reqs w-SoSE = SoS requirements allocated to system + SE reqs in upgrade cycle Single system SE Effort in an Acknowledged SoS = 38.55*[1.15*( (SoS CSalloc / CS TreqSoSE )*( CS TreqSoSE ) 1.06 * EM CS-CRwSOSE ) + (CS nonSoS / CS TreqSoSE )*( CS TreqSoSE ) 1.06 * EM CSnonSOS ] /152 Computed for each constituent system in the SoS... Approach is recursive: Can also model each constituent system as a complex system or SoS... SoS capability effort Constituent system upgrade effort CS “tax” to support SoSE team

22. 22 Total SoS SE Effort SoS effort includes SoS capability effort and Constituent system non-SoS effort associated with single system enhancements To compute SoS capability effort, subtract out the total constituent system non-SoS effort Approach incorporates the diseconomy of scale at the constituent system level associated with the additional SoS capability requirements SoS effort = SoSE effort + ∑ constituent system effort i where i ranges from 1 to the number of constituent systems within the SoS

23. Using Alternative Size Drivers

24. SoSE Cost Model: Alternative Size Drivers 24 Size Drivers Cost Factors Estimated Engineering Effort Calibration Number of System Requirements Number of System Interfaces Number of Algorithms Number of Operational Scenarios People characteristics Process characteristics Product characteristics COSYSMO

25. SoSE Capability Effort Calculation Using Alternative Size Drivers Constituent system (CS) effort depends upon SoS alternative selected CS i effort depends upon types of changes required for CS i New interface(s)/interface change(s) Internal algorithm change(s)/data conversions Size driver options Number of requirements Number of algorithms Number of interfaces Number of operational scenarios Each size driver characterized with respect to complexity All size drivers converted to equivalent # of nominal reqs 25 SoS effort = SoSE effort + ∑ constituent system effort i

26. SoSE Estimation Steps for New Capability 1.Understand/review current CS capabilities 2.Identify new capability alternatives 3.For each alternative, identify CSs that contribute to each alternative For each contributing CS, changes needed to support alternative New interfaces/interface change(s) Data element/algorithm change(s) Capability size count(s) and associated complexity of each 4.Conduct alternative tradeoffs and finalize cost estimate for selected alternative 5.Identify CS changes required for desired architecture enhancements 6.Calculate COSYSMO effort multipliers at SoS and CS levels 7.Calculate SoSE effort for alternative 26

27. SoSE Estimation Steps for New Capability: Focus of Discussion 1.Understand/review current CS capabilities 2.Identify new capability alternatives 3.For each alternative, identify CSs that contribute to each alternative For each contributing CS, changes needed to support alternative New interfaces/interface change(s) Data element/algorithm change(s) Capability size count(s) and associated complexity of each 4.Conduct alternative tradeoffs and finalize cost estimate for selected alternative 5.Identify CS changes required for desired architecture enhancements 6.Calculate COSYSMO effort multipliers at SoS and CS levels 7.Calculate SoSE effort for alternative 27

28. Primary SoS Core Elements Determining SoSE Size Drivers 28 SoS : A set or arrangement of systems that results when independent and useful systems are integrated into a larger system that delivers unique capabilities

29. SysML Models for Characterizing SoS/SoS Capabilities Use cases Characterize both CS and SoS capabilities from the different user perspectives Sequence diagrams Characterize and analyze the operational flow for an SoS capability Object blocks Characterize each SoS CS and its capabilities Interface classes Describe each CS interface Input/output entity classes Express the associated data attributes of each data item transferred over that interface May include units, coordinate system, reference frame, source algorithm, etc. 29

30. Example SoS: Regional Area Crisis Response SoS (RACRS) 30 Command Control Center (CCC) Context Diagram

31. Mission Scenarios: Use Cases and Sequence Diagrams

32. CCC Interface Class and Evacuate Area I/O Entities by Actor

33. Using SoSE Cost Model to Evaluate Alternatives: Collaborative vs. Acknowledged SoS

34. 34 Systems Engineering Requirements Categories Requirements related to SoS capabilities a) Acknowledged SoS: Initially engineered at SoS level by SoSE team with support from CS engineers for those systems impacted by the SoS capability, then allocated to CSs for further SE b) Collaborative SoS: Not engineered at the SoS level, but must be engineered fully at the CS level through collaborative efforts with other CS engineers Non-SoS requirements related to CS stakeholder needs Must be monitored by SoSE team to identify changes that might adversely impact SoS Represents on-going volatility at the CS level that is occurring in parallel with SoS capability changes

35. 35 System Capability Effort for a “collaborative” SoS Effort using an “acknowledged” SoSE team Equivalent set of “sea-level” requirements Conversion to COSYSMO size units Calculations based on SoS characteristics/size and capability implementation approach using COSYSMO algorithm Overview of SoSE Comparative Model

36. 36 Summary of Comparative Model Effort Multipliers EMValue*Modified Cost Parameters SoSE effort2.50 Requirements understanding (low) Level of service requirements (high) # of recursive levels in the design (high) Multisite coordination (low) SoSE monitoring of CS Reqs0.47 Technology risk (very low) Documentation (very low) Personnel/team capability (high) Capability SE at CS level with SoSE Support1.06 Architecture understanding (high) Level of service requirements (high) Capability SE at CS level without SoSE Support 1.79 Requirements understanding (low) Level of service requirements (high) SE of non-SoS reqs0.72Architecture understanding (high) # of recursive levels in the design (low) * Default value: 1.0 (all cost parameters set to nominal)

37. 37 SoSE Effort Calculations SoSE Effort = 38.55*[((SoS CR /SoS Treq )*(SoS Treq ) 1.06 *EM SoS-CR )+ ((SoS MR /SoS Treq )*(SoS Treq ) 1.06 * EM SoS-MR )/152] Where: Total SoSE requirements = SoS Capability Requirements + SoS “Monitored” Requirements SoS “monitored” reqs = [∑SE non-SoS requirements being addressed current upgrade cycles for all SoS constituent systems] * “Oversight Factor” “Oversight Factor” = 5%, 10%, 15% (these values are based on the COSYSMO reuse work and expert judgment from various CSSE affiliates and the SoS SE Guidebook team) SoS capability effort Oversight of CSs

38. 38 Single CS Effort Calculation (Acknowledged) Total single system reqs w-SoSE = SoS requirements allocated to system + SE reqs in upgrade cycle Effort = 38.55*[1.15*( (SoS CSalloc / CS TreqSoSE )*( CS TreqSoSE ) 1.06 * EM CS-CRwSOSE ) + (CS nonSoS / CS TreqSoSE )*( CS TreqSoSE ) 1.06 * EM CSnonSOS ] /152 SoS capability effort CS upgrade effort CS “tax” to support SoSE team

39. 39 Total SoS SE Effort (Acknowledged) SoS effort includes SoS capability effort and CS non-SoS effort associated with single system enhancements Approach incorporates the diseconomy of scale at the CS level associated with the additional SoS capability requirements SoS effort = SoSE effort + ∑ CS effort i where i ranges from 1 to the number of CSs within the SoS

40. 40 Single CS Effort Calculation (Collaborative) Total single system reqs w-SoSE = SoS requirements + SE reqs in upgrade cycle Effort = 38.55*[( (SoS CR / CS TreqwoSoSE )*( CS TreqwoSoSE ) 1.06 * EM CS-CRnSoSE ) + (CS nonSoS / CS TreqwoSoSE )*( CS TreqwoSoSE ) 1.06 * EM CSnonSOS ] /152 SoS capability effort w/o SoSE support CS upgrade effort

41. 41 Range of SoS Complexity Factor Values SoSE Model Parameter DescriptionRange of Values SoS Size Number of constituent systems within the SoS 2-200 SoS Capability Size Number of equivalent nominal requirements as defined by COSYSMO 1-1000 Constituent System Volatility Number of non-SoS changes being implemented in each constituent system in parallel with SoS capability changes 0-2000 Scope of SoS Capability Number of constituent systems that must be changed to support capability One to SoS Size (total number of constituents systems within the SoS) SoSE Oversight FactorOversight adjustment factor to capture SoSE effort associated with monitoring constituent system non-SoS changes 5%, 10%, and 15%

42. 42 Model Results Scenario 1 (SoS Size Varies) Scenario 2 (SoS Size Varies) Each graph shows for each OSF value: (SoSE effort + ∑Acknowledged CS i effort*) – (∑Collaborative CS i effort *) * CS effort is the sum of the SoS capability effort and the non-SoS requirements effort 0 Cost savings with SoSE Team Extra cost of SoSE Team Person Months

43. 43 Model Results (continued) Scenario 3 (SoS Size Varies) Scenario 4 (SoS Size Varies) Scenario 5 (SoS Size Varies) Scenario 6 (SoS Size Varies)

44. 44 Model Results (continued) Scenario 7-a (SoS Size = 10) Scenario 7-b (SoS Size = 100) Scenario 8-a (SoS Size = 10) Scenario 8-b (SoS Size = 100)

45. 45 Model Results (continued) Scenario 9 (SoS Size = 10) Scenario 10 (SoS Size = 5) Scenario 11 (SoS Size = 5) Scenario 12 (SoS Size = 5)

46. 46 Summary Presented approach for extending COSYSMO cost model to estimate systems engineering effort for Complex systems Systems of systems Additional accuracy improvements can be provided through Local calibrations of the COSYSMO constants Incorporation of reuse factors [Wang, et al., 2008] Examples provided for Using alternative size drivers Showing how cost model can be used to evaluate alternatives

47. 47 Acknowledgements The author would like to acknowledge The pioneering work done by Dr. Ricardo Valerdi in the development of the initial COSYSMO cost model upon which this research effort is based The research support received from Stevens Institute of Technology and the International Council on Systems Engineering (INCOSE) Foundation through the 2007 INCOSE Foundation/Stevens Doctoral Award

48. 48 References 1.Dahmann, J. and K. Baldwin. 2008. Understanding the current state of US defense systems of systems and the implications for systems engineering. Proceedings of the IEEE Systems Conference, April 7-10, in Montreal, Canada. 2.Department of Defense. 2008. Systems engineering guide for system of systems, version 1.0. 3.Maier, M. 1998. Architecting principles for systems-of-systems. Systems Engineering 1, no. 4: 267-284. 4.Valerdi, R. 2005. Constructive systems engineering cost model. PhD. Dissertation, University of Southern California. 5.Valerdi, R. and M. Wheaton. 2005. ANSI/EIA 632 as a standardized WBS for COSYSMO, AIAA-2005-7373, Proceedings of the AIAA 5th Aviation, Technology, Integration, and Operations Conference, Arlington, Virginia. 6.Wang, G., R. Valerdi, A. Ankrum, C. Millar, and G. Roedler. 2008. COSYSMO reuse extension, Proceedings of the 18th Annual International Symposium of INCOSE, The Netherlands.