When Do You Need Systems of Systems Engineering: A Quantitative Analysis Jo Ann Lane 17 March 2009 University of Southern California Center for Systems and Software Engineering

March 2009 USC CSSE Annual Research Review2 Overview Key definitions Scope of research Methodology Model implementation Results of research Conclusions and future work

March 2009 USC CSSE Annual Research Review3 What is a “System of Systems”? Very large systems developed by creating a framework or architecture to integrate constituent systems SoS constituent systems independently developed and managed –New or existing systems in various stages of development/evolution –May include a significant number of COTS products –Have their own purpose –Can dynamically come and go from SoS SoS exhibits emergent behavior not otherwise achievable by component systems Typical domains –Business: Enterprise-wide and cross-enterprise integration to support core business enterprise operations across functional and geographical areas –Military: Dynamic communications infrastructure to support operations in a constantly changing, sometimes adversarial, environment Based on Mark Maier’s SoS definition [Maier, 1998]

March 2009 USC CSSE Annual Research Review4 Types of SoS Virtual [Maier, 1998] –Lacks a central management authority and a clear SoS purpose –Often ad hoc and may use a service-oriented architecture where the constituent systems are not necessarily known Collaborative [Maier, 1998] –Constituent system engineering teams work together more or less voluntarily to fulfill agreed upon central purposes –No SoSE team to guide or manage activities of constituent systems Acknowledged [Dahmann, 2008] –Have recognized objectives, a designated manager, and resources at the SoS level (SoSE team) –Constituent systems maintain their independent ownership, objectives, funding, and development approaches Directed [Maier, 2008] –SoS centrally managed by a government, corporate, or Lead System Integrator (LSI) and built to fulfill specific purposes –Constituent systems maintain ability to operate independently, but evolution subordinated to centrally managed purpose This research focused on identifying the “home-ground” for these two types of SoSs...

March 2009 USC CSSE Annual Research Review5 Scope of Research Research question –When is it cost effective to establish and use a system of systems engineering (SoSE) team to oversee and guide the evolution of a system of systems (SoS)? Hypothesis –There exists a threshold where it is more cost effective to manage and engineer capability changes to an SoS using an SoSE team and this threshold can be determined by modeling the SoS system complexity and desired capability interdependency characteristics. Focus is on software-intensive SoSs owned by the United States Department of Defense (DoD)...

March 2009 USC CSSE Annual Research Review6 Statement of Topic and Contribution (continued) Research contribution –Provides guidance to DoD leadership with respect to the management of sets of inter-related systems that are functioning as a system of systems. –Guidance also applies to SoSs in other domains that are managed as collaborative or acknowledged SoSs –Model for management and engineering guidance also provides A method for conducting trade-off analyses for different approaches for implementing a given SoS capability for a given SoS A model that can evolve into an SoSE cost model through calibration for a given SoS or SoS domain A cost model that can better model complex systems

March 2009 USC CSSE Annual Research Review7 Methodology Using COSYSMO, developed a process model that can compare the SoS management strategies as SoS characteristics are varied –SoS size (number of constituent systems) –Size of SoS capability (number of equivalent nominal requirements) –Scope of SoS capability (number of constituent systems affected by SoS capability) –Constituent system volatility (level of constituent system change being engineered at the same time as SoS capability) Process model based on data from –18 large-scale DoD SoS programs –16 DoD systems that participate as constituent systems in one or more SoSs Analyze model outputs to determine under what conditions an SoSE team is cost effective

March 2009 USC CSSE Annual Research Review8 SoSE Process Model Overview Purpose –Estimate and compare the effort required to implement an SoS capability using two different management approaches Collaborative (no SoSE team) Acknowledged (SoSE with limited authority/control) Assumptions and constraints –All constituent systems currently exist and have their own evolutionary paths based on system-level stakeholder needs/desires –Model assumes SoSE and traditional SE teams are using relatively mature processes –SoS capabilities are software-intensive –No SoS capability/requirements volatility –SoS internal volatility represented by constituent system volatility –No accommodation of schedule factors or the asynchronous nature of SoS constituent system upgrades –Management of SoS internal interfaces reduces complexity for systems

March 2009 USC CSSE Annual Research Review9 Systems Engineering Requirements Categories Requirements related to SoS capabilities a)Acknowledged SoS: Initially engineered at SoS level by SoSE team with support from constituent system engineers for those systems impacted by the SoS capability, then allocated to constituent systems for further SE b)Collaborative SoS: Not engineered at the SoS level, but must be engineered fully at the constituent system level through collaborative efforts with other constituent system engineers Non-SoS requirements related to constituent system stakeholder needs –Must be monitored by SoSE team to identify changes that might adversely impact SoS –Represents on-going volatility at the constituent system level that is occurring in parallel with SoS capability changes

March 2009 USC CSSE Annual Research Review10 SoSE Model Structure Focus is on software- intensive SoSs owned by the US DoD, the number and volatility of constituent systems within an SoS, and the complexity of typical capability enhancements to the SoS...

March 2009 USC CSSE Annual Research Review11 System Capability Effort for a “collaborative” SoS Effort using an “acknowledged” SoSE team Equivalent set of “sea-level” requirements Conversion to COSYSMO size units Calculations based on SoS characteristics/size and capability implementation approach using COSYSMO algorithm Overview of SoSE SDM Flow

March 2009 USC CSSE Annual Research Review12 Model Parameters by SDM Construct Stocks –Inputs SoS Equivalent Requirements –Outputs SoSE Effort SoS Upgrade Effort with SoSE SoS Upgrade Effort without SoSE Flows –Capability Rate –SoSE Effort Rate –SE Effort Rate with SoSE –SE Effort Rate without SoSE Converter Parameters –COSYSMO effort multipliers COSYSMO SoSE EM COSYSMO SE EM with SoSE COSYSMO SE EM without SoSE COSYSMO SE EM –SoS complexity factors Number of systems in SoS Number of systems affected by capability Average system rate of change General Form of COSYSMO Equation Effort (person months) = [38.55 * EM * (size) 1.06 ] / 152

March 2009 USC CSSE Annual Research Review13 SoSE Effort Multiplier 2.50

March 2009 USC CSSE Annual Research Review14 Effort Multiplier for SoSE Monitoring of Constituent System Requirements 0.47

March 2009 USC CSSE Annual Research Review15 SE Effort Multiplier for SoS Requirements with SoSE Support 1.06

March 2009 USC CSSE Annual Research Review16 SE Effort Multiplier SoS Requirements without SoSE Support 1.79

March 2009 USC CSSE Annual Research Review17 SE Effort Multiplier for System-Specific (Non-SoS) Requirements 0.72

March 2009 USC CSSE Annual Research Review18 Effort Calculations SoSE Effort SoSE Effort = 38.55*[((SoS CR /SoS Treq )*(SoS Treq ) 1.06 *EM SoS-CR )+ ((SoS MR /SoS Treq )*(SoS Treq ) 1.06 * EM SoS-MR )/152] Where: Total SoSE requirements = SoS Capability Requirements + SoS “Monitored” Requirements SoS “monitored” reqs = [∑SE non-SoS requirements being addressed current upgrade cycles for all SoS constituent systems] * “Oversight Factor” “Oversight Factor” = 5%, 10%, 15% (these values are based on expert judgment from various CSSE affiliates and the SoS SE Guidebook team) Based on COCOMO II approach for combining components with different EMs (SoS changes and Constituent System oversight)

March 2009 USC CSSE Annual Research Review19 Effort Calculations (continued) Single System Effort with Support from SoSE Team Total single system reqs w-SoSE = SoS requirements allocated to system + SE reqs in upgrade cycle Single system SE Effort with SoSE Team = 38.55*[1.15*( (SoS CSalloc / CS TreqSoSE )*( CS TreqSoSE )1.06* EM CS-CRwSOSE ) + (CS nonSoS / CS TreqSoSE )*( CS TreqSoSE) 1.06* EM CSnonSOS ] /152 Based on COCOMO II approach for combining components with different EMs plus including a 15% “tax” to support SoSE team in their engineering effort for the SoSE requirements. 15% represents half of the system design effort in the EIA 632 tasks.

March 2009 USC CSSE Annual Research Review20 Effort Calculations (continued) Single System Effort with No SoSE Team Support Total single system reqs wo-SoSE = SoSE capability reqs + SE non-SoS requirements Single system SE Effort without SoSE Team = 38.55*[(( SoS CR / CS TreqwoSoSE )*( CS TreqwoSoSE )1.06* EM CS-CRnSOSE ) + ((CS nonSoS / CS TreqwoSoSE )*( CS TreqwoSoSE )1.06* EM CSnonSOS )] /152 Based on COCOMO II approach for combining components with different EMs (SoS changes and non-SoS changes)

March 2009 USC CSSE Annual Research Review21 Range of SoS Complexity Factor Values SoSE Model Parameter DescriptionRange of Values SoS SizeNumber of constituent systems within the SoS SoS Capability SizeNumber of equivalent nominal requirements as defined by COSYSMO Constituent System Volatility Number of non-SoS changes being implemented in each constituent system in parallel with SoS capability changes Scope of SoS Capability Number of constituent systems that must be changed to support capability One to SoS Size (total number of constituents systems within the SoS) SoSE Oversight Factor Oversight adjustment factor to capture SoSE effort associated with monitoring constituent system non-SoS changes 5%, 10%, and 15%

March 2009 USC CSSE Annual Research Review22 Results of Research Scenario 1 (SoS Size Varies) Scenario 2 (SoS Size Varies) Scenario 3 (SoS Size Varies) Scenario 4 (SoS Size Varies)

March 2009 USC CSSE Annual Research Review23 Results of Research (continued) Scenario 5 (SoS Size Varies) Scenario 6 (SoS Size Varies) Scenario 7-a (SoS Size = 10) Scenario 7-b (SoS Size = 100)

March 2009 USC CSSE Annual Research Review24 Results of Research (continued) Scenario 8-a (SoS Size = 10) Scenario 8-b (SoS Size = 100) Scenario 9 (SoS Size = 10) Scenario 10 (SoS Size = 5)

March 2009 USC CSSE Annual Research Review25 Results of Research (continued) Scenario 11 (SoS Size = 5) Scenario 12 (SoS Size = 5)

March 2009 USC CSSE Annual Research Review26 There exists a threshold where it is more cost effective to manage and engineer changes to an SoS using an SoSE team and this threshold can be determined by modeling the SoS’ interdependency and complexity characteristics. When is it cost effective to create and empower an SoSE team to oversee and guide the evolution of an SoS? SoSE Model Model parameters: SoS size Scope/size of SoS change CS volatility SoSE oversight Conclusions

March 2009 USC CSSE Annual Research Review27 Conclusions (continued) SoSE team is cost effective when –SoS contains more than a “few” systems –SoS capability changes typically affect a “significant percentage” of constituent systems –SoS capability requirements are a “significant percentage” of the total requirements being addressed by constituent systems in an upgrade cycle –SoS oversight activities and the rate of capability modifications/changes being implemented are sufficient to keep an SoSE team engaged (i.e., little-to-no slack time) SoSE team is NOT cost effective when –The number of systems in an SoS is “small” –The constituent system volatility is high and the SoS changes are small

March 2009 USC CSSE Annual Research Review28 Conclusions (continued) The “oversight factor” (the amount of effort spent by the SoSE team to monitor non-SoS changes in the constituent systems) is a key factor in determining the cost effectiveness of the SoSE team –More work is needed to determine a more accurate “oversight factor” –This factor may be variable across multiple SoSs There may be reasons other than cost to engage an SoSE team –Importance of SoS –Critical SoS performance requirements requiring extensive analysis at the SoS level

March 2009 USC CSSE Annual Research Review29 Future Work Expand SoSE model to –Include schedule factors to allow trade-offs between “faster” and “cheaper” –Include quality factors based on complexities and the resulting rework due to inadequate SoS engineering –Allow users to specify specific constituent system configurations to allow capability alternative trade-offs Investigate the factors in going from an Acknowledged SoS to a Directed SoS

March 2009 USC CSSE Annual Research Review30 Backup Charts

March 2009 USC CSSE Annual Research Review31 Translating capability objectives Translating capability objectives Translating capability objectives Addressing new requirements & options Addressing new requirements & options Addressing requirements & solution options Understanding systems & relationships (includes plans) Understanding systems & relationships (includes plans) Understanding systems & relationships External Environment Developing, evolving and maintaining SoS design/arch Developing, evolving and maintaining SoS design/arch Developing & evolving SoS architecture Assessing (actual) performance to capability objectives Assessing (actual) performance to capability objectives Assessing performance to capability objectives Orchestrating upgrades to SoS Orchestrating upgrades to SoS Orchestrating upgrades to SoS Monitoring & assessing changes Monitoring & assessing changes Monitoring & assessing changes Traditional SE and SoSE Activities Traditional SE (Defense Acquisition Guide [DoD, 2006] View) SoSE (SoS SE Guidebook View Based on Interviews and Analysis of 18 DoD SoSs in Various Stages)

March 2009 USC CSSE Annual Research Review32 Key COSOSIMO Research Findings Limitations of COSYSMO for “Directed” SoSE effort estimation –Missing cost factors Cost/schedule compatibility of proposed SE approach Level of overall risk resolution Number of constituent systems and associated organizations Constituent system maturity and stability Constituent system readiness –Need to adjust for SoSE oversight of constituent system SE –Need ability to assign different EMs to various parts of SE

March 2009 USC CSSE Annual Research Review33 Using System Dynamics Models to Explore Alternatives or Influences in the Development of Large Software-Intensive Systems System dynamics modeling tool: visual modeling tools that allow one to conceptualize, simulate and analyze models of dynamic systems and processes Consist of causal loops or stock and flow diagrams Models are executable, allowing use to explore behaviors of the model as variables representing process influences are changed Examples: –Hybrid/plan-driven ICM [Madachy et al., 2007] –Intergovernmental collaboration [Cresswell et al., 2002] –Inter-Organizational Baseline Alignment [Greer et al., 2005] –Requirements volatility [Ferreira, 2002] –Under-allocation of resources in early phases of a project [Black and Repenning, 2001] –Interactions between concurrently developed projects [Ford and Sterman, 2003]

March 2009 USC CSSE Annual Research Review34 Model Validity Rationale SoSE model description –Comparison model based on a modified version of the validated academic systems engineering cost model, COSYSMO –Modifications based upon key findings of the OSD SoSE case studies Validation goal: Show that the SoSE cost model is a valid method conducting sensitivity analyses for two different SoS management strategies –Collaborative –Acknowledged Not part of the validation goal: The estimation of actual effort associated with a specific SoS or a given set of SoSs –The calibration/validation of the SoSE model for this purpose is left for future work

March 2009 USC CSSE Annual Research Review35 Model Validity Rationale (continued) Validity argument –COCOMO II and Academic COSYSMO are multiple regression models that have been calibrated and validated with actual data from primarily DoD programs –Academic COSYSMO calibration data contains 3 SoS data points –Most other COSYSMO calibration data points interface to other systems, which implies that they are part of one or more SoSs –SoSE model was developed using Academic COSYSMO that includes ability to distribute effort across SE phases Locally validated COSYSMO extension to adjust effort for reuse/oversight of evolving system components COCOMO II technique for using multiple effort multipliers to characterize components with different characteristics and complexities –SoSE model parameters Based on ranges of size drivers determined through case studies and surveys Uses nominal cost driver values unless reasons identified in SoSE or SE survey data to indicate otherwise Resulting in a relative comparison of the two management approaches

March 2009 USC CSSE Annual Research Review36 Model Validity Rationale (continued) Validity argument (continued) –The prediction accuracies (PRED factors) are COCOMO [Clark and Reifer, 2007] –PRED(30) = 75% (with no stratification of projects) –PRED(30) = 80% (with stratification of projects) COSYSMO [Valerdi, 2005] –PRED(30) = 75% (with stratification of projects) –PRED(30) = 85% (anecdotal evidence from local calibrations) –The OSD SoSE cases studies show that SoS systems engineers perform the same types of activities as addressed by the SE cost model, COSYSMO – The OSD SoSE case studies identify differences between SoSE and SE for a single system and most of these differences are with respect to parameters in the SE cost model, COSYSMO –There exists a local (single organization) calibrated and validated method within COSYSMO to estimate effort for oversight of related/interfacing systems or reusable components [Wang et al, 2008]

March 2009 USC CSSE Annual Research Review37 References Ackoff, R. (1971); “Towards a System of Systems Concepts”, Management Science, Vol 17, No. 11, Theory Series, pp ANSI/EIA (1999). ANSI/EIA Processes for Engineering a System. Berry, B. (1964); “Cities as Systems within Systems of Cities”, The Regional Science Association Papers, Volume 13.. Black, L. and Repenning, N. (2001); “Why Firefighting is Never Enough: Preserving High-Quality Product Development”, System Dynamics Review, Vol. 17, No: 1, pp Blanchard, B. and Fabrycky, W. (1998). Systems Engineering and Analysis, Prentice Hall. Boehm, B., Abts, C., Brown, A. W., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer, D. J. and Steece, B. (2000). Software Cost Estimation With COCOMO II, Prentice Hall. Boehm, B. and Lane J. (2006) "21st Century Processes for Acquiring 21st Century Software-Intensive Systems of Systems." CrossTalk - The Journal of Defense Software Engineering, Vol. 19, No. 5, pp.4-9. Boehm, B., Valerdi, R., Lane, J., Brown, A., (2005) “COCOMO Suite Methodology and Evolution,” CrossTalk - The Journal of Defense Software Engineering, Vol. 18, No. 4, pp , April Cocks, D. (2006); “How Should We Use the Term “System of Systems” and Why Should We Care?”, Proceedings of the 16th Annual INCOSE International Symposium. Cresswell, A. et al. (2002); "Modeling Intergovernmental Collaboration: A System Dynamics Approach", Proceedings of the 35th Annual Hawaii International Conference on System Sciences. Dahmann, J. and Baldwin. K. (2008); “Understanding the Current State of US Defense Systems of Systems and the Implications for Systems Engineering”, Montreal, Canada: Proceedings of the IEEE Systems Conference, 7-10 April. Department of Defense (DoD) (2006); Defense Acquisition Guidebook, Version 1.6, accessed at on 2/2/2007. Department of Defense (DoD) (2008); Systems Engineering Guide for System of Systems, version 1.0. Dorner, D. (1996); The Logic of Failure, Metropolitan Books. Ferreira S. (2002); Measuring the Effects of Requirements Volatility on Software Development Projects. Ph.D. Dissertation, Arizona State University. Finley, J. (2006); “Keynote Address”, Proceedings of the 2nd Annual System of Systems Engineering Conference Ford D. and Sterman J. (2003); "Iteration Management for Reduced Cycle Time in Concurrent Development Projects", Concurrent Engineering Research and Application (CERA) Journal. Friedman, T. (2005), The World is Flat: A Brief History of the Twenty-First Century, Farrar, Straus and Giroux, New York. Greer, D., Black, L., Adams, R. (2005), "Improving Inter-Organizational Baseline Alignment in Large Space System Development Programs", Proceedings of IEEE Aerospace Conference. Highsmith, J. (2000); Adaptive Software Development: A Collaborative Approach to Managing Complex Systems, Dorset House Publishing.

March 2009 USC CSSE Annual Research Review38 References (continued) INCOSE (2006); Systems Engineering Handbook, Version 3, INCOSE-TP isee Systems (2007), "iThink", accessed on 2/10/2007. ISO/IEC (2002). ISO/IEC 15288:2002(E) Systems Engineering - System Life Cycle Processes. Ferreira S. (2002); Measuring the Effects of Requirements Volatility on Software Development Projects. Ph.D. Dissertation, Arizona State University. Finley, J. (2006); “Keynote Address”, Proceedings of the 2nd Annual System of Systems Engineering Conference Ford D. and Sterman J. (2003); "Iteration Management for Reduced Cycle Time in Concurrent Development Projects", Concurrent Engineering Research and Application (CERA) Journal. Friedman, T. (2005), The World is Flat: A Brief History of the Twenty-First Century, Farrar, Straus and Giroux, New York. Greer, D., Black, L., Adams, R. (2005), "Improving Inter-Organizational Baseline Alignment in Large Space System Development Programs", Proceedings of IEEE Aerospace Conference. Highsmith, J. (2000); Adaptive Software Development: A Collaborative Approach to Managing Complex Systems, Dorset House Publishing. INCOSE (2006); Systems Engineering Handbook, Version 3, INCOSE-TP isee Systems (2007), "iThink", accessed on 2/10/2007. ISO/IEC (2002). ISO/IEC 15288:2002(E) Systems Engineering - System Life Cycle Processes. Kreitman, K.(1996), "From 'The Magic Gig' to Reliable Organizations: A New Paradigm for the Control of Complex Systems", Symposium on Complex Systems Engineering, Symposium_on_Complex_Systems_Engineering, accessed on 1/11/2007. Krygiel, A. (1999); Behind the Wizard’s Curtain; CCRP Publication Series, July, 1999, p. 33 Lane, J. and Boehm, B. (2007); Modern Tools to Support DoD Software Intensive System of Systems Cost Estimation: A DACS State-of-the-Art Report, Data and Analysis Center for Software, DACS Report Number Lane, J. and Valerdi, R., (2007); "Synthesizing System-of-Systems Concepts for Use in Cost Estimation", Systems Engineering, Vol. 10, No. 4. Lu, S. (2003); Engineering as Collaborative Negotiation: A New Paradigm for Collaborative Engineering, accessed on 2/14/2007. Madachy, R., B. Boehm, and J. Lane (2007); “Assessing Hybrid Incremental Processes for SISOS Development”, Software Process: Improvement and Practice, Vol. 12, Issue 5, pp Maier, M. (1998); “Architecting Principles for Systems-of-Systems”; Systems Engineering, Vol. 1, No. 4 (pp )

March 2009 USC CSSE Annual Research Review39 References (continued) NAVSTAR Global Positioning System Joint Program Office, accessed on 12/6/2006. Northrop, L., et al. (2006); Ultra-Large-Scale Systems: The Software Challenge of the Future, Software Engineering Institute. Pinney, B. (2001); Projects, Management, and Protean Times: Engineering Enterprise in the United States, , PhD Dissertation, Massachusetts Institute of Technology. Pressman, J. and Wildavsky, A. (1973); Implementation: How Great Expectations in Washington are Dashed in Oakland; Or, Why It’s Amazing that Federal Programs Work at All, This Being a Saga of the Economic Development Administration as Told by Two Sympathetic Observers Who Seek to Build Morals on a Foundation of Ruined Hopes, University of California Press. Rechtin, E. (1991); Systems Architecting: Creating & Building Complex Systems, Prentice Hall. SEI (2001), Capability Maturity Model Integration (CMMI), CMU/SEI-2002-TR-001. Sheard, S. (2006), "Foundations of Complexity Theory for Systems Engineering of Systems of Systems", Proceedings of the IEEE Conference on System of Systems Engineering. United States Air Force (USAF) Scientific Advisory Board (SAB) (2005); Report on System-of-Systems Engineering for Air Force Capability Development; Public Release SAB-TR Valerdi, R. (2005); Constructive Systems Engineering Cost Model. PhD. Dissertation, University of Southern California. Valerdi, R. and Wheaton, M. (2005); "ANSI/EIA 632 as a Standardized WBS for COSYSMO", AIAA , Proceedings of the AIAA 5th Aviation, Technology, Integration, and Operations Conference, Arlington, Virginia. Wang, G., Valerdi, R., Ankrum, A., Millar, C., and Roedler, G. (2008), "COSYSMO Reuse Extension", Proceedings of the 18th Annual International Symposium of INCOSE, The Netherlands.