1. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop SoS Pain Points & Implications for MBSE Dr. Judith Dahmann The MITRE Corporation Co-Chair INCOSE SoSWG January 2012 1

2. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Topics INCOSE SoS WG SoS ‘Pain Points’ Implementer’s View of SoS SE M&S Opportunities, Challenges and Exemplars Summary and Next Steps Objective is to share information and open a dialog 2

3. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop 1 Systems of Systems Working Group The SoS Charter is to SoS Chairs Alan Harding (co-chair) Judith Dahmann (co-chair) INCOSE Connect address: https://connect.incose.org/tb/soswg/ INCOSE Web page: TBD Published Products New WG Draft SoS Pain Points Paper 1.BKCASE: Contribute to development and support of SoS Section 2.SoS Bibliography: Develop and share 3.SoS ‘Pain Points’: Identify key SoS challenges to guide WG activities 4.Related organization: identify and develop relationships (inside and outside INCOSE) 5.Webinars: Program to share SoS activities and experiences Promote application of SE to SoS through 1.Understand and share what we mean by SoS in our various contexts 2.Develop guidance and advice 3.Inform and up-skill practitioners 4.Exert influence on BKCASE, Standards, SE Vision 2025 etc. 5.Develop the practice of SE for SoS. 6.Work in partnership with other groups addressing aspects of SoS (e.g. INCOSE WGs, IEEE, NDIA). 7.Understanding and applying insights from relevant research Planned Work 3

4. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop SoS Pain Point Survey Purpose To collect information on major issues or 'pain points' in the area of Systems of Systems operation, management and systems engineering To support planning for activities of the WG Respondents 38 survey respondents 65 SoS ‘pain points’ reported Respondent location US (86%). UK (8%) Australia (6%) Respondent SoS experience Extensive (60%) Some (37%) Almost all (94%) are from defense sector Survey Logistics Developed during February and March 2012, with several drafts and pretests Released to the community in April with a cutoff of respondents in Mid- May. Administered over the internet using KWIK Surveys (www.kwiksurverys.com)www.kwiksurverys.com Questions & Analysis Asked respondents to identify and describe their priority SoS areas of concern: describe up to three 'pain points' including a short name, a description and an example Results were analyzed, a paper on the results was drafted and circulated for comment 4

5. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop SoS Pain Points Pain PointsQuestion Lack of SoS Authorities & Funding What are effective collaboration patterns in systems of systems? Leadership What are the roles and characteristics of effective SoS leadership? Constituent Systems What are effective approaches to integrating constituent systems into a SoS? Capabilities & Requirements How can SE address SoS capabilities and requirements? Autonomy, Interdependencies & Emergence How can SE provide methods and tools for addressing the complexities of SoS interdependencies and emergent behaviors? Testing, Validation & Learning How can SE approach the challenges of SoS testing, including incremental validation and continuous learning in SoS? SoS Principles What are the key SoS thinking principles, skills and supporting examples? Survey identified seven ‘pain points’ raising a set of SoS SE questions 5

6. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Implementers View of SoS SE More familiar and intuitive time-sequenced “wave” model representation Information is thus rendered in a form more readily usable by SoS SE practitioners in the field Representation that corresponds with incremental development approaches that are the norm for SoS capability evolution Concept of Wave Planning was developed by Dr. David Dombkins See “Complex Project Management” Booksurge Publishing, South Carolina: 2007. Presented at IEEE Systems Conference April 2011 [1] [1] “An Implementers View of Systems of Systems” Dahmann, Baldwin, Rebovich, Lane and Lowry External Environment 6

7. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Implementers View of SoS SE Pain points affect SE throughout the process 7 Pain Points Lack of SoS Authorities & Funding Leadership Constituent Systems Capabilities & Requirements Autonomy, Interdependencies & Emergence Testing, Validation & Learning SoS Principles

8. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Initiate SoS Decision has been made to establish an SoS SE organization –An entity is responsible for the SoS with SE support to the SoS –As an acknowledged SoS, the systems which constitute the SoS maintain operational and management independence At the initiation of an SoS, the information typically available includes initial or first order –Statement of top-level objectives for the SoS (SoS capability objectives) –Description of how systems in the SoS will be employed in an operational setting (SoS CONOPS) and –Programmatic and technical information about systems that affect SoS capability objectives (systems information) –Risks are identified when an SoS is launched and mitigation actions are tracked and updated throughout each cycle, along with new risks (Risks and Mitigations) Pain Points Lack of SoS Authorities & Funding Leadership 8

9. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Conduct SoS Analysis Provides analysis of the ‘as is’ and basis for SoS evolution Results provide basis for architecture development and planning for SoS updates 9 Pain Points Lack of SoS Authorities & Funding Leadership Constituent Systems Capabilities & Requirements Autonomy, Interdependencies & Emergence

10. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Conduct SoS Analysis M & S Opportunities Understanding CONOPs………… implications for functional baseline …… performance needs ……… current performance Wide variety of opportunities M&S could be used to explore many more use cases than would be otherwise possible Potential for uncovering “emergent behavior” and impacts of interdependencies that would not be predictable by other means Enable more complete exploration of SoS CONOPs and functionality SoS requirements space and more potential constituent systems that could contribute to the SoS 10

11. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Conduct SoS Analysis M & S Challenges Understanding CONOPs………… implications for functional baseline …… performance needs ……… current performance Wide variety of opportunities Impact of governance issues on cross system access to data and validation –Difficulties in clearly identifying SoS capabilities and successfully mapping these to constituent systems Available constituent system models/data may not lend themselves to reuse SoS complexity and uncertainty magnify challenges for model validity and predictability … and challenges 11 Pain Points Lack of SoS Authorities & Funding Leadership Constituent Systems Capabilities & Requirements Autonomy, Interdependencies & Emergence

12. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Conduct SoS Analysis Exemplars (1 of 2) Man-in-the-Loop Simulation Experimentation Assess CONOPS and systems interoperability Assess the vulnerability of current architecture 12

13. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Conduct SoS Analysis Exemplars (2 of 3) Sandia - SoS Analysis Toolset (SoSAT) 13 “provides logistics analysts with the ability to define operational and support environments of a SoS, ascertain measures of platform and SoS level performance effectiveness, and determine logistics support issues.” http://reliability.sandia.gov/tools_sosat.html

14. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Conduct SoS Analysis Exemplars (3 of 3) Mission Level Modeling (MLM) 14

15. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Develop/Evolve SoS Architecture Develops and evolves the persistent technical framework for addressing SoS evolution 15 Pain Points Lack of SoS Authorities & Funding Leadership Constituent Systems Capabilities & Requirements Autonomy, Interdependencies & Emergence

16. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Develop/Evolve SoS Architecture M & S Opportunities Explore many more architectural alternatives than would be otherwise possible Potential for uncovering “emergent behavior” of potential architectures enable more complete exploration of the: SoS architecture alternative space; to include its evolution over time More potential constituent systems that could contribute to the architecture More detailed exploration of potential constituent systems that could contribute to the SoS architecture, to include Ease or difficulty with which a constituent could evolve to the SoS architecture over time 2 nd order changes to the constituents caused by their participation in different SoS Representing E2E flow………,… architecture options Opportunities …….…… systems implementations 16

17. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Develop/Evolve SoS Architecture M & S Challenges Impact of governance issues on cross system access to architecture data and validation of cross system architecture alternatives Available constituent system architecture representations may not lend themselves to reuse SoS complexity and uncertainty magnify challenges for model/architecture validity and predictability Representing E2E flow………,… architecture options …….…… systems implementations Opportunities … and challenges 17 Pain Points Lack of SoS Authorities & Funding Leadership Autonomy, Interdependencies & Emergence Capabilities & Requirements Constituent Systems

18. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Develop/Evolve SoS Architecture Exemplars (1 of 2) SERC Research - SoS Interdependency Analysis Analysis of Architecture Alternatives 1 1: Systems Engineering Research Center, DoD University Affiliated Research Center for SE Han, Marais & DeLaurentis,2012 18 Research: SoS Analysts Workbench Assessing the Impact of Developmental Disruptions and Dependencies in Analysis of Alternatives of Systems of Systems Final Technical Report SERC- 2012-TR035, Dec 31,2012

19. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Conduct SoS Analysis Exemplars SERC Research – SoS Architecture Analysis Using Agent-Based Modeling Agent-based SoS Acquisition and Architecting Model Three dimensional representation of the surface of SoS evaluation for all values of Performance and Affordability 19 An Advanced Computational Approach to Systems of Systems Analysis and Architecting Using Agent- Based Behavioral Model Final Technical Report SERC- 2012-TR021, Sept 30,2012

20. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Plan SoS Update Evaluates the SoS priorities, options and backlogs to define the plan for the next SoS upgrade cycle 20 Pain Points Lack of SoS Authorities & Funding Leadership Constituent Systems Capabilities & Requirements Autonomy, Interdependencies & Emergence Testing, Validation & Learning

21. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Plan SoS Update M & S Opportunities Understand capability shortfalls in the broader SoS mission context Identify and explore many more alternative options Better scrubbing/exploration of alternative space Enable more complete analysis of impacts of change Could include simulation based experimentation, to identify areas of uncertainty or risk Analysis of needs in SoS context…. Assessment of trades……Analysis of interdependencies Opportunities 21

22. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Plan SoS Update M & S Challenges Depends upon a solid representation of architecture and SoS baseline including dependencies …with the all the challenges discussed above Understanding dependencies is critical here; hard to model something you don’t understand May depend on system level data which could be difficult to obtain or incompatible with the SoS context Analysis of needs in SoS context…. Assessment of trades……Analysis of interdependencies Opportunities … and challenges 22 Pain Points Lack of SoS Authorities & Funding Leadership Constituent Systems Capabilities & Requirements Autonomy, Interdependencies & Emergence Testing, Validation & Learning

23. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Plan SoS Update Exemplars 23

24. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Implement SoS SE Update Monitors implementations at the system level and plans and conducts SoS level testing, resulting in a new SoS product baseline 24 Pain Points Lack of SoS Authorities & Funding Leadership Constituent Systems Capabilities & Requirements Autonomy, Interdependencies & Emergence Testing, Validation & Learning

25. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Implement SoS SE Update M & S Opportunities Address risk when testing is impossible or too costly Address asynchronous development issues Employ system simulations as surrogates in testing Enable more complete testing Conduct many more test threads than would be otherwise possible in live testing Make better use of live test time Help select tests that the SoS SE and PM are willing to invest in live testing because of their importance to the SoS mission when M&S “testing” shows marginal performance in a critical capability One “opportunity” flows primarily from “SoS SE Update” to “M&S of the SoS” Feedback from field ops and exercises is used to refine the M&S of the SoS, with the result that M&S better serves “Conduct SoS Analysis” and “Develop/Evolve Architecture “in future waves. Improve testing … and feedback to SE Opportunities 25

26. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Implement SoS SE Update M & S Challenges Depends upon a solid representation of architecture and SoS baseline including dependencies …with the all the challenges discussed above Independence of constituent systems may make access to surrogate systems simulations as difficult as coordinated testing Systems simulations may not be technically suitable for use as surrogates Opportunities Improve testing … and feedback to SE Challenges 26 Pain Points Lack of SoS Authorities & Funding Leadership Constituent Systems Capabilities & Requirements Autonomy, Interdependencies & Emergence Testing, Validation & Learning

27. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Implement SoS SE Update Exemplars 27 Simulations used to augment defense systems testing PM Magazine, Sept-Oct 1997, Defense Acquisition University One-SAF C0Developer Technical Exchange Meeting http://www.onesaf.net/community/static/web/Vienna/1 23_OneSAF_CoDeveloper_TEM_InterTEC_v2.pdf

28. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Implications of Pain Points for SE and M&S of SoS (1 of 3) Lack of SoS Authorities & Funding and Leadership –Poses serious governance and management issues for SoS –Recognized impact on ability to implement SE as traditionally applied to products –Similar impacts on M&S based on ability to implement effective crosscutting M&S activities Constituent Systems –Beyond this, when the systems in an SoS are designed for different purposes and contexts which persist, there are technical issues with aligning systems to SoS architecture and context –Likewise, models, simulations and data for these systems will naturally be attuned to the specific needs of the systems, and may not lend themselves to supporting SoS analysis or engineering 28

29. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Implications of Pain Points for SE and M & S of SoS (2 of 3) Autonomy, Interdependencies & Emergence –Complexity characterizes SoS derived from the autonomy (and often complexity) of the constituents, their interdependencies, and the potential for emergent behavior which is often unpredictable –While M & S can aid in representing and assessing these complexities to the degree they are understood, these issues pose challenges for M & S particularly when the understanding is limited and can magnify issues of validation Capabilities & Requirements –Top level SoS capability needs may be high level and need definition in order to align them with requirements which are supported by constituent systems, which may not be able (or willing) to address them –M & S can support tools to assist in this process, but these too are dependent on a articulation of capabilities in order to effective represent them 29

30. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Implications of Pain Points for SE and M & S of SoS (3 of 3) Testing, Validation & Learning –The independence and asynchronous evolution of the constituent systems in an SoS, poses challenges for SoS testing, as do the costs and practicalities of creating test environments for large dynamic SoS –M & S provides options clear options for augmenting live test and addressing risk in SoS when testing is not feasible; however, the requires a solid representation of the SoS which can be difficult as discussed in earlier points SoS Principles –The lack of well understood SoS principles and examples of success hinders progress in SoS SE –Similarly, the absence of a well established base of SoS principles to drive development of M & S and knowledge on how to best apply M & S in this context is an obstacle to effective use of potentially powerful tools 30

31. International Workshop 26 Jan – 29 Jan 2013 Jacksonville, FL, USA MBSE Workshop Summary and Next Steps Challenges to application of SE to SoS as reflected in the SoSWG pain points –Similar challenges when considering SoS context in engineering an individual system Modeling and Simulation can help address these challenges when supporting SoS SE …however, same challenges can also modeling simulation Collaboration between SoSWG and MBSE teams provides an opportunity for progress with increased likelihood of success for both sides 31