1. Information Systems Overview Organizing Complex Projects Around Risks Arising From System Dynamic Behavior Neil Siegel Sector Vice-President & Chief Engineer Northrop Grumman 9 March 2011

2. Agenda Scope Hypothesis The design-based technique Research results Summary 2

3. Scope of proposed study This study aims to assess the efficacy of one particular method of improving the outcomes on complicated, large-scale, software-intensive system development projects 3

4. “Bottom-line up front” 4

5. Systems are important to society, but many system development efforts fail 5 What do we mean by “fail”: –Do not produce a product that meets the original specifications –Produce such a product only after taking significantly more time and/or money than originally expected. –In the extreme, many such projects are cancelled before completion Failure is apparently common: –(Glass 2001) cites data indicating that only about 16% of the system development projects that he surveyed were listed as successful by their own developers –(Johnson 2006) cites data from the Standish Group, which describes results from a 2004 survey: just 29% of development projects succeeded

6. Scope of the systems of interest Complex emergent behavior, as described by (Rechtin 1991) Interactions with physical devices (physically-moving mechanisms, other time-sensitive mechanical devices, etc.) Stressing asynchronous stimuli (such as extraordinary high data-ingest rates, or highly-stressed communications structures) Extraordinarily high availability / reliability requirements Development efforts of large size Systems that need to display much early progress through prototyping and re-use 6

7. Hypotheses During the development phase of a large- scale, complex computer-based system, the use of a specific design-based technique that centralizes the control of the dynamic behavior of a system will lower the density of those defects that are attributable to unplanned adverse dynamic system behavior 7

8. Partitioning the work I propose using the design process to partition the work into different “skill bins”, so as to provide better ways to assign people to tasks. –This allows a particular difficult task (control and management of the system’s dynamic behavior) to be partitioned away from most of the development team. Under current methods, the “hard” parts of the work can be disseminated into a large portion of the tasks 8

9. How I accomplish that partitioning The System Architecture Skeleton 9

10. Results 10

11. All 6 periods / all 4 cases 11

12. Defect density “ Use of the design-based technique will reduce the density of defects attributable to errors in managing system dynamic 12 Project YYYY, periods I-III contractor tests, attributable problem reports by month. behavior”

13. 13 Cost performance also tracks the dependent variable

14. 14 Summary & interpretations

15. 15 The data indicate that the proposed design- based technique may in fact lead to better outcomes. Indicated by a materially-lower density of defects (on the order of four times lower) that were attributable to errors in controlling system dynamic behavior Summary & interpretations

16. The design-based technique considered herein only applies to projects where such centralization is possible –The study did demonstrate that this set of projects is not vanishingly small This specific technique, however, is only one possible technique for creating a partitioning of a project into easy and difficult portions –Future studies could propose and assess other such techniques. 16 Summary & interpretations

17. Implications for practice Control structures may be more important than generally recognized New goal for the design process 17

18. Questions? 18

19. NGIS clearance approval number: ISHQ-2011-0010 19