Understanding Frequent Root Causes of System-development Failure 7 March 2012 Neil Siegel Vice-President & Chief Engineer

Failure is not Uncommon The record indicates that the development of large-scale systems remains an endeavor that often fails. –Requiring significantly more money &/or time to complete than originally planned –Under-delivery of specified functionality –Lack of suitability of the delivered system for the actual intended use –Cancellation of the development project before a useful product has been delivered For example, (Glass 2001) cites data indicating that only about 16% of system development projects that he examined were listed as successful by their own developers. Analyses of root causes * tend to focus on factors such as incomplete requirements, changing requirements, and so forth. –These are sometimes symptoms, and not causes. –I offer four candidate root causes, and discuss how to address each. 2 * For example, (Boehm )

“More Precision than Accuracy” “Effective but not Suitable” Failures Too Late / Too Expensive to be useful 3 Four Root Causes for Failures

We may have a great system specification, but the “wicked” nature of the problem prevents us from actually achieving consensus on what they system needs to do, even if we think we have already done so. –Ill-defined –Involve many stakeholders with strong and opposing views. –Have conditions that change midstream. –Are misunderstood until a solution is in hand.* In many large-scale endeavors, the social factors must be addresses in synchronicity with the technical problems. –So our specification – and contract, and statement-of-work, and design baseline, etc – are likely of little real value in reaching a satisfactory conclusion to the project. 4 * Quoted from Steve Nixon, “Wicked Problems, November Used with permission. More Precision than Accuracy

Adapted from Steve Nixon, “Wicked Problems, November Used with permission. 5 Every time we discuss it with the users, we get important new insights about what the problem actually is that we are trying to solve. The problem seems actually to change. We can’t get the stakeholders to agree. We don’t seem actually to know who are all of the stakeholders – we keep finding new ones. “Everything should talk to everything” – we can’t seem to bound the problem. Recognizing Wicked Problems

Experimentation Collaboration Social complexity from integrated networks is a key driver. Traditional linear solution styles are not well-suited. 6 Adapted from Steve Nixon, “Wicked Problems, November Used with permission. Solving Wicked Problems

95%+ of our specifications describe desired functionality, but experience suggests: –That while the resulting systems may be effective (in the sense that they provide the specified functionality), they are not suitable (in the sense that they fail to operate appropriately within the intended environment, falling short in areas such as reliability, response times, ease-of-use, being excessively prone to configuration-driven errors, and so forth). –There are many systems that are considered failures... even after being shown to meet their specification! What to do: –Achieve far higher reliability in software-based systems. –Design to stay within the capability and interest-level of the intended user. –etc. 7 “Effective but not Suitable”

8 “90-90” Failures Example scenario: –We have decomposed our system into a set of small components, each of which has been implemented. –When we start putting the system together, however, all sorts of failures and difficulties arise, performance is unacceptable, and the schedule and cost estimates are repeatedly exceeded. The problem is often unplanned dynamic behavior. What can we do better: –“Design for integration”

9 Too Late / Too Expensive to be Useful Example scenario: –The amount of time (or money, or both) required to build the capability makes it no longer of interest. –Due to repeated breaches of cost and schedule estimates, the development team has lost credibility with the funders &/or users. What can we do: –Agile methods –Radical reduction in SLOC counts

10 Summary Cost increases of 2x, 3x, even 10x are signals of something other than “requirements creep” –Attributing failure to “lack of complete requirements” could be interpreted as passing the blame to someone else –I believe that we in the development community need to take more responsibility for achieving more consistently-better performance How: –Recognize the social aspect of our job, and thereby, deal with the “wicked” aspects of systems development –Recognize that we have to deliver systems that are suitable, as well as effective –Deal better with projected dynamic behavior in our designs, and thereby avoiding “90-90” failures –Create methods that will allow us to deliver system within budgets and schedules that are of interest

NORTHROP GRUMMAN PRIVATE / PROPRIETARY LEVEL 1 Q & A Questions? 11