The Chaos Report © The Standish Group 1995

Terminology Large Company: more than $500 million/year in revenue. Medium Company: between $200 and $500 million/year in revenue. Small Company: between $100 and $200 million/year in revenue.

Why do we care? In the United States - more than $250 billion spent each year on approximately 175,000 projects. Avg Cost large company is $2,322,000 Medium company, it is $1,331,000 small company, it is $434,000.

That wouldn’t be bad but... Estimates that in 1995 American companies and government agencies will spend $81 billion for cancelled software projects As well as an additional $59 billion for software projects that exceed their original time estimates.

What was found: A staggering 31.1% of projects will be cancelled before they ever get completed. On success side, avg. is 16.2% for on time + budget. (Worse for large company’s, it’s only 9%) About 52.7% of projects will cost 189% of their original estimates. Risk may be higher for newer project but many are mundane.

Example City of Denver – failure to produce reliable software to handle baggage at the new Denver airport is costing the city $1.1 million per month. Originally scheduled for October 31, 1993 ended on February 28, 1995. Originally single system for all three concourses, was actually a separate one for each concourse.

How was the report created? Across industry (e.g. banking, securities,... etc) Respondents were IT executive managers. Across corporation size. (Small, medium and large companies). Total sample size was 365, and represented 8,380 applications (or projects).

How was the report created? Cont. Resolution Type 1, - Success : On-time, on-budget with all features and functions initially specified. (16.2%) (lowest in Large companies) Resolution Type 2, - Challenged: Completed and operational, but over-budget, over time, and/or fewer features/functions. (52.7%) (largest in Large companies) Resolution Type 3 – Impaired: The project was cancelled at some point during development cycle. (31.1%) (largest in medium companies)

Reasons why projects are not Type 1: Restarts: For every 100 project there were 94 restarts (not 94/100 some may have restarted more than once.) See California Department of Motor Vehicles. Overruns: This may be time or cost or both. For Type 2 and Type 3 projects almost a third had a cost overrun of 150 to 200%. Over a third was 200 to 300% over their estimated time. Content Deficiencies: For challenged projects only, there was an average of 61% of originally specified features.

Enough about stats, why did they fail? Top three things that help make or break a project: User involvement. (15.9%) (12.8%) Executive management support. (13.9%) (7.5%) Clear statement of requirements. (13.0%) (12.3% / 11.8%) Some others included lack of resources, planning, time frames, expectations, objectives. % - when participants were asked about factors that cause projects to succeed. % - when participants were asked about factors that cause projects to be challenged. 12.3- incomplete req. 11.8 – changing requirements.

Some quotes from people in the field (from the focus groups): John, Director of MIS at a government agency added: "Probably 90% of application project failure is due to politics!" Bill, the Director of MIS at a securities firm, "Changes, changes, changes; they're the real killers." "Brain-dead users, just plain brain-dead users," said Peter, an application analyst at a bank. Sid, a project manager at an insurance company. "The project was two years late and three years in development, We had thirty people on the project. We delivered an application the user didn't need. They had stopped selling the product over a year before."

Case Study1: California DMV (Department of Motor Vehicles) 1987, the California DMV tried to overhaul their drivers license and registrations application process. 1993, after $45 million dollars were spent the project was cancelled. Their objective: To have a system that was able to rapidly respond to change. Reasons why it failed: No monetary payback, lack of executive management, no user involvement, poor planning, poor design specifications, and unclear objectives.

Case Study2: Hyatt Hotels Their objective: To update their reservation system. They succeeded, total cost was $15 million (under budget), and ahead of schedule. Reasons why it succeeded: High user involvement, executive management support, a clear statement of requirements, proper planning, and small milestones.

Conclusion Still need a more comprehensive report. Developed a success potential chart (which weights each component like user involvement) to help predict the success of a project. Failures and successes have to be continually examined, and reported. Suggested a similar process to what was talked about in the SCUM video. (That is short iterations, deploy small elements, etc.)

Problem with the report.... Specific data cannot be made publically available, so no way to verify data. Problem with definitions of the Types. (i.e. a project which arrives on budget and time, but didn’t have all the features doesn’t have a specific type.) Estimations are one-sided (ignores projects that are under-cost, time, or have more features) There must have been biases/generalizations made in order to get the cross intuitional figures. More details in Rise and Fall of Chaos.

References Chaos Report: http://www.projectsmart.co.uk/docs/chaos-report.pdf The Rise and Fall of Chaos: http://www.cs.vu.nl/~x/chaos/chaos.pdf