The Cost of Inadequate S/W Testing The Economic Impacts of Inadequate Infrastructure for Software Testing National Institute of standards & Technology: Planning Report 02-3 May 2002 www.nist.gov/director/prog-ofc/report02-3.pdf An improved infrastructure for testing has the potential to affect developers and users by: removing more bugs before the s/w product is released. Detecting bugs earlier in the s/w life-cycle. Locating the source of bugs earlier and with more precision.

The Costs of Inadequate S/W Testing The study investigates the economic impacts of an inadequate infrastructure for s/w testing in the U.S. Study conducted surveys with both developers and industry users. Two industry groups were selected for detailed analysis. Findings from these two groups were then used to estimate total economic impacts for U.S. Note - Impact estimates do not reflect “costs” associated with mission critical s/w where failure can lead to extremely high costs such as loss of life or catastrophic failure.

Economic Impact on Automotive and Aerospace Industries Study focused on the use of Computer-aided design (CAD), computer-aided manufacturing (CAM), computer-aided engineering (CAE), and product data management (PDM). Interviews were conducted with 10 s/w developers (vendors), and 179 users of these products. Users reported an average of 40 major and 70 minor bugs per year in their CAD/CAM/CAE/PDM software systems.

Cost Impacts on Developers and Users Developers (vendors) - Costs Impacts of $.373 (bil) Potential cost savings from feasible infrastructure improvements - $.158 bil) Software Users Automotive - Cost impacts of $1.230 (bil). Potential savings of $370 (bil) Aerospace - Cost Impacts of $.237(bil). Potential savings of $.055 (bil) Total Cost Impacts of $1.840(bil). Total Potential savings 0f $.598 (bil) Users account for approx.. three-quarters of total impact, Developers account for the remaining one- forth of costs.

Economic impact on Financial Services Sector Study focused on developers and users of Financial Electronic Data Interchange (FEDI) and clearinghouse s/w, as well as the s/w embedded in routers and switches that support electronic data exchange. Four s/w developers, and 98 users. Users reported an average of 40 major and 49 minor bugs. Approx.. 16% of bugs attributed to router/switch problems, and 48% to transaction s/w problems. The remaining 36% of errors were unknown.

Cost Impacts on developers and Users Routers and Switches - Cost impact of $1.898 (bil). Potential savings from feasible infrastructure improvements of $.975 (bil) FEDI and Clearinghouse - Cost impact of $.438 (bil). Potential savings of $.225 (bil) Software Users Banks and savings institutions - Cost impacts of $.789(bil). Potential savings of $.244 (bil) Credit Unions - Cost impacts of $.217 (bil). Potential savingsof $.068 (bil) Total cost impacts of $3.343(bil). Total potential savings of $1.513 (bil) Developers account for 75% of costs, with users the remaining 25%.

National Impact Estimates The per-employee impacts of these sectors were extrapolated to develop an estimate of the economic impacts of an inadequate s/w infrastructure for s/w testing for the total US economy. Developers - Cost impacts of $21.2 (bil). Potential savings of $10.6 (bil). Users - Cost impacts of $38.3 (bil). Potential savings of $11.7 (bil) Total cost impacts of $59.5(bil). Potential savings of $22.2 (bil). Developers accounted for about 40% of total impacts, with Users accounting for the remaining 60%.