Statistical Analysis at BAE NS Making Statistics Part of Decision Making in an Engineering Organization Card, Domzalski, Davies IEEE Software, May/June 2008

Why use statistical analysis? To help achieve CMMI and ISO 9001 Statistical methods part of Six Sigma and Lean Inspired by manufacturing – from Gantt charts to SPC Decisions made under uncertainty

Agree budget and schedule, where target values are approximate Decide an anomaly represents a problem to be resolved Release a system for acceptance testing, even though it is not possible that every fault has been identified

Focus on Process Management Two levels of decision making:  Real-time control of individual activities or sub-processes  Predictions of process outcomes based on current or completed activities Many statistical methods to choose from Need for training Nontrivial to implement statistical methods

Implementing statistical methods Use external expertise  Can cause delays  External experts have limited knowledge of the organization Use for occasional projects  No ongoing analysis for lifetime of process Embed statistical analysis in the regular jobs of engineers and managers – the focus here

BAE NS 1989: start capability maturity assessment 1992: CMM level : CMM level : from software to systems engineering; SPC for QA and for systems 2002: CMM level : CMM level 5 Data collection essential to this

Statistical Techniques Many techniques to choose from Aim to 'manage process performance so that it reliably produces desired results' Objectives:  Stabilize the process; essential for prediction  Manage the process to achieve objectives  Identify process improvements BAE NS chose SPC initially

Why SPC? It fits the CMMI requirements Easy to teach and to understand Make explicit the task of building statistical models Allows for distributions other than normal and for variances that are unequal or even unstable

Where should SPC be applied? Easy to make mistakes in application Individual and moving range charts used  To identify anomalies  To track changes in the mean or standard deviation  To identify patterns These trigger further investigation Defect profiles indicate actual problem (if any)

Deployment Strategy Previously, statistical analysis confined to small groups With high-maturity initiative, this was extended Extensive training Small teams of experts who worked with project teams

Results Variability of effort reduced Post-delivery defect density reduced Trends consistent across different project types

Improved decision making Statistical techniques improve decision making Decisions based on facts rather than opinions Decision making approach is visible The same methods and rules for making decisions can be repeated Uncertainty can be quantified and so contained

Challenges Transition from 'measure as goal' to 'measure as feedback' Mistakes and misunderstandings; e.g. SPC calculations for mean and standard deviation differ from those embedded in Excel. Targeting subprocesses is difficult Large quantities of data to process quickly

Challenges at all levels Team leaders and members perceive SPC as telling them what they already know Managers may misuse measures Large amounts of data are made available to decision makers; they must resist the temptation to micromanage

Recommendations SPC is not just arithmetic Provide practical training Ensure data quality Define objectives and connect to project activities Define measurement and analysis tasks Emphasise need to act on perceived anomalies Management must participate

Recommendations (continued) Adapt the approach in response to feedback Provide support in projects, as well as providing training Start with simple techniques (like SPC); introduce advanced techniques later Automation can help, but should not drive the approach Everyone must be able to contribute to improvement