1. Using Bayesian Nets to Predict Software Defects in Arbitrary Software Lifecycles Martin Neil Agena Ltd London, UK Web: www.agenarisk.com Email: martin@agena.co.uk Telephone: +44 (20) 7404 9722

2. Effective risk assessment in software projects See problems early and take mitigating actions Predict when project will be late and/or over-budget Decide early enough if project is infeasible Predict product quality and the impact of poor reliability Know when it makes sense to stop testing Support process improvement by determining the impact of good/poor processes

3. Impediments to effective software risk measurement Lack of data Difficulty of communicating risks Poor requirements management Poor coordination with stakeholders and customers Difficulty in managing expectations Difficulty of making progress visible Lack of applicability of traditional project management tools

4. New solutions for software project risk Risk maps for high-level project risk assessment Risk maps for managing and predicting defects during and after development Risk maps for project risk health checks

5. What the defect prediction risk maps provide Accurate predictions of defects at different phases Know when to stop testing Identify where to allocate testing Minimise the cost of rework Minimise danger and cost to user Highlight areas for improvement Approach fully customisable Models arbitrary project life-cycles

6. AgenaRisk modelling and validation AgenaRisk template risk maps designed using –Expert judgement –Published data –Company data –Published models Risk maps validated on projects at: –QinetiQ –Israel Aircraft Industries –Philips Netherlands and India

7. What’s special about the approach? Structured Method (for quantifying ALL risks) –Based on well proven Bayesian technology –Beyond current statistical & Monte Carlo techniques –Combines subjective judgements with historical data Visual Communication (of complex risk issues) –Manages risk through pictures –Useable by non-specialists as well as risk experts –Makes analysis of complex risks easy to communicate Robust (for large & complex risk applications) –Enables scalable, reusable & auditable risk models –Integrates easily with databases and spreadsheet systems –Enables developers to build end-user applications

8. Traditional metrics and statistical approaches Software metrics studies provide useful empirical baseline Regression-based models are good for predicting the past.. but are of little use for true risk assessment

9. Background: History of Reliability Modeling at Philips - Bangalore Study of Reliability Models AID 2000 MODIST Agena Risk 200120032002 2004

10. Phase-level risk map for defect prediction

11. Schematic of (single) phase-level risk map for defect prediction New functionality implemented Spec & doc Process quality Development Process quality Testing Process quality Rework Process quality Defects PRE Total defects in Defects POST Defects found Defects fixed

12. Running the defects risk map New functionality implemented Spec & doc Process quality Development Process quality Testing Process quality Rework Process quality Defects PRE Total defects in Defects POST Defects found Defects fixed 250 0 Median value 257

13. High quality development New functionality implemented Spec & doc Process quality Development Process quality Testing Process quality Rework Process quality Defects PRE Total defects in Defects POST Defects found Defects fixed 150 0 Very high Median value 40 Median value 25

14. … but suppose more than expected defects found in testing New functionality implemented Spec & doc Process quality Development Process quality Testing Process quality Rework Process quality Defects PRE Total defects in Defects POST Defects found Defects fixed 150 0 Very high Median value 204 400 So process assumptions “overruled” by hard facts from QA/test results

15. How good must your testing and rework be to get ZERO residual defects? New functionality implemented Spec & doc Process quality Development Process quality Testing Process quality Rework Process quality Defects PRE Total defects in Defects POST Defects found Defects fixed 150 0 0

16. Linking phase level risk maps in AgenaRisk Enables us to model arbitrary projects and process Level of granularity can be chosen to suit the available data

17. AgenaRisk Validation Objectives To evaluate AgenaRisk (Method+Toolset) for its defect prediction capabilities in software projects To help Philips decide whether it is a potential candidate for deployment within software development centers of Philips CE Validation was conducted during Q4/2004 - Q1/2005

18. Distribution of selected projects (actual number of projects shown in brackets)

19. Results: Actual vs Predicted Defects (Round 1, 2 & 3)

20. Actuals Vs Predictions

21. Validation Summary Approach perceived as innovative and promising Our evaluation showed excellent results for project sizes between 10 and 90 KLOC Initially, projects outside this range were not within the scope of the default model so predictions were less accurate, but accuracy was significantly improved after tailoring standard model

22. The Road Ahead Revised model tailored as Philips-specific for even more accurate results and for projects outside the scope of the default model –New structure already proposed –Replacing the underlying assumptions (where required) –Tailoring questionnaire interface for end-user (with explanations using organization-specific terms) –Training potential “users” for “what-if-scenarios” and “risk assessment” Used AgenaRisk on live projects Use AgenaRisk for post-release (Field Call Rate) defect prediction Incorporate other factors like effort, schedule, etc

23. Summary New approach to defect prediction and quality control based on Bayesian Networks, enables us to –Take account of process and people factors (with support for process improvement) –Monitor and predict fault injection-detection and rework through lifecycle –Avoid project failures and improve project communication –With minimal additional data collection requirements and costs Validation results show that the AgenaRisk tool and models can be used as is and that simple tailoring can enable even greater power and accuracy for Philips

24. Further Information Web: www.agenarisk.com Email: martin@agena.co.uk © Agena Ltd 2005