1 Predicting Bugs From History Software Evolution Chapter 4: Predicting Bugs from History T. Zimmermann, N. Nagappan, A Zeller

2 Motivations l Managers want to know where to spend resources in quality assurance activities. l In practice, this is based on developer expertise. By experience, developers have some intuition of where the bugs are more likely to occur. But memory is limited and, sometimes, inaccurate. l A solution: mine the bug database. Which modules have been known to be defect prone? In practice, not every module is defect prone. 80% of defects are in 20% of modules.

3 Research questions l Why are some modules more defect-prone than others? l Can we predict how defect prone a module will be?

4 What makes a module defect prone? l The defect likelihood of a module depends on its history. All activities that led to changes that affected it. History is never changed, only accumulated. Strategy: look for invariant properties of this history.

5 What makes a module defect prone? l Complexity Intuitively, the likelihood of making mistakes (injecting defects) into a software artifact increases with its complexity. Number of “details” in a software artifact. How these “details” relate and interact to each other. Challenge: how to measure complexity. l Problem domain The problem domain leads to product requirements. Some domains appear to be more difficult to work with than others. What is it about a domain that makes it harder? l Evolution When code is changed very frequently, it increases the likelihood of introducing defects. How can we decrease the likelihood? l Process Field problems imply that defects managed to slip through the quality assurance process. How do we take the quality of the process into account?

6 Measuring complexity l How do we measure complexity? l Traditional complexity measures: Size Lines of code, function points Code structure Cyclomatic complexity, number of variables Coupling relationships Module fan-in, fan-out Object-oriented metrics WMC, DIT, NoC, … l Do these metrics correlate with complexity? l Do these metrics really capture complexity?

7 Traditional O-O metrics l Weighted Methods per Class (WMC) l Depth of Inheritance Tree (DIT) l Number of Children (NOC) l Coupling Between Objects (CBO) l Response for a Class (RFC) l Lack of Cohesion in Methods (LCOM)

8 Empirical studies l Basili, et al. Student experiment. WMC, CBO, DIT, NOC, RFC were correlated with defects. l Briand, et al. Industrial case study. CBO, RFC, LCOM were correlated with defects. l Nagappan, et al. Case study of 5 Microsoft projects. Results are not that simple. Basili, Briand, Melo. A validation of object-oriented design metrics as quality indicators. IEEE TSE, Oct Briand, et al. Investigating quality factors in object-oriented designs: an industrial case study. ICSE Nagappan, Ball, Zeller. Mining metrics to predict component failures. ICSE 2006.

9 Microsoft case study l Which complexity metrics have a high correlation with defects? l 5 projects HTML rendering module for IE6. Application loader for IIS. Process Messaging Component. DirectX. NetMeeting. l Findings: In each case, defects correlated with some complexity metrics. But every project correlated with a different set of complexity metrics! No universal complexity metric was found. Hence, no universal prediction model can be formulated using complexity metrics. Never blindly trust a complexity metric.

10 General study methodology l Model development Variables Indices: complexity metrics, etc. Response: number of defects, defect density. Study the distribution of the variables to find suitable models. Use transformation functions as needed. Model formulation Use supervised learning models: regression, Bayesian, SVM, etc. Balance goodness-of-fit with simplicity (Ockham’s Razor). Examine significance, direction and magnitude of contribution for each independent variable. l Model validation Calculate precision and recall. Compare against the null hypothesis (is it better than guessing?) Test sensitivity of different factors, e.g., using stepwise regression. Test with hold out data – separate training set from validation set. Cross-validate: Leave one out. K-fold cross validation. For those interested, check out Andrew Moore’s statistical data mining tutorials:

11 Problem domain l Does the problem domain contribute to the defect proneness of a module? l Algorithms for some domains are difficult to get right. l Even reusing existing code is no guarantee.

12 Eclipse case study l Study of 52 Eclipse 2.0 plug-ins. l Determine the domain of each plug-in. Use the list of packages imported as a surrogate for the domain. Plug-ins that come with a GUI will require UI classes. Those that need access to the parse tree will need compiler classes. l Methodology: use supervised learning techniques. l Findings Java code that imported packages from the following are the most likely to have defects: org.eclipse.jdt.internal.compiler (> 0.7) org.eclipse.jdt.internal.ui (> 0.6) l Findings are specific to the Eclipse project. Schroter, Zimmermann, Zeller. Predicting component failures at design time. ISESE 2006.

13 Eclipse classes

14 Code churn l Code churn – the number of times a piece of code has been modified. l Some code churn metrics: Number of lines in a new version. Number of lines added/deleted. Number of files in a new version. Number of files added/modified/deleted. Number of changes (deltas) applied. Churn interval: from initial to final change. l Can code churn metrics correlate with defects?

15 Empirical studies l Ostrand, et al., Munson, et al., Nagappan, et al. [Ostrand] Industrial inventory system. [Munson] 300 KLOC embedded real-time system. [Nagappan] Windows Server New, changed and unchanged files were good predictors of defect proneness. l Graves, et al. 5ESS subsystem, 20-year history of changes. Churn metrics correlate to defects better than complexity metrics. Modules with more recent changes are more likely to have higher defect proneness. Ostrand, Weyuker, Bell. Where the bugs are. ISSTA Munson, Elbaum. Code churn: a measure for estimating the impact of code change. ICSM Nagappan, Ball. Use of relative code churn measures to predict system defect density. ICSE Graves, et al. Predicting fault incidence using software change history. IEEE TSE, July 2000.

16 Open issues l External validity Most findings come from case studies. Replication is needed to improve confidence on the results. l Construct validity Defect count is not the most important response variable. Some defects are more severe than others. We are interested in the ones that lead to failures. l Learning from history Findings are difficult to apply outside of specific projects. What lessons can we transfer to a new project that has no history? How can we abstract/consolidate/generalize our knowledge of the nature of defects from one project to another? Are there universal properties of programs and processes that likely result in more defects?

17 Some future work l Leverage process data Incorporate process features (process model used, quality assurance activities employed, etc.) into prediction models. l Better metrics and models Examine failures, not just defects. l Combined approaches Combine knowledge of complexity, domain, churn and process into models. l Finer-grained granularity White-box treatment of modules. Examine calling relationships, data dependencies, etc.