1 Forecasting Field Defect Rates Using a Combined Time-based and Metrics-based Approach: a Case Study of OpenBSD Paul Luo Li Jim Herbsleb Mary Shaw Carnegie Mellon University

2 Open Source Software Systems are Critical Infrastructure

3 Problem for Decision Makers Considering Open Source Software Systems Lack of quantitative information on open source software systems:  What is the quality?  How many defects are there?  When are they going to occur?

4 Possible Benefits of Field Defect Predictions Make informed choices between open source software systems Decide whether to adopt the latest software release Better manage resources to deal with possible defects Insure users against the costs of field defect occurrences

5 We Predict the Rate of Field Defects at the Time of Release Months after release Field defects Field defects for release 2.4

6 Months after release Field defects Field defects for release 2.4 Captured by the Model Parameters of a Software Reliability Model λ(t) = N α e – α t

7 Months after release Field defects Field defects for release 2.4 Predicting Parameters Using Metrics-based Methods λ(t) = N α e – α t i = information available before release f N (i) f a (i)

8 Months after release Field defects Field defects for release 2.4 Compare Predictions of Different Combinations

9 Talk Outline Field defect predictions can enabled quantitatively-based decisions regarding open source software systems There are good reasons for predicting the rate of field defect occurrences for software releases using a combined time- based and metrics-based approach evaluated using a real- time evaluation procedure We vary the Software Reliability Growth Model, the metrics- based prediction method, and the metrics used We evaluate our approach using OpenBSD We produce accurate predictions

10 We Take the Customer’s Perspective Predicted field defects for  Individual software changes (Mockus et al.)  Files (Ostrand et al.)  Modules (Khoshgoftaar et al.)  Entire system (Kenney) The system is what the customer sees

11 We Predict the Rate of Field Defect Occurrences Relationships: e.g. Harter et. al. Classifications: e.g. Khoshgoftaar et al. Quantities: e.g. Khoshgoftaar et al. Rates of occurrences over time: e.g. Kenny Effective quantitatively-based decision making require knowing the rate of field defect-occurrences over time

12 We Use a Novel Approach Time-based approach: Musa and Lyu Metrics-based approach: Harter et al. and Khoshgoftaar et al. We use metrics-based modeling methods to predict model parameters of software reliability growth models

13 We Cannot Fit a Model Outright Months since first defect Release date Field defects for release 2.6 Defects

14 We Evaluate Predictions Using a Real-time Evaluation Procedure Split data from the same release into fitting and testing sets e.g. Khoshgoftaar et al. Use a model fitted using data from a historical release to predict for future releases e.g. Ostrand et al. Estimate the model parameters for active historical releases using a software reliability model and the field defect data available at the time of release

15 At the Time of Release Release date of R2.7 Have predictor information Field defects Months relative to current time Field defects at release 2.7

16 Some Historical Releases Release date of R2.4 Field defects Months relative to current time Field defects at release 2.7

17 Have Complete Field Defect Information Release date of R2.4 Fitted model for R2.4λ(t) = *.25 e –.25 t Field defects Months relative to current time Field defects at release 2.7

18 Other Historical Releases are Still Active Release date of R2.5 Release date of R2.6 Field defects Months relative to current time Field defects at release 2.7

19 We Only Use Information Available at the Time of Release Field defects at release 2.7 Estimated model for R2.5 λ(t) = *.17 e –.17 t Estimated model for R2.5 λ(t) = *.14 e –.14 t Field defects Months relative to current time

20 Talk Outline Field defect predictions can enabled quantitatively-based decisions regarding open source software systems There are good reasons for predicting the rate of field defect occurrences for software releases using a combined time- based and metrics-based approach evaluated using a real- time evaluation procedure We vary the Software Reliability Growth Model, the metrics- based prediction method, and the metrics used We evaluate our approach using OpenBSD We produce accurate predictions

21 We Vary the Type of Software Reliability Growth Model Exponential model λ(t) = N α e – α t Field defect fits for release 2.6 Field defects Months since release

22 Some are More Complex Gamma model λ (t) = N β α t α-1 e – β t Field defect fits for release 2.6 Field defects Months since release

23 Post-facto Fits Show that the Weibull Model is Better Weibull model λ (t) = N α β t α-1 e – β t α Field defect fits for release 2.6 Field defects Months since release

24 Prediction Methods Tried Moving averages (Li et al.) Exponential smoothing (Li et al.) Linear modeling with model selection (Khoshgoftaar et al.) Principle component analysis, clustering, and linear modeling (Khoshgoftaar et al.) Trees (Khoshgoftaar and Seliya) Nonlinear modeling (Khoshgoftaar et al.) Neural networks (Khoshgoftaar et al.)

25 A Close Look at Moving Averages Parameter N R 2.4R2.5R Moving average 1 release:

26 Does Not Use Any Metrics Parameter N R 2.4R2.5R Moving average 2 release:

27 A Closer Look at Trees

28 Predictors Used Product metrics  Used in many prior studies e.g. Munson and Khoshgoftaar Development metrics  Used in many prior studies e.g. Khoshgoftaar et al. Deployment and usage metrics  Khoshgoftaar et al.  Mockus et al. Software and hardware configurations metrics  Mockus et al. Details in Li et al. in Metrics 2005

29 Talk Outline Field defect predictions can enabled quantitatively-based decisions regarding open source software systems There are good reasons for predicting the rate of field defect occurrences for software releases using a combined time- based and metrics-based approach evaluated using a real- time evaluation procedure We vary the Software Reliability Growth Model, the metrics- based prediction method, and the metrics used We evaluate our approach using OpenBSD We produce accurate predictions

30 OpenBSD We examine 10 releases ( ) OpenBSD is a Unix like operating system The OpenBSD project uses the Berkley copyrights The OpenBSD project puts out a release approximately every six months The OpenBSD project uses a CVS code repository The OpenBSD project uses a problem tracking system The OpenBSD project has multiple mailing lists.

31 Forecast Evaluation Parameter prediction method Historical releases: Current release: Theil forecasting statistic: √ (Σ(Actual – Predicted) 2 ) √( Σ(Actual) 2) A1A1 A2A2 Actual = (A2-A1) P2P2 Predicted = (P2-A1) Perfect forecast: P2 = A2 (Actual – Predicted) = ((A2-A1) – (A2-A1)) = 0 Theil statistic of 0 P2P2 P2P2 Uninformed forecast: P2 = A1 (Actual – Predicted) =((A2-A1) – 0) = ((A2-A1) – 0) = Actual Theil statistic of 1 Special cases:

32 Talk Outline Field defect predictions can enabled quantitatively-based decisions regarding open source software systems There are good reasons for predicting the rate of field defect occurrences for software releases using a combined time- based and metrics-based approach evaluated using a real- time evaluation procedure We vary the Software Reliability Growth Model, the metrics- based prediction method, and the metrics used We evaluate our approach using OpenBSD We produce accurate predictions and…

33 Using Only Information Before Release Field defect forecasts for release 2.7 Field defects Months since release Field defects

34 Forecast Field Defect Rates for the Next Release Field defect forecasts for release 2.7 Field defects Months since release

35 Our Forecasts are Accurate Field defect forecasts for release 2.7 Field defects Months since release

36 Theil Statistics Model, method, predictor combination R2.5R2.6R2.7R2.8R2.9R3.0R3.1R3.3R3.4AvgRange Exponential model using the moving averages method of 2 releases using no predictors Exponential model using the non- linear regression method using lines of code (same predictors as referenced work) Exponential model using the trees method splitting with six observations using all predictors Exponential model using the exponential smoothing method of five releases using no predictors Gamma model using the non- linear method using lines of code (same predictors as referenced work)

37 Exponential Model Produces Better Results 9 out of the 10 best methods ranked by average Theil use the Exponential model

38 Why? Model form is simpler λ(t) = N α e – α t λ (t) = N β α t α-1 e – β t λ (t) = N α β t α-1 e – β t Only two model parameter No power term α

39 Averaging Methods are Better Judge by average Theil  6 out of 10 use weighted averaging methods (moving averages and exponential smoothing)  2 out of 10 use trees  2 out of 10 use non-linear regression methods Trees is best if we add:  number of releases for which a method produced the best forecast  number of releases that it predicted for

40 Why? Neural network: referenced work had 188 and 226 training observations Trees: referenced work had 4648 training observations We had at most 9 observations

41 What Predictors to Use… Moving average and exponential smoothing methods used no predictors Trees method used all the predictors Non-linear regression method used lines of code

42 Why?

43 Deployment and Usage and Software and Hardware Configuration Metrics are Important MetricDefinitionPrediction used AllDefectHW Sparc Number of field defects reported against all active release during the development period that identify the machine as type Sparc parameter N for R3.0 and R3.3 LOCLines of code parameter α for R3.0 and N for R3.1 Comment Inline Number of inline comment parameter α for R3.1 and R3.3 TechMailing Number of messages to the technical mailing list during the development period parameter N for R3.4 NotCUpdate Number of updates (deltas) to files that are not c source files during the development period parameter α for R3.4

44 Talk Outline Field defect predictions can enabled quantitatively-based decisions regarding open source software systems There are good reasons for predicting the rate of field defect occurrences for software releases using a combined time- based and metrics-based approach evaluated using a real- time evaluation procedure We vary the Software Reliability Growth Model, the metrics- based prediction method, and the metrics used We evaluate our approach using OpenBSD We produce accurate predictions and…

45 Where to Go From Here? Validate results using:  Commercial systems  Other open source systems Update predictions as more data becomes available:  Bayesian approach (Liu et al.)  U-plot (Brocklehurst and Littlewood) Case studies applying our technique

46 Forecasting Field Defect Rates Using a Combined Time-based and Metrics-based Approach: a Case Study of OpenBSD Paul Luo Li Jim Herbsleb Mary Shaw Carnegie Mellon University