1. © ABB Corporate Research - 1 - January, 2004 Experiences and Results from Initiating Field Defect Prediction and Product Test Prioritization Efforts at ABB Inc. Paul Luo Li (Carnegie Mellon University) James Herbsleb (Carnegie Mellon University) Mary Shaw (Carnegie Mellon University) Brian Robinson (ABB Research)

2. © ABB Corporate Research - 2 Field defects matter Citizen Thermal Energy, Indianapolis, Indiana, USA Molson, Montreal, Canada Heineken Spain, Madrid, Spain Shajiao Power Station, Guandong, China

3. © ABB Corporate Research - 3 ABB’s risk mitigation activities include… Remove potential field defects by focusing systems/integration testing Higher quality product Earlier defect detection (when its cheaper to fix)

4. © ABB Corporate Research - 4 ABB’s risk mitigation activities include… Remove potential field defects by focusing systems/integration testing Higher quality product Earlier defect detection (when its cheaper to fix) Plan for maintenance by predicting the number of field defects within the first year Faster response for customers Less stress (e.g. for developers) More accurate budgets

5. © ABB Corporate Research - 5 ABB’s risk mitigation activities include… Remove potential field defects by focusing systems/integration testing Higher quality product Earlier defect detection (when its cheaper to fix) Plan for maintenance by predicting the number of field defects within the first year Faster response for customers Less stress (e.g. for developers) More accurate budgets Plan for future process improvement efforts by identifying important characteristics (i.e. category of metrics) More effective improvement efforts

6. © ABB Corporate Research - 6 Practical issues How to conduct analysis with incomplete information? How to select an appropriate modeling method for systems/integration testing prioritization and process improvement planning? How to evaluate the accuracy of predictions across multiple releases in time? Experience on how we got to the results

7. © ABB Corporate Research - 7 Talk outline ABB systems overview Field defect modeling overview Outputs Inputs Insight 1: Use available information Modeling methods Insight 2: Select a modeling method based on explicability and quantifiability Insight 3: Evaluate accuracy using forward prediction evaluation Empirical results Conclusions

8. © ABB Corporate Research - 8 We examined two systems at ABB Product A Real-time monitoring system Growing code base of ~300 KLOC 13 major/minor/fix-pack releases ~127 thousand changes committed by ~ 40 different people Dates back to 2000 (~5 years) Product B Tool suite for managing real-time modules Stable code base of ~780 KLOC 15 major/minor/fix-pack releases ~50 people have worked on the project Dates back to 1996 (~9 years)

9. © ABB Corporate Research - 9 We collected data from… Request tracking system (Serena Tracker) Version control system (Microsoft Visual Source Safe) Experts (e.g. team leads and area leads)

10. © ABB Corporate Research - 10 Talk outline ABB systems overview Field defect modeling overview Outputs Inputs Insight 1: Use available information Modeling methods Insight 2: Select a modeling method based on explicability and quantifiability Insight 3: Evaluate accuracy through time using forward prediction evaluation Empirical results Conclusions

11. © ABB Corporate Research - 11 Breakdown of field defect modeling Predictors (metrics available before release) Product (Khoshgoftaar and Munson, 1989) Development (Ostrand et al., 2004) Deployment and usage (Mockus et al., 2005) Software and hardware configurations (Li et al., 2006) Inputs

12. © ABB Corporate Research - 12 Inputs Metrics-based methods Modeling method Breakdown of field defect modeling

13. © ABB Corporate Research - 13 Inputs Field defects Modeling method Outputs Breakdown of field defect modeling

14. © ABB Corporate Research - 14 Modeling process Inputs Take historical Inputs and Outputs to construct model Modeling method Outputs

15. © ABB Corporate Research - 15 Talk outline ABB systems overview Field defect modeling overview Outputs Inputs Insight 1: Use available information Modeling methods Insight 2: Select a modeling method based on explicability and quantifiability Insight 3: Evaluate accuracy through time using forward prediction evaluation Empirical results Conclusions

16. © ABB Corporate Research - 16 Outputs Field defects: valid customer reported problems attributable to a release in Serena Tracker Relationships What predictors are related to field defects? Quantities What is the number of field defects?

17. © ABB Corporate Research - 17 Outputs Field defects: valid customer reported problems attributable to a release in Serena Tracker Relationships Plans for improvement Targeted systems testing Quantities Maintenance resource planning Remember these objectives

18. © ABB Corporate Research - 18 Talk outline ABB systems overview Field defect modeling overview Outputs Inputs Insight 1: Use available information Modeling methods Insight 2: Select a modeling method based on explicability and quantifiability Insight 3: Evaluate accuracy through time using forward prediction evaluation Empirical results Conclusions

19. © ABB Corporate Research - 19 Inputs (Predictors) Product metrics Lines of code Fanin Halstead’s difficulty … Development metrics Open issues Deltas Authors … Deployment and usage (DU) metrics … we’ll talk more about this Software and hardware configuration (SH) metrics Sub-system Windows configuration …

20. © ABB Corporate Research - 20 Insight 1: use available information ABB did not officially collect DU information (e.g. the number of installations) Do analysis without the information? We collected data from available data sources that provided information on possible deployment and usage Type of release Elapsed time between releases Improved validity More accurate models Justification for better data collection

21. © ABB Corporate Research - 21 Talk outline ABB systems overview Field defect modeling overview Outputs Inputs Insight 1: Use available information Modeling methods Insight 2: Select a modeling method based on explicability and quantifiability Insight 3: Evaluate accuracy through time using forward prediction evaluation Empirical results Conclusions

22. © ABB Corporate Research - 22 Methods to establish relationships Rank Correlation (for improvement planning) Single predictor Defect modeling (for improvement planning and for systems/integration test prioritization) Multiple predictors that complement each other

23. © ABB Corporate Research - 23 Insight 2: select a modeling method based on explicability and quantifiability Previous work use accuracy, however… To prioritize product testing Identify faulty configurations Quantify relative fault-proneness of configurations For process improvement Identify characteristics related to field defects Quantify relative importance of characteristics

24. © ABB Corporate Research - 24 Insight 2: select a modeling method based on explicability and quantifiability Previous work use accuracy, however… To prioritize product testing Identify faulty configurations Quantify relative fault-proneness of configurations For process improvement Identify characteristics related to field defects Quantify relative importance of characteristics Explicability

25. © ABB Corporate Research - 25 Insight 2: select a modeling method based on explicability and quantifiability Previous work use accuracy, however… To prioritize product testing Identify faulty configurations Quantify relative fault-proneness of configurations For process improvement Identify characteristics related to field defects Quantify relative importance of characteristics Quantifiability

26. © ABB Corporate Research - 26 Insight 2: select a modeling method based on explicability and quantifiability Previous work use accuracy, however… To prioritize product testing Identify faulty configurations Quantify relative fault-proneness of configurations For process improvement Identify characteristics related to field defects Quantify relative importance of characteristics Not all models have these qualities e.g. Neural Networks, models with Principal Component Analysis

27. © ABB Corporate Research - 27 The modeling method we used Linear modeling with model selection 39% less accurate than Neural Networks (Khoshgoftaar et al.) example only: not a real model

28. © ABB Corporate Research - 28 Linear modeling with model selection example only: not a real model Explicability: distinguish the effects of each predictor The modeling method we used Function (Field defects) = B 1 *Input 1 + B 2 *Input 2 + B 3 *Input 4

29. © ABB Corporate Research - 29 Linear modeling with model selection example only: not a real model Quantifiability: compare the effects of predictors The modeling method we used Function (Field defects) = B 1 *Input 1 + B 2 *Input 2 + B 3 *Input 4

30. © ABB Corporate Research - 30 Talk outline ABB systems overview Field defect modeling overview Outputs Inputs Insight 1: Use available information Modeling methods Insight 2: Select a modeling method based on explicability and quantifiability Insight 3: Evaluate accuracy through time using forward prediction evaluation Empirical results Conclusions Skipping ahead… Read the paper

31. © ABB Corporate Research - 31 Systems/Integration test prioritization Product A PredictorsEstimated Effect Sub-system: Sub-system 19.85x more Sub-system 28.39x more Sub-system 38.13x more Sub-system 47.22x more Software platforms: Not Windows Server Versions 1.91x more Other predictors: Service Pack5.55x less Open Issues1.01x less Num Authors1.08x less Months Before Next Release1.16x more Months Since 1 st Release1.03x less Log (Field defects) = B 1 Input 1 + B 2 Input 2 …

32. © ABB Corporate Research - 32 Systems/Integration test prioritization Select a modeling method based on explicability and quantifiability Product A PredictorsEstimated Effect Sub-system: Sub-system 19.85x more Sub-system 28.39x more Sub-system 38.13x more Sub-system 47.22x more Software platforms: Not Windows Server Versions1.91x more Other predictors : Service Pack5.55x less Open Issues1.01x less Num Authors1.08x less Months Before Next Release1.16x more Months Since 1 st Release1.03x less

33. © ABB Corporate Research - 33 Systems/Integration test prioritization Select a modeling method based on explicability and quantifiability Product A PredictorsEstimated Effect Sub-system: Sub-system 19.85x more Sub-system 28.39x more Sub-system 38.13x more Sub-system 47.22x more Software platforms: Not Windows Server Versions 1.91x more Other predictors : Service Pack5.55x less Open Issues1.01x less Num Authors1.08x less Months Before Next Release1.16x more Months Since 1 st Release1.03x less

34. © ABB Corporate Research - 34 Systems/Integration test prioritization Select a modeling method based on explicability and quantifiability Product A PredictorsEstimated Effect Sub-system: Sub-system 19.85x more Sub-system 28.39x more Sub-system 38.13x more Sub-system 47.22x more Software platforms: Not Windows Server Versions 1.91x more Other predictors : Service Pack5.55x less Open Issues1.01x less Num Authors1.08x less Months Before Next Release1.16x more Months Since 1 st Release1.03x less

35. © ABB Corporate Research - 35 Systems/Integration test prioritization Select a modeling method based on explicability and quantifiability Use available information Improved validity More accurate model Justification for better data collection Product A PredictorsEstimated Effect Sub-system: Sub-system 19.85x more Sub-system 28.39x more Sub-system 38.13x more Sub-system 47.22x more Software platforms: Not Windows Server Versions 1.91x more Other predictors : Service Pack5.55x less Open Issues1.01x less Num Authors1.08x less Months Before Next Release1.16x more Months Since 1 st Release1.03x less

36. © ABB Corporate Research - 36 Systems/Integration test prioritization Experts validated results Quantitative justification for action ABB found additional defects Product A PredictorsEstimated Effect Sub-system: Sub-system 19.85x more Sub-system 28.39x more Sub-system 38.13x more Sub-system 47.22x more Software platforms: Not Windows Server Versions1.91x more Other predictors : Service Pack5.55x less Open Issues1.01x less Num Authors1.08x less Months Before Next Release1.16x more Months Since 1 st Release1.03x less

37. © ABB Corporate Research - 37 Talk outline ABB systems overview Field defect modeling overview Outputs Inputs Insight 1: Use available information Modeling methods Insight 2: Select a modeling method based on explicability and quantifiability Insight 3: Evaluate accuracy through time using forward prediction evaluation Empirical results Conclusions

38. © ABB Corporate Research - 38 Risk mitigation activities enabled Focusing systems/integration testing Found additional defects Plan for maintenance by predicting the number of field defects within the first year Do not yet know if results are accurate enough for planning purposes Plan for future process improvement efforts May combine with prediction method to enable process adjustments

39. © ABB Corporate Research - 39 Experiences recapped Use available information when direct/preferred information is unavailable Consider explicability and quantifiability of a modeling method when objectives are improvement planning and test prioritization Use forward prediction evaluation procedure to assesses accuracy of prediction for multiple releases in time Details on insights and results in our paper

40. © ABB Corporate Research - 40 - January, 2004 Experiences and Results from Initiating Field Defect Prediction and Product Test Prioritization Efforts at ABB Inc. Paul Luo Li (Carnegie Mellon University) James Herbsleb (Carnegie Mellon University) Mary Shaw (Carnegie Mellon University) Brian Robinson (ABB Research) Thanks to: Ann Poorman Janet Kaufman Rob Davenport Pat Weckerly

41. © ABB Corporate Research - 41 Insight 3 : use forward prediction evaluation Accuracy is the correct criterion when predicting the number of field defects for maintenance resource planning Current accuracy evaluation methods are not well-suited for multi-release systems Cross-validation Random data with-holding

42. © ABB Corporate Research - 42 Accuracy is the correct criterion when predicting the number of field defects for maintenance resource planning Current accuracy evaluation methods are not well-suited for multi-release systems Cross-validation Random data with-holding Release 1Release 2Release 3Release 4 Insight 3 : use forward prediction evaluation

43. © ABB Corporate Research - 43 Accuracy is the correct criterion when predicting the number of field defects for maintenance resource planning Current accuracy evaluation methods are not well-suited for multi-release systems Cross-validation Random data with-holding Release 1Release 2Release 3Release 4 Insight 3 : use forward prediction evaluation

44. © ABB Corporate Research - 44 Accuracy is the correct criterion when predicting the number of field defects for maintenance resource planning Current accuracy evaluation methods are not well-suited for multi-release systems Cross-validation Random data with-holding Release 1Release 2Release 3Release 4 Insight 3 : use forward prediction evaluation

45. © ABB Corporate Research - 45 Accuracy is the correct criterion when predicting the number of field defects for maintenance resource planning Current accuracy evaluation methods are not well-suited for multi-release systems Cross-validation Random data with-holding Release 1Release 2Release 3Release 4 Insight 3 : use forward prediction evaluation

46. © ABB Corporate Research - 46 Accuracy is the correct criterion when predicting the number of field defects for maintenance resource planning Current accuracy evaluation methods are not well-suited for multi-release systems Cross-validation Random data with-holding Release 1Release 2Release 3Release 4 Insight 3 : use forward prediction evaluation

47. © ABB Corporate Research - 47 Accuracy is the correct criterion when predicting the number of field defects for maintenance resource planning Current accuracy evaluation methods are not well-suited for multi-release systems Cross-validation Random data with-holding Release 1Release 2Release 3Release 4 Insight 3 : use forward prediction evaluation

48. © ABB Corporate Research - 48 Accuracy is the correct criterion when predicting the number of field defects for maintenance resource planning Current accuracy evaluation methods are not well-suited for multi-release systems Cross-validation Random data with-holding Release 1Release 2Release 3Release 4 Insight 3 : use forward prediction evaluation

49. © ABB Corporate Research - 49 Accuracy is the correct criterion when predicting the number of field defects for maintenance resource planning Current accuracy evaluation methods are not well-suited for multi-release systems Cross-validation Random data with-holding Release 1Release 2Release 3Release 4 Insight 3 : use forward prediction evaluation

50. © ABB Corporate Research - 50 Accuracy is the correct criterion when predicting the number of field defects for maintenance resource planning Current accuracy evaluation methods are not well-suited for multi-release systems Cross-validation Random data with-holding Only a non-random sub-set is available Predicting for a past release is not the same as predicting for a future release Not realistic! Insight 3 : use forward prediction evaluation

51. © ABB Corporate Research - 51 Accuracy is the correct criterion when predicting the number of field defects for maintenance resource planning Current accuracy evaluation methods are not well-suited for multi-release systems Cross-validation Random data with-holding We use a forward prediction evaluation method Release 1Release 2Release 3Release 4 Insight 3 : use forward prediction evaluation

52. © ABB Corporate Research - 52 Accuracy is the correct criterion when predicting the number of field defects for maintenance resource planning Current accuracy evaluation methods are not well-suited for multi-release systems Cross-validation Random data with-holding We use a forward prediction evaluation method Release 1Release 2Release 3Release 4 Insight 3 : use forward prediction evaluation

53. © ABB Corporate Research - 53 Accuracy is the correct criterion when predicting the number of field defects for maintenance resource planning Current accuracy evaluation methods are not well-suited for multi-release systems Cross-validation Random data with-holding We use a forward prediction evaluation method Release 1Release 2Release 3Release 4 Insight 3 : use forward prediction evaluation

54. © ABB Corporate Research - 54 Field defect prediction ARE Product AR1.0R1.1R1.2Avg Moving Average 1 Release3.0%51.7%19.2%24.6% Linear Regression with Model Selection17.6% Tree Split with 2 Releases3.0%51.7%19.2%24.6% Tree Split with 3 Releases3.0%54.0%19.2%25.4% With cost of field defects, can combine with field defect predictions to allocate initial maintenance resources Need to evaluate if ~24% average error is adequate

55. © ABB Corporate Research - 55 Improvement planning Product A Open Issues Service Pack Product B Open Issues Months Before Next Release Model selection selected predictors

56. © ABB Corporate Research - 56 Improvement planning Select a modeling method based on explicability and quantifiability Product A Open Issues Service Pack Product B Open Issues Months Before Next Release

57. © ABB Corporate Research - 57 Improvement planning Select a modeling method based on explicability and quantifiability Use available information Improved validity More accurate model Justification for better data collection Product A Open Issues Service Pack Product B Open Issues Months Before Next Release

58. © ABB Corporate Research - 58 Improvement planning Product A Open Issues Service Pack Product B Open Issues Months Before Next Release Can delay deployment to conduct more testing Can reduce scope of next release to resolve field defects

59. © ABB Corporate Research - 59 Image from smig.usgs.gov Explicability example: neural networks

60. © ABB Corporate Research - 60 Z1 =Z1 = 1 Input 1 *weight 1 + Input 2 *weight 2 + Input 3 *weight 3 + Input 4 *weight 4 Z1Z1 Explicability example: neural networks

61. © ABB Corporate Research - 61 Output = 1 Z 1 *weight 1 + Z 2 *weight 2 + Z 3 *weight 3 + Z 4 *weight 4 Z1Z1 Z2Z2 Z3Z3 Z4Z4 Z5Z5 Explicability example: neural networks

62. © ABB Corporate Research - 62 Z1Z1 Z2Z2 Z3Z3 Z4Z4 Z5Z5 ? How does input relate to the output? Explicability example: neural networks

63. © ABB Corporate Research - 63 Z1Z1 Z2Z2 Z3Z3 Z4Z4 Z5Z5 Improvement planning and test prioritization both need to attribute effects to predictors ? Explicability example: neural networks

64. © ABB Corporate Research - 64 Z1 =Z1 = 1 Input 1 *weight 1 + Input 2 *weight 2 + Input 3 *weight 3 + Input 4 *weight 4 Z1Z1 Quantifiability example: neural networks

65. © ABB Corporate Research - 65 Z1Z1 Z2Z2 Z3Z3 Z4Z4 Z5Z5 How do the predictors compare? ? Quantifiability example: neural networks

66. © ABB Corporate Research - 66 Z1Z1 Z2Z2 Z3Z3 Z4Z4 Z5Z5 Improvement planning and test prioritization both need to compare importance of predictors ? Quantifiability example: neural networks