Doe Bug Prediction Support Human Developers? Findings From a Google Case Study Chris Lewis, ZhongPeng Lin, Caitlin Sadowski, Xiaoyan Zhu, Rong Ou, E.James Whitehead Jr. University of California, Santa Cruz Google Inc. Xi’an Jiaotong University

Motivations  Little empirical data validating that areas predicted to be bug-prone match the expectations of expert developers  Little data showing whether the information provided by bug prediction algorithms leads to modification of developer behavior

Three QuestionsThree Questions  Q1: According to expert opinion, given a collection of bug prediction algorithms, how many bug-prone files do they find and which algorithm is preferred?  Q2: What are the desire characteristics a bug prediction algorithm should have?  Q3: Using the knowledge gained from the other two questions to design a likely algorithm, do developers modify their behavior when presented with bug prediction results?

Algorithm ChoiceAlgorithm Choice  FixCache  If a file is recently changed, it is likely to contain faults  If a file contains a fault, it is likely to contain more faults  Files that change alongside faulty files are more likely to contain faults  LRU  Problem: 10% of files, no severity  Reduce the cache size to 20  Order the cache by duration( total commits )  Rahman

Project ChoiceProject Choice

User StudiesUser Studies  19 interviewees ( A: 9 B: 10 )  3 lists of files  Choices:  Bug-prone  Not bug-prone  No strong feelings either way about  No experience with the file

Results

Results

Results

Results

Q2: desirable characteristicsQ2: desirable characteristics  Actionable( take clear steps that will result in the area no longer being flagged )  Obvious reasoning  Bias towards the new file  Parallelizable  Effectiveness scaling

Time-Weighted Risk Algorithm ( TWR )  Modify Rahman  i: bug-fixing commit  n: number of bug-fixing commit  t i : normalized time of the current bug-fixing commit  w: how strong the decay should be

Experiment  Mondrian ( code review software ) + lint  Duration: 3 months in Google Inc.  Metrics:  The average time a review containing a bug-prone file takes from submission to approval  The average number of comments on a review that contains a bug-prone file

Results

Conclusion  Failure due to TWR  No actionable means of removing the flag

Transfer Defect LearningTransfer Defect Learning Jaechang Nam, Sinno Jialin Pan, Sunghun Kim Department of Computer Science and Engineering The Hong Kong University of Science and Technology, China Institute for Infocomm Research, Singapore

Motivations  Poor cross-project prediction performance  Same feature space  Different data distribution  On the basis of transfer learning, propose transfer defect learning. Modify existing method: TCA( Transfer Component Analysis )  TCA is sensitive to normalization

TCA  TCA tries to learn a transformation to map the original data of source and target domains to a latent space where the difference between domains is small and the data variance after transformation is large.

TCA

TCA+  Choose Normalization options automatically

Rules  5 rules

Process

Experiment ProjectsExperiment Projects  ReLink  Apache HTTP Server  OpenIntents Safe  Zxing  AEEEM  Equinox  Eclipse JDT Core  Apache Lucene  Mylyn  Eclipse PDE UI

TCA with different normalization options

TCA+

Contributions  First to observe improved prediction performance by applying TCA for cross-project defect prediction  Proposed TCA+