Predicting Fault-Prone Modules Based on Metrics Transitions Yoshiki Higo, Kenji Murao, Shinji Kusumoto, Katsuro Inoue {higo,k-murao,kusumoto,inoue}@ist.osaka-u.ac.jp The title of my talk is “predicting fault-prone modules based on metrics transitions”. 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Outline Background Preliminaries Proposal Case Study Conclusion Software Metrics Version Control System Proposal Predict fault-prone modules Case Study Conclusion This is the outline of my talk. Firstly, I talk about the background of my research, and then, I introduce some preliminaries. Next, I explain the proposed method for predicting fault-prone modules, and then, I present the result of a case study conducted on open source software. At last, I conclude my presentation. 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Background It is becoming more and more difficult for developers to devote their energies to all modules of a developing system Larger and more complex Faster time to market It is important to identify modules that hinder software development and maintenance, and we should concentrate on such modules Manual identification requires much costs depending on the size of the target software Firstly, I talk about the background of my research. As you know, Software system is becoming larger and more complex, but the development period is becoming shorter. As a result, it is becoming more and more difficult for developers to devote their energies to all modules of the developing system. So, it is important to identify modules that hinder software development and maintenance, we should concentrate on such modules. However, manual identification requires much costs depending on the size of the target software. Therefore, automatic identification is essential for efficient software development and maintenance. Automatic identification is essential for efficient software development and maintenance 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Preliminaries -Software Metrics- Measures for evaluating various attributes of software There are many software metrics CK metrics suite is one of the most widely used metrics CK metrics suite evaluates complexities of OO systems from Inheritance (DIT, NOC) Coupling between classes (RFC, CBO) Complexity within each class (WMC, LCOM) CK metrics suite is a good indicator to predict fault-prone classes[1] Here, I talk about software metrics. Software metrics are measures for evaluating various attributes of software. At present, there are many software metrics. CK metrics suite is one of the most widely used metrics. CK metrics suite evaluates complexities of object oriented systems from 3 viewpoints, inheritance, coupling between classes, and complexity within each class. It was experimentally reported that CK metrics suite is a good indicator to predict fault-prone classes. [1] V. R. Basili, L. C. Briand, and W. L. Melo. A Validation of Object-Oriented Design Metrics as Quality Indicators. IEEE Transactions on Software Engineering, 22(10):751–761, Oct 1996. 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Preliminaries -Version Control System- Tool for efficiently developing and maintaining software systems with many other developers Every developer gets a copy of the software from the repository (checkout) modifies the copy sends the modified copy to the repository (commit) The repository contains various data Modified code of every commitment Developer names of every commitment Commitment time of every commitment Log messages of every commitment I must say about version control system. Version control system is a tool for efficiently developing and maintaining software systems with many other developers. Every developer gets a copy of the software from the repository of the version control system, and modifies the copy. After that, sends the modified copy to the repository. The repository contains various data. For example, modified code of every commitment, developer names of every commitment, commitment time, and log messages. 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Motivation Software Metrics evaluate the latest (or the past) software product They represent the states of the software at the version How the software evolved is an important attribute of the software Let me talk about the motivation. Software metrics evaluate the latest or the past software product. Software metrics represent the states of the software at the version. However, we think how the software evolved is an important attribute of the software. 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Motivation -example- In the latest version, the complexity of a certain module is high The complexity of the module is stable at high through multiple versions? The complexity is getting higher according to development progress? The complexity is up and down through the development? The stability of metrics is an indicator of maintainability If the complexity is stable, the module may not be problematic If the complexity is unstable, big changes may be added repeatedly For example, in the latest version, the complexity of a certain module is high. In this case, the complexity of the module is stable at high through multiple version? Or, the complexity is getting higher according to development progress? Or, the complexity is up and down through the development? We can’t know how the complexity changed through the development using software metrics. We think that the stability of metrics is an indicator of maintainability. For example, if the complexity is stable, the module may not be problematic. If the complexity is unstable, big changes may be added repeatedly. 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Proposal: Metrics Constancy Metrics Constancy (MC) is proposed for identifying problematic modules MC evaluates the changeability of the metrics of each module MC is calculated using the following statistical tools Entropy Normalized Entropy Quartile Deviation Quartile Dispersion Coefficient Hamming Distance Euclidean Distance Mahalanobis Distance In this presentation, Metrics Constancy, MC, is proposed for identifying problematic modules. MC evaluates the changeability of the metrics of each module. MC is calculated using the following statistical tools. But due to time limitation, I can’t explain all of them. In this presentation, I talk about only Entropy. If you are interested in the others, please refer my paper. 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Entropy An indicator to represent the degree of uncertainty Given that MC is uncertainty of metrics, Entropy can be used as a measure of MC (pi is probability) 1 2 3 c1 c2 c3 c4 4 c5 m1 m2 m3 changes Metric value m1: 5 changes, value 2: 4 times, value 3: 1 time ≒0.72 Entropy is an indicator to represent the degree of uncertainty. Given that MC is uncertainty of metrics, Entropy can be used as a measure of MC. This is the formula for calculating Entropy, and this is an example of metrics transitions. For example, the transition of Metric 1 is 2, 3, 2, ,2, 2. Metric 1 has 5 changes, value 2 is 4 times, and value 3 is 1 time. From this transition, Entropy of metric 1 becomes 0.72. As well as Metric 1, entropies of metrics 2 and 3 become 1.9 and 1.6 respectively. If the metric changed drastically, its Entropy becomes high. m2: 5 changes, value 1,2,3: 1 time, value4: 2 times ≒1.9 m3: 3 changes, value 1,3,4: 1 time ≒1.6 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Calculating MC from Entropy MC of module i is calculated from the following formula MT is a set of used metrics The more unstable the metrics of module i are, the greater MC(i) is MC of module i is calculated from the following formula. The more unstable the metrics of module i are, the greater MC(i) is. 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Procedure for calculating MC STEP1: Retrieves snapshots A snapshot is a set of source files just after at least one source file in the repository was updated by a commitment STEP2: Measures metrics from all of the snapshots It is necessary to select appropriate software metrics fitting for the purpose If the unit of modules is class, class metrics should be used If we focus on the coupling/cohesion of the target software, coupling/cohesion metrics should be used STEP3: Calculates MC Currently, the 7 MCs are calculated Here, I explain the procedure for calculating MC. STEP1 is retrieving snapshots. Here, a snapshot is a set of source files just after at least one source file in the repository was updated by a commitment. STEP2 is measuring metrics from all of the snapshots. In this step, it is necessary to select appropriate software metrics fitting for the purpose. For example, If the unit of modules is class, class metrics should be used. If we focus on the coupling of the target software, coupling metrics should be used. STEP3 is calculating MC. From metrics of multiple snapshots, MC is calculated. Currently, the 7MCs described in the previous slide are calculated. 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Case Study: Outline Target: open source software written in Java FreeMind, JHotDraw, HelpSetMaker Module: class (≒ source file) Used Metrics: CK Metrics, LOC Software FreeMind JHotDraw HelpSetMaker # of Developers 12 24 2 # of snapshots 104 196 260 First commit time 01/Aug/2000 19:56:09 12/Oct/2000 14:57:10 20/Oct/2003 13:05:47 Last commit time 06/Feb/2004 06:04:25 25/Apr/2005 22:35:57 07/Jan/2006 15:08:41 # first source files 67 144 14 # last source files 80 484 36 First total LOC 3,882 12,781 797 Last total LOC 14,076 60,430 9,167 We implemented a software tool, and conducted a case study on open source software, FreeMind, JHotDraw, and HelpSetMaker. In this case study, we used CK metrics and LOC for calculating MCs. 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Case Study: Procedure Divides snapshots into anterior set (1/3) and posterior set (2/3) Calculates MCs from the anterior set Metrics of the last version in the anterior set were used for comparison Identifies bug fixes from the posterior set Commitments including both ``bug’’ and ``fix’’ in their log messages were regarded as bug fixes Sorts the target classes in the order of MCs and raw metrics values Also, bug coverage is calculated based on the orders This is the procedure of the case study. Firstly, we divided snapshots into anterior set and posterior set. Then, we calculated MCs from the anterior set. Also, metrics of the last version in the anterior set were used for comparison. Thirdly, we identified bug fixes from the posterior set. In this case study, commitment including both ``bug’’ and ``fix’’ in there log messages were regarded as bug fixes. Fourth, we sorted the target classes in the order of MCs and raw metrics values. Also, bug coverage is calculated based on the orders. 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Case Study: Results (FreeMind) MCs could identify fault-prone classes more precisely than raw metrics RED: MCs BLUE: raw metrics Bug coverage (%) At top 20% files MCs: 94-100% bugs Raw: 30-80% bugs This graph represents the result of FreeMind. X-axis is sorted classes, and Y-axis is bug coverage. The red is MC, and the blue is raw metrics. You can see that MCs could identify fault-prone classes more precisely than raw metrics. For example, at top 20% files, MCs cover 94 to 100% bugs. On the other hand, Raw metrics cover 30 to 80% bugs. Ranking coverage (%) 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Case Study: Results (Other software) JHotDraw HelpSetMaker These graphs represent the results of JHotDraw and HelpSetMaker. For all of the 3 software, MCs could identify fault-prone classes more precisely than raw metrics. For all of the 3 software, MCs could identify fault-prone classes more precisely than raw metrics 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Case study: different breakpoints In this case study, we used 3 breakpoints 1/4, 1/3, 1/2 The previous graphs are the results in case that anterior set is 1/3 anterior set posterior set First snapshot 1/4 1/3 1/2 last snapshot In this case study, we used 3 breakpoints, one fourth, one third, and one second. The previous graphs are the results in case that anterior set is one third. 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Case study: Results (different breakpoints) FreeMind: anterior ¼, posterior ¾ FreeMind: anterior ½, posterior ½ Left graph’s anterior set is one fourth, and right graph’s anterior set is one second. From these graphs, you can see that MC’s identifications are good on all of the breakpoints. MC’s identifications are good on all of the breakpoints 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Case study: Results (different breakpoints) Top 20% files bug coverage, FreeMind 1/4 1/3 1/2 MCs 94-100% 97-100% Raw metrics 22-72% 30-80% 22-86% MC’s bug coverage is very high for all of the breakpoints Raw metrics are not suited for predicting far future This table represents top 20% files bug coverage of FreeMind. MC’s bug coverage is very high for all of the breakpoints. However, raw metrics are not suited for predicting far future. 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Discussion MCs are good indicators for identifying fault-prone modules as well as CK metrics Calculating MCs required much more time than measuring raw metrics from a single version FreeMind MCs: 95-100% bugs Raw: 30-80% bugs JHotDraw MCs: 44-59% bugs Raw: 10-48% bugs HelpSetMaker MCs: 60-75% bugs Raw: 28-63% bugs From the results of the case study, I can say that MCs are good indicator for identifying fault-prone modules as well as CK metrics. However, Calculating MCs requires much more time than measuring raw metrics from a single version. Raw metrics measurements required less than 1 minutes for all of 3 software. On the other hand, MCs calculations required 28, 40, and 18 minutes respectively. If we calculate MCs from large scale or long history software, it will take much more time. FreeMind MCs: 28 minutes Raw: 1 minute JHotDraw MCs: 40 minutes Raw: 1 minutes HelpSetMaker MCs: 18 minutes Raw: 1 minutes 12/7/2018 Graduate School of Information Science and Technology, Osaka University

Conclusion Metrics Constancy (MC), which is an indicator for predicting problematic modules, was proposed MCs were compared with raw CK metrics The case study showed that MCs could identify fault-prone modules more precisely than raw CK metrics In the future, we are going to conduct more case studies on software written in other programming languages (e.g., C++, C#) compare MCs with other identification methods Let me conclude my presentation. In this presentation, Metrics Constancy, which is an indicator for predicting problematic modules, was proposed. And, MCs were compared with raw CK metrics. The case study showed that MCs could identify fault-prone modules more precisely than raw CK metrics. In the future, we are going to conduct more case studies on software written in other programming languages such as C++ or C#. Also, we are going to compare MCs with other identification methods. 12/7/2018 Graduate School of Information Science and Technology, Osaka University