1. Predicting Fault-Prone Modules Based on Metrics Transitions
Yoshiki Higo, Kenji Murao, Shinji Kusumoto, Katsuro Inoue
The title of my talk is “predicting fault-prone modules based on metrics transitions”.
12/7/2018
Graduate School of Information Science and Technology, Osaka University

2. Outline Background Preliminaries Proposal Case Study Conclusion
Software Metrics
Version Control System
Proposal
Predict fault-prone modules
Case Study
Conclusion
This is the outline of my talk.
Firstly, I talk about the background of my research, and then, I introduce some preliminaries.
Next, I explain the proposed method for predicting fault-prone modules, and then, I present the result of a case study conducted on open source software.
At last, I conclude my presentation.
12/7/2018
Graduate School of Information Science and Technology, Osaka University

3. Background
It is becoming more and more difficult for developers to devote their energies to all modules of a developing system
Larger and more complex
Faster time to market
It is important to identify modules that hinder software development and maintenance, and we should concentrate on such modules
Manual identification requires much costs depending on the size of the target software
Firstly, I talk about the background of my research.
As you know, Software system is becoming larger and more complex, but the development period is becoming shorter.
As a result, it is becoming more and more difficult for developers to devote their energies to all modules of the developing system.
So, it is important to identify modules that hinder software development and maintenance, we should concentrate on such modules.
However, manual identification requires much costs depending on the size of the target software.
Therefore, automatic identification is essential for efficient software development and maintenance.
Automatic identification is essential for efficient software development and maintenance
12/7/2018
Graduate School of Information Science and Technology, Osaka University

4. Preliminaries -Software Metrics-
Measures for evaluating various attributes of software
There are many software metrics
CK metrics suite is one of the most widely used metrics
CK metrics suite evaluates complexities of OO systems from
Inheritance (DIT, NOC)
Coupling between classes (RFC, CBO)
Complexity within each class (WMC, LCOM)
CK metrics suite is a good indicator to predict fault-prone classes[1]
Here, I talk about software metrics.
Software metrics are measures for evaluating various attributes of software.
At present, there are many software metrics.
CK metrics suite is one of the most widely used metrics.
CK metrics suite evaluates complexities of object oriented systems from 3 viewpoints, inheritance, coupling between classes, and complexity within each class.
It was experimentally reported that CK metrics suite is a good indicator to predict fault-prone classes.
[1] V. R. Basili, L. C. Briand, and W. L. Melo. A Validation of Object-Oriented Design Metrics as Quality Indicators. IEEE Transactions on Software Engineering, 22(10):751–761, Oct 1996.
12/7/2018
Graduate School of Information Science and Technology, Osaka University

5. Preliminaries -Version Control System-
Tool for efficiently developing and maintaining software systems with many other developers
Every developer
gets a copy of the software from the repository (checkout)
modifies the copy
sends the modified copy to the repository (commit)
The repository contains various data
Modified code of every commitment
Developer names of every commitment
Commitment time of every commitment
Log messages of every commitment
I must say about version control system.
Version control system is a tool for efficiently developing and maintaining software systems with many other developers.
Every developer gets a copy of the software from the repository of the version control system, and modifies the copy.
After that, sends the modified copy to the repository.
The repository contains various data.
For example, modified code of every commitment, developer names of every commitment, commitment time, and log messages.
12/7/2018
Graduate School of Information Science and Technology, Osaka University

6. Motivation
Software Metrics evaluate the latest (or the past) software product
They represent the states of the software at the version
How the software evolved is an important attribute of the software
Let me talk about the motivation.
Software metrics evaluate the latest or the past software product.
Software metrics represent the states of the software at the version.
However, we think how the software evolved is an important attribute of the software.
12/7/2018
Graduate School of Information Science and Technology, Osaka University

7. Motivation -example-
In the latest version, the complexity of a certain module is high
The complexity of the module is stable at high through multiple versions?
The complexity is getting higher according to development progress?
The complexity is up and down through the development?
The stability of metrics is an indicator of maintainability
If the complexity is stable, the module may not be problematic
If the complexity is unstable, big changes may be added repeatedly
For example, in the latest version, the complexity of a certain module is high.
In this case, the complexity of the module is stable at high through multiple version?
Or, the complexity is getting higher according to development progress?
Or, the complexity is up and down through the development?
We can’t know how the complexity changed through the development using software metrics.
We think that the stability of metrics is an indicator of maintainability.
For example, if the complexity is stable, the module may not be problematic.
If the complexity is unstable, big changes may be added repeatedly.
12/7/2018
Graduate School of Information Science and Technology, Osaka University

8. Proposal: Metrics Constancy
Metrics Constancy (MC) is proposed for identifying problematic modules
MC evaluates the changeability of the metrics of each module
MC is calculated using the following statistical tools
Entropy
Normalized Entropy
Quartile Deviation
Quartile Dispersion Coefficient
Hamming Distance
Euclidean Distance
Mahalanobis Distance
In this presentation, Metrics Constancy, MC, is proposed for identifying problematic modules.
MC evaluates the changeability of the metrics of each module.
MC is calculated using the following statistical tools.
But due to time limitation, I can’t explain all of them.
In this presentation, I talk about only Entropy.
If you are interested in the others, please refer my paper.
12/7/2018
Graduate School of Information Science and Technology, Osaka University

9. Entropy An indicator to represent the degree of uncertainty
Given that MC is uncertainty of metrics, Entropy can be used as a measure of MC
(pi is probability) 1 2 3 c1 c2 c3 c4 4 c5 m1 m2 m3
changes
Metric value
m1: 5 changes, value 2: 4 times, value 3: 1 time
≒0.72
Entropy is an indicator to represent the degree of uncertainty.
Given that MC is uncertainty of metrics, Entropy can be used as a measure of MC.
This is the formula for calculating Entropy, and this is an example of metrics transitions.
For example, the transition of Metric 1 is 2, 3, 2, ,2, 2.
Metric 1 has 5 changes, value 2 is 4 times, and value 3 is 1 time.
From this transition, Entropy of metric 1 becomes 0.72.
As well as Metric 1, entropies of metrics 2 and 3 become 1.9 and 1.6 respectively.
If the metric changed drastically, its Entropy becomes high.
m2: 5 changes, value 1,2,3: 1 time, value4: 2 times
≒1.9
m3: 3 changes, value 1,3,4: 1 time
≒1.6
12/7/2018
Graduate School of Information Science and Technology, Osaka University

10. Calculating MC from Entropy
MC of module i is calculated from the following formula
MT is a set of used metrics
The more unstable the metrics of module i are, the greater MC(i) is
MC of module i is calculated from the following formula.
The more unstable the metrics of module i are, the greater MC(i) is.
12/7/2018
Graduate School of Information Science and Technology, Osaka University

11. Procedure for calculating MC
STEP1: Retrieves snapshots
A snapshot is a set of source files just after at least one source file in the repository was updated by a commitment
STEP2: Measures metrics from all of the snapshots
It is necessary to select appropriate software metrics fitting for the purpose
If the unit of modules is class, class metrics should be used
If we focus on the coupling/cohesion of the target software, coupling/cohesion metrics should be used
STEP3: Calculates MC
Currently, the 7 MCs are calculated
Here, I explain the procedure for calculating MC.
STEP1 is retrieving snapshots.
Here, a snapshot is a set of source files just after at least one source file in the repository was updated by a commitment.
STEP2 is measuring metrics from all of the snapshots.
In this step, it is necessary to select appropriate software metrics fitting for the purpose.
For example, If the unit of modules is class, class metrics should be used.
If we focus on the coupling of the target software, coupling metrics should be used.
STEP3 is calculating MC.
From metrics of multiple snapshots, MC is calculated.
Currently, the 7MCs described in the previous slide are calculated.
12/7/2018
Graduate School of Information Science and Technology, Osaka University

12. Case Study: Outline Target: open source software written in Java
FreeMind, JHotDraw, HelpSetMaker
Module: class (≒ source file)
Used Metrics: CK Metrics, LOC
Software
FreeMind
JHotDraw
HelpSetMaker
# of Developers 12 24 2
# of snapshots
104
196
260
First commit time
01/Aug/ :56:09
12/Oct/ :57:10
20/Oct/ :05:47
Last commit time
06/Feb/ :04:25
25/Apr/ :35:57
07/Jan/ :08:41
# first source files 67
144 14
# last source files 80
484 36
First total LOC
3,882
12,781
797
Last total LOC
14,076
60,430
9,167
We implemented a software tool, and conducted a case study on open source software, FreeMind, JHotDraw, and HelpSetMaker.
In this case study, we used CK metrics and LOC for calculating MCs.
12/7/2018
Graduate School of Information Science and Technology, Osaka University

13. Case Study: Procedure
Divides snapshots into anterior set (1/3) and posterior set (2/3)
Calculates MCs from the anterior set
Metrics of the last version in the anterior set were used for comparison
Identifies bug fixes from the posterior set
Commitments including both ``bug’’ and ``fix’’ in their log messages were regarded as bug fixes
Sorts the target classes in the order of MCs and raw metrics values
Also, bug coverage is calculated based on the orders
This is the procedure of the case study.
Firstly, we divided snapshots into anterior set and posterior set.
Then, we calculated MCs from the anterior set.
Also, metrics of the last version in the anterior set were used for comparison.
Thirdly, we identified bug fixes from the posterior set.
In this case study, commitment including both ``bug’’ and ``fix’’ in there log messages were regarded as bug fixes.
Fourth, we sorted the target classes in the order of MCs and raw metrics values.
Also, bug coverage is calculated based on the orders.
12/7/2018
Graduate School of Information Science and Technology, Osaka University

14. Case Study: Results (FreeMind)
MCs could identify fault-prone classes more precisely than raw metrics
RED: MCs
BLUE: raw metrics
Bug coverage (%)
At top 20% files
MCs: % bugs
Raw: 30-80% bugs
This graph represents the result of FreeMind.
X-axis is sorted classes, and Y-axis is bug coverage.
The red is MC, and the blue is raw metrics.
You can see that MCs could identify fault-prone classes more precisely than raw metrics.
For example, at top 20% files, MCs cover 94 to 100% bugs.
On the other hand, Raw metrics cover 30 to 80% bugs.
Ranking coverage (%)
12/7/2018
Graduate School of Information Science and Technology, Osaka University

15. Case Study: Results (Other software)
JHotDraw
HelpSetMaker
These graphs represent the results of JHotDraw and HelpSetMaker.
For all of the 3 software, MCs could identify fault-prone classes more precisely than raw metrics.
For all of the 3 software, MCs could identify fault-prone classes more precisely than raw metrics
12/7/2018
Graduate School of Information Science and Technology, Osaka University

16. Case study: different breakpoints
In this case study, we used 3 breakpoints
1/4, 1/3, 1/2
The previous graphs are the results in case that anterior set is 1/3
anterior set
posterior set
First snapshot
1/4
1/3
1/2
last snapshot
In this case study, we used 3 breakpoints, one fourth, one third, and one second.
The previous graphs are the results in case that anterior set is one third.
12/7/2018
Graduate School of Information Science and Technology, Osaka University

17. Case study: Results (different breakpoints)
FreeMind: anterior ¼, posterior ¾
FreeMind: anterior ½, posterior ½
Left graph’s anterior set is one fourth, and right graph’s anterior set is one second.
From these graphs, you can see that MC’s identifications are good on all of the breakpoints.
MC’s identifications are good on all of the breakpoints
12/7/2018
Graduate School of Information Science and Technology, Osaka University

18. Case study: Results (different breakpoints)
Top 20% files bug coverage, FreeMind
1/4
1/3
1/2
MCs
94-100%
97-100%
Raw metrics
22-72%
30-80%
22-86%
MC’s bug coverage is very high for all of the breakpoints
Raw metrics are not suited for predicting far future
This table represents top 20% files bug coverage of FreeMind.
MC’s bug coverage is very high for all of the breakpoints.
However, raw metrics are not suited for predicting far future.
12/7/2018
Graduate School of Information Science and Technology, Osaka University

19. Discussion
MCs are good indicators for identifying fault-prone modules as well as CK metrics
Calculating MCs required much more time than measuring raw metrics from a single version
FreeMind
MCs: % bugs
Raw: 30-80% bugs
JHotDraw
MCs: 44-59% bugs
Raw: 10-48% bugs
HelpSetMaker
MCs: 60-75% bugs
Raw: 28-63% bugs
From the results of the case study, I can say that MCs are good indicator for identifying fault-prone modules as well as CK metrics.
However, Calculating MCs requires much more time than measuring raw metrics from a single version.
Raw metrics measurements required less than 1 minutes for all of 3 software.
On the other hand, MCs calculations required 28, 40, and 18 minutes respectively.
If we calculate MCs from large scale or long history software, it will take much more time.
FreeMind
MCs: 28 minutes
Raw: 1 minute
JHotDraw
MCs: 40 minutes
Raw: 1 minutes
HelpSetMaker
MCs: 18 minutes
Raw: 1 minutes
12/7/2018
Graduate School of Information Science and Technology, Osaka University

20. Conclusion
Metrics Constancy (MC), which is an indicator for predicting problematic modules, was proposed
MCs were compared with raw CK metrics
The case study showed that MCs could identify fault-prone modules more precisely than raw CK metrics
In the future, we are going to
conduct more case studies on software written in other programming languages (e.g., C++, C#)
compare MCs with other identification methods
Let me conclude my presentation.
In this presentation, Metrics Constancy, which is an indicator for predicting problematic modules, was proposed.
And, MCs were compared with raw CK metrics.
The case study showed that MCs could identify fault-prone modules more precisely than raw CK metrics.
In the future, we are going to conduct more case studies on software written in other programming languages such as C++ or C#.
Also, we are going to compare MCs with other identification methods.
12/7/2018
Graduate School of Information Science and Technology, Osaka University