1. A UML Approximation of a Subset of the CK Metrics and Their Ability to Predict Faulty Classes
CAMARGO CRUZ Ana Erika Advisor: Prof. OCHIMIZU Koichiro July 28th, 2011
Good Afternoon every one, thank you for being here.
My name is Ana Erika Camargo Cruz, and today I am presenting my Thesis on “A UML …”, which has been done under the supervision of Prof. OCHIMIZU Koichiro here present in the audience

2. Achievements Major Achievements Journal IEICE, ICSE, ESEM, 4WCSQ
UML metrics → to approximate the code Chidamber-Kemerer (CK) metrics.
Fault prediction before coding → univariate logistic regression modeling
Data Normalization → to reduce the systematic error, to improve fault prediction.
Minor Achievements
A methodology to report software implementation progress (Technical Report)
Mathematical definition of our UML metrics (Not published)
Most of my research work has been already published, and mainly concern three topics:
First, We propose a set of UML metrics …
Second,
Third,
Some minor achievements concern:
A methodo …
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

3. JAIST – CAMARGO CRUZ Ana Erika
Agenda
Motivation
Research Outline
Background
Our Approach
Metrics Definition
Code CK metrics vs UML metrics
Normalization
Fault Prediction
Fault Prediction Multivariate Analysis
Related Work
Conclusions
Future Work
These are the different topics to be covered in this presentation during the following 50 min. Let me stat with my presentation …
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

4. JAIST – CAMARGO CRUZ Ana Erika
1. Motivation
A basic software development process
How to make earlier detections of low-quality elements within the life-cycle of the software?
Analysis
Design
Implementation
The greater the number of defects found,
the greater the amount of resources needed to fix them back in the process
Testing
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

5. 2. Research Outline STEP 2 STEP 1 Analysis D C Design B Implementation
CODE BASED PREDICTION MODEL
UML
Design Complexity Metrics
Faulty Code
Design
STEP 1 B
Approximation
CODE BASED PREDICTION MODEL
CODE
Design Complexity Metrics
Faulty Code
Implementation
(*) Good Predictors:
The CK Metrics A
Existing Research Works
Test
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

6. 3. Background : The CK Metrics
What are they?
CK and Fault proneness of Code
The CK and UML artifacts
Understanding the CK Metrics
Conclusion
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

7. JAIST – CAMARGO CRUZ Ana Erika
3.1.What are the CK Metrics ?
To measure complexity of the design
Measured from the code
Related to fault-proneness, maintenance effort, re-work effort for re-using classes, etc.
Six metrics:
DIT (Depth of Inheritance)
NOC (Number of Children)
WMC (Weighted Method per Class)
RFC (Response For Class)
CBO (Coupling Between Objects)
LCOM (Lack of Cohesion in Methods)
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

8. 3.2. The CK Metrics and Fault Prediction
Study
Used Metrics
Good Predictors
Basili et al. (1996) CK
RFC, DIT, NOC
→ CBO → WMC
Briand et al. (2000)
49 metrics of coupling, cohesion
and inheritance metrics (CK
among them)
RFC and other coupling measures → NOC
and DIT and other measures of inheritance
Kanmani et al. (2004)
CK among 64 metrics of inheritance, size, cohesion and coupling
No details
Nachiappan et al. (2005)
STREW-J suite: 3 CK metrics
and others
Olague et al. (2007)
CK, MOOD, QMOOD
CK: RFC, WMC and QMOOD: NOM, CIS
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

9. JAIST – CAMARGO CRUZ Ana Erika
3.3. The CK Metrics and UML
Research works found in the literature:
Focus on UML class diagrams [Baroni, Mc Quillan et al., 2003, 2006]:
CBO, RFC and LCOM measurement is not straightforwardly.
HINT: Behavioral diagrams could provide more precise measures of the CBO and RFC metrics, but they do not indicate how
Require very well detailed design information, such as that every non-abstract operation must be represented in one activity diagram and instance variable usages [Tang et al.,2002].
Mei-Huei Tang and Mei-Hwa Chen. Measuring oo design metrics from UML. In UML
’02: Proc. of the 5th Intl. Conference on The Unified Modeling Language, pages 368–
382, London, UK, Springer-Verlag.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

10. 3.4. Understanding the CK Metrics [2/4]
WMC is the sum of the complexities of every method of the given class. If the complexities are considered unity then WMC is equal to NOM (Number of Methods of a class).
class A{
private C c;
private int var;
public int m1A(){
B b;
b.m1B();
b.m2B(); }
public int m2A(){
return (var+10);
public void m3A(){
var = var*c.m2C;
WMC(A) = 3 methods
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

11. 3.4. Understanding the CK Metrics [3/4]
RFC is measured as the number of methods of a given class, PLUS the number of different methods of other classes directly called by the given class.
class A{
private C c;
private int var;
public int m1A(){
B b;
b.m1B();
b.m2B(); }
public int m2A(){
return (var+10);
public void m3A(){
var = var*c.m2C;
RFC(A) = = 6 methods
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

12. 3.4. Understanding the CK Metrics [1/4]
CBO is the number of other classes to which a class is coupled.
class A{
private C c;
private int var;
public int m1A(){
B b;
b.m1B();
b.m2B(); }
public int m2A(){
return (var+10);
public void m3A(){
var = var*c.m2C;
CBO(A) = 2 classes
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

13. 3.4. Understanding the CK Metrics [4/4]
LCOM measures the dissimilarity of methods in a class by instance variable or attributes.
class A{
private C c;
private int var;
public int m1A(){
B b;
b.m1B();
b.m2B(); }
public int m2A(){
return (var+10);
public void m3A(){
var = var*c.m2C;
LCOM(A) = |P| − |Q|; IF |P| >= |Q|
0; IF |P| < |Q|
|P| number of pairs of methods with NO variable in COMMON
|Q| number of pairs of methods with AT LEAST ONE variable in COMMON
m1A m2A = {}
m1Am3A = {}
m2Am3A= {var}
|P| = 2
|Q| = 1
LCOM(A) = 2 − 1 = 1
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

14. JAIST – CAMARGO CRUZ Ana Erika
3.5. Conclusion
CBO, RFC and LCOM metrics cannot be obtained precisely from UML class diagrams.
Hypothesis: Because
CBO and RFC are coupling measures,
and UML communication diagrams describe how objects of classes interact to a method instance level,
UML communication diagrams can provide good approximations of such metrics
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

15. STEP 2 STEP 1 Analysis D C E Design B B1 How? Implementation A Test
CODE BASED PREDICTION MODEL
UML
Design Complexity Metrics
Faulty Code
Design
STEP 1 B
Approximation B1
How?
CODE
Design Complexity Metrics
Implementation
(*) Good Predictors:
The CK Metrics A
Test
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

16. JAIST – CAMARGO CRUZ Ana ErikaB1
UML CK Metrics How to?
Goal: Analyze UML Communication Diagrams for the purpose of obtaining an approximation of the CK RFC, CBO and WMC metrics with respect to their objects interactions .
Q3. How can we approximate the CBO metric?
Q1. How can we approximate the WMC metric?
Q2. How can we approximate the RFC metric?
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

17. UML CK Metrics[4/11] Direct and Indirect MetricsB1
UML CK Metrics[4/11] Direct and Indirect Metrics
In order to provide a UML approximation of the CBO, RFC and WMC CK metrics:
Set of Direct Metrics for Objects
Set of Indirect Metrics for Classes
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

18. UML CK Metrics Direct Metrics[1/3]B1
UML CK Metrics Direct Metrics[1/3]
x being an object of a class X and x acting as a RECEIVER:
[D1] NDRCMsgs(x)
Number of Different
Received Call Messages by x
Example:
NDRCMsgs(: Calculator) = 1
NDRCMsgs(: Display) = 1
The objects depicted in a communication diagram can play two basic roles:
The can be acting as a SENDER and/or as a RECEIVER.
Looking at the objects acting a receivers, we measure our first direct metric NDRCMsgs
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

19. UML CK Metrics Direct Metrics[2/3]B1
UML CK Metrics Direct Metrics[2/3]
x being an object of a class X and x acting as a SENDER:
[D2] NDSCMsgs(x)
Number of the Different
Sent Call Messages by x
Example:
NDSCMsgs(: Calculator) = 2
NDSCMsgs(: Display) = 0
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

20. UML CK Metrics Direct Metrics[3/3]B1
UML CK Metrics Direct Metrics[3/3]
X being an object of a class X and x acting as a SENDER:
[D3] NDRO(x)
Number Different Receiving Objects of a Message from x
Example:
NDRO(: Calculator) = 2
NDRO(: Display) = 0
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

21. UML CK Metrics Indirect Metrics: UWMCB1
UML CK Metrics Indirect Metrics: UWMC
class Calculator(){
private Display display = new Display();
public Multiply(int a, int b){
Multiplier multiplier = new Multiplier();
result = multiplier.Multiply(a,b);
display.display(this.result); }
public Sum(int a, int b){
Adder adder = new Adder();
this.result = adder.Sum(a,b)
Diagram
[D1] NDRCMsgs(:Calculator) A
1 = |{A1}| B
1 = |{B1}|
Total
2 = |{A1, B1}| = NDIM(Calculator)
Number of Different Instantiated Methods = NDIM(X)
Now, Let us say we would like to approximate WMC using our direct metrics for objects.
For example, for class Calculator WMC is equal two, because it has two methods …
To approximate this metric we define a NEW indirect metric call NDIM (X), which uses our first metric defined as NDRSCM …
We would have to measures thise metrics for ALL OBJECT OF CLASS CALCULATOR.
WMC(Calculator) = 2
UWMC (X) = NDIM(X) = 2
11/14/2018

22. UML CK Metrics Indirect Metrics: URFCB1
UML CK Metrics Indirect Metrics: URFC
class Calculator(){
private Display display = new Display();
public Multiply(int a, int b){
Multiplier multiplier = new Multiplier();
result = multiplier.Multiply(a,b);
display.display(this.result); }
public Sum(int a, int b){
Adder adder = new Adder();
this.result = adder.Sum(a,b)
Diagram
[D2] NDSCMsgs(:Calculator) A
2 = |{A4, A2 = A3 = A6}| B
2 = |{B4, B2 = B3 = B5}|
Total
3 = |{A4, A2 = B2, B4}| = NDIMO(Calculator)
Number of Different Instantiated Methods of Other classes = NDIMO (X)
RFC(Calculator) = = 5
URFC(X) = UWMC(X) + NDIMO(X)
11/14/2018

23. UML CK Metrics Indirect Metrics: UCBO
class Calculator(){
private Display display = new Display();
public Multiply(int a, int b){
Multiplier multiplier = new Multiplier();
result = multiplier.Multiply(a,b);
display.display(this.result); }
public Sum(int a, int b){
Adder adder = new Adder();
this.result = adder.Sum(a,b)
Diagram
[D3] NDRO(:Calculator) A
2 = |{: Display, : Multipier}| B
2 = |{: Display, : Adder}|
Total
3 = |{: Display, : Multiplier, : Adder}| =
NDRC(Calculator)
Number of Different Receiving Classes
CBO(Calculator) = 3
UCBO(X) = NDRC(X)
11/14/2018

24. JAIST – CAMARGO CRUZ Ana ErikaB1
UML CK Metrics Summary
Goal: Analyze UML Communication Diagrams for the purpose of obtaining an approximation of the CK RFC, CBO and WMC metrics with respect to their objects interactions.
Q1. How can we approximate the CK-WMC metric?
Q2. How can we approximate the CK-RFC metric?
Q3. How can we approximate the CK-CBO metric?
UWMC(X) = NDIM(X)
URFC(X) = NDIM(X) +
NDIMO(X)
UCBO(X) = NDRC(X)
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

25. STEP 2 STEP 1 Analysis D C E Design B B1 How? B2 Implementation
CODE BASED PREDICTION MODEL
UML
Design Complexity Metrics
Faulty Code
Design
STEP 1 B
Approximation B1
How?
CODE
Design Complexity Metrics B2
Implementation
How Good?
(*) Good Predictors:
The CK Metrics A
Test
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

26. UML CK Metrics Evaluation [1/11]B2
UML CK Metrics Evaluation [1/11]
Process:
1. Collection of code measures of 4 different small-size software projects written in Java by students of JAIST. They developed their programs based on the UML diagrams provided in Gomaa’s book*
• An e-commerce system (ECS).
• Two banking system (BNA, BNB).
• A cruise control and monitoring system (CRS).
2. Measurement of the corresponding UML metrics from their UML communication diagrams provided in Gomaa’s book*
3. Comparison and Evaluation: Accuracy, Precision and Linearity.
(*) Hassan Gomaa. Designing Concurrent, Distributed, and Real-Time Applications with UML. Addison
Wesley-Object Technology Series Editors, Boston, MA, USA, 2000.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

27. UML CK Metrics Evaluation[3/11] Measures for EvaluationB2
UML CK Metrics Evaluation[3/11] Measures for Evaluation
Accuracy: Degree of conformity with the true measure.
Relative Error = |Code Metric– UML Approximation |
|Code Metric|
Precision: It refers to the degree of repeatability of measurements. It does not require to know the correct or true value → standard deviation.
Linear Relationship between our UML approximations and code CK metrics → Pearson correlation coefficient.
Precision: It refers to the degree of repeatability of measurements
under unchanged conditions showing the same results.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

28. UML CK Metrics Evaluation [2/11] Accuracy and PrecisionB2
UML CK Metrics Evaluation [2/11] Accuracy and Precision
Accuracy(+)
Probability
Density
Systematic
Error (-)
Error approximation =
Systematic Error +
Random Error
Systematic Error: Predictable, Proportional to the True Value
Random Error : Unpredictable, Inconsistent measurements
Assuming that our Relative Errors have a Normal distribution, we would have a curve like this.
The distance between the mean of our relative errors and
a reference value 0 (desirable) would represent
how accurate is our measurement methodology.
As for precision, it can be observed in
the closeness of the measurements to each other.
The closer the more precise our measurement method is.
_________________________
The theory of measurements considers that all measurements have some error approximation, which is a sum of a systematic error and random error.
Systematic Error, is considered to be predictable, and typically
constant or proportional to the true value.
In contrast Random Errors are unpredictable and lead to inconsistent measurements.
Reference Value
Relative Error = 0
Mean
Precision (+)
Random Error (-)
Systematic Error (-) → Accuracy (+)
Random Error (-) → Precision (+)
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

29. UML CK metrics Evaluation[4/11] UCBO
Accuracy
Mean(RE)
0.5
Precision 
SD(RE)
0.36
Linearity
0.8
Accurate but not Precise
Strong linear relationship
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

30. UML CK metrics Evaluation[5/11] URFCB2
UML CK metrics Evaluation[5/11] URFC
Accuracy  Mean(RE)
0.7
Precision
SD(RE)
0.15
Linearity
0.86
Precise but not Accurate
Strong linear relationship
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

31. UML CK metrics Evaluation[6/11] UWMCB2
UML CK metrics Evaluation[6/11] UWMC
Accuracy 
Mean(RE)
0.65
Precision 
SD(RE)
0.4
Linearity 
0.46
Neither Accurate nor Precise
Weak linear relationship
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

32. UML CK metrics Evaluation[7/11] Comparison UCBO, URFC and UWMC
Probability Density
Accuracy:
UCBO <UWMC < URFC
Precision:
URFC<UWMC<
UCBO
URFC
UWMC
UCBO
0.5
0.7
11/14/2018
Average Relative Errors
JAIST – CAMARGO CRUZ Ana Erika

33. The Gap [1/2] Proportional DifferencesB2
The Gap [1/2] Proportional Differences
Metric
Same level of detail in the design and
1 Developer or
Developers with the same programming style.
Classes
Code Measures
Systematic Type of Error
UML Measures
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

34. The Gap [2/2] Non-Proportional DifferencesB2
The Gap [2/2] Non-Proportional Differences
Metric
Classes dealing with other requirements different from the ones specific to the domain problem, such as:
GUIS, DB connectivity, IPC communication, etc.
Random Type of Error
Classes
Code Measures
Systematic Type of Error
UML Measures
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

35. UML CK metrics Evaluation[8/11] Sensitivity of UML MetricsB2
UML CK metrics Evaluation[8/11] Sensitivity of UML Metrics
Same Design , Different Implementations
BNA entangles GUIs’ functionalities with those specific to the domain problem
BNB was implemented different from the original design
Use of Interfaces
Decoupling GUIs’
BNB-00 , no considered in the design, Implements
Interfaces BNB-05 and BNB-04
GUI’s Functionalities
BNA-04 and BNA-05
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

36. UML CK metrics Evaluation[10/11] Sensitivity of UML MetricsB2
UML CK metrics Evaluation[10/11] Sensitivity of UML Metrics
Same Design , Different Implementations
UCBO
[classes]
BNA
BNB
Average(RE)
0.5
0.6
SD(RE)
0.37
0.33
Linearity
0.85
0.87
Different Implementations → Similar Evaluation Results
Different Implementations →
Similar Correlation
URFC
[methods]
BNA
BNB
Average(RE)
0.85
0.6
SD(RE)
0.1
0.19
Linearity
0.7
0.8
UWMC
[methods]
BNA
BNB
Average(RE)
0.69
0.56
SD(RE)
0.15
0.41
Linearity
0.71
Different Implementations →
Similar Precision
Similar Correlation
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

37. UML CK metrics Evaluation[11/11] ConclusionsB2
UML CK metrics Evaluation[11/11] Conclusions
UCBO is the most accurate
CBO Unity [Class]
URFC is the most precise
RFC Unity [Method]
UWMC less accurate than UCBO and less precise than URFC
WMC Unity [Method]
UCBO and URFC show to have strong linear relationship to its code metric
URFC and UCBO can approximate better their respective code metrics.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

38. STEP 2 STEP 1 Analysis D C E Design B B1 How? C1 B2 Implementation
CODE BASED PREDICTION MODEL
UML
Design Complexity Metrics
Faulty Code
Design
STEP 1 B
Approximation B1
How?
CODE
Design Complexity Metrics C1 B2
Implementation
How Good?
(*) Good Predictors:
The CK Metrics A
Q1. How to build a code-based prediction
model?
Q2. Why not a UML-based prediction model?
Q3 . How to enable our UML metrics to
be used with a code-based prediction model?
Test
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

39. Q1. How to build a prediction model?
Different techniques could be used to model data and predict faulty code: linear regression, Artificial Neural Networks, etc.
Logistic Regression is preferred because: No assumptions on the distribution of variables are made.
Use: When we have one dependent variable (y) with two values e.g. Most Faulty (MF) / Least Faulty (LF), 1/0; and one or more independent variables (xs).
Goal: To predict the probability of getting a particular value of y (e.g. A Most Faulty class), given xs variables, through a logit model.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

40. Q1. How to build a prediction model?x
P(MF)
Univariate Model: only one independent variable “x”
Multivariate Model: more than one independent variable “x1, x2, …”
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

41. Q2. Why NOT a UML-Based Prediction Model?
Our previous work → Univariate UML-Based Prediction Models using JAIST projects
Multivariate Analysis → Larger amount of data is required (~10 data points per independent variable)
JAIST projects have a maximum number of 16 designed classes (BNS) → cannot be used for Multivariate Analysis
P(MF) x
x1, x2, … xn
design-complexity
metrics
UML-based
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

42. Q3. How to enable our UML metrics be used with code-based models?
P (MF)
LEAST
FAULTY (LF)
MOST
FAULTY (MF)
Smaller Size SW project 1
Prediction Model derived from the code of Project “A” OR
UML Measures of Project “A”
0.5 10
X = Number of Methods 20
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

43. Solution: NormalizationC1
Solution: Normalization
In order to apply efficiently the same prediction model across projects and
to use our UML approximations using a code-based prediction model.
Application of Data Normalization:
A common scale
Reduction of the systematic error of our approximations
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

44. STEP 2 STEP 1 Analysis D C E Design C1
CODE BASED PREDICTION MODEL
UML
Design Complexity Metrics
Faulty Code
Design
STEP 1
Approximation C1
Q1. How to build a code-based prediction
model?
CODE
Design Complexity Metrics
Q2. Why not a UML-based prediction model?
Implementation
(*) Good Predictors:
The CK Metrics A
Q3 . How to enable our UML metrics to
be used with a code-based prediction model? C2
Data Normalization: 2 candidates
Test
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

45. Preparation for Normalization [1/7]C2
Preparation for Normalization [1/7]
Further Data Exploration:
In addition to the JAIST projects, we analyzed 13 packages of the Mylyn from Eclipse (MYL) → used for building our code-based prediction models
A total of 37 classes from the JAIST projects, and 602 classes from the MYL project are used for this analysis.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

46. Preparation for Normalization [2/7] BoxplotC2
Preparation for Normalization [2/7] Boxplot
Lower Fourth = FL (25%)
Upper Fourth = FU (75%)
Fourth-spread
dF = FU − FL
Outliers cutoffs:
FL − 1.5dF and
FU + 1.5dF
OUTLIERS produces biased regression estimates and reduces the predictive power of regression models.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

47. Preparation for Normalization [3/7] Results of Boxploting for CBO
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

48. JAIST – CAMARGO CRUZ Ana Erika
Preparation for Normalization [4/7] Results of Boxploting for RFC and WMC
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

49. Preparation for Normalization [5/7] General ObservationsC2
Preparation for Normalization [5/7] General Observations
As the size of the dataset increases, the number of outliers do too.
Most of the data are positively skewed.
Tendency for data to spread as the level (Median) increases.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

50. Preparation for Normalization [6/7] Spread VS LevelC2
Preparation for Normalization [6/7] Spread VS Level
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

51. Preparation for Normalization[7/7]C2
Preparation for Normalization[7/7]
Can a prediction model built with MYL-P13 applied to BNA-COD or BNA-UML ?
Can a prediction model built with MYL-P13 be applied to BNA-COD or BNA-UML ?
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

52. Normalization [1/5] Candidate 1
Log Transformations:
Especially useful to reduce the relationship between spread and level.
Elimination of Outliers
Improve Normality
x’ is the new normalized
x raw value
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

53. Normalization [2/5] Candidate 2
Linear Scaling to Unit Variance:
To acquire a common Scale [0..1]
x’ is the new normalized value [0..1]
x raw value
x media
s standard deviation
It is based in the widely used z-score transformation, with transformed
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

54. Normalization [3/5] Candidate 2
Approach 2: Linear Scaling to Unit Variance
An additional l shift and re-scaling guarantees …
Guarantees 99% of the
normalized data fall [0..1]
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

55. Normalization [4/5] Comparison: Candidate 1 vs Candidate 2
LOG
Linear Scaling to Unit Variance
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

56. Normalization [5/5] Conclusion: Candidate 1 vs Candidate 2
UCBO
LOG(UCBO +1)
LS(UCBO)
Accuracy - Mean(RE)
0.5
0.35
0.17
Precision - SD(RE)
0.36
0.32
0.11
Linear Scaling to Unit Variance
performed better than LOG transformations
The systematic error was reduced.
URFC
LOG(URFC+1)
LS(URFC)
Accuracy -Mean(RE)
0.7
0.55
0.19
Precision - SD(RE)
0.17
0.14
UWMC
LOG(UWMC+1)
LS(UWMC)
Accuracy -Mean(RE)
0.65
0.38
0.29
Precision - SD(RE)
0.4
0.2
0.28
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

57. STEP 2 STEP 1 D1 Analysis C D Design B C1
CODE BASED PREDICTION MODEL
UML
Design Complexity Metrics
Faulty Code
Design
STEP 1 B
Approximation
CODE
Design Complexity Metrics C1
Q1. How to build a prediction model?
Implementation
(*) Good Predictors:
The CK Metrics A
Q2. Why a code-based prediction model?
Q3 . How to enable our UML metrics to
be used with these models? C2
Data Normalization: 2 candidates
Test
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

58. Construction of Prediction Models[1/10]
We use the code of 13 packages of the MYL system. To construct several univariate logistic models to predict Most Faulty classes:
P(y=1) x
design-complexity
metric
P(MF)
Code-based
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

59. Construction of Prediction Models[2/10]
Procedure:
Elimination of Outliers
Normalization using Linear Scaling to Unit Variance.
Construction of 3 prediction models (RFC, CBO, WMC): For every MYL package
Selection of the best 3 resulting models.
Test of the 3 best models using the 12 remained packages of the MYL and the 3 JAIST projects.
Other 3 models were build using raw data for comparison.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

60. Construction of Prediction Models[3/10] Evaluation Parameters
To evaluate how well our prediction models can discriminate between MF and LF classes, we use the following indicators:
1. Correctness: It is the percentage of classes correctly classified.
2. Specificity: It is the percentage of LEAST FAULTY (LF) classes correctly classified.
3. Sensitivity: It is the percentage of MOST FAULTY (MF) classes correctly classified.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

61. Construction of Prediction Models[4/10] CBO Prediction Model
CBO Prediction Model: Normalized Data VS Raw Data
The best prediction model was the one modeling the data of package M-11
Correctness: %of classes correctly classified.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

62. Construction of Prediction Models[5/10] CBO Prediction Model
CBO Prediction Model: Normalized Data VS Raw Data
Sensitivity: % of MF classes correctly classified
Specificity: % of LF classes correctly classified
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

63. Construction of Prediction Models[6/10] RFC Prediction Model
RFC Prediction Model: Normalized Data VS Raw Data
The best prediction model was the one modeling the data of package M-12
Correctness: %of classes correctly classified.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

64. Construction of Prediction Models[7/10] RFC Prediction Model
RFC Prediction Model: Normalized Data VS Raw Data
Sensitivity: % of MF classes correctly classified
Specificity: % of LF classes correctly classified
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

65. Construction of Prediction Models[8/10] WMC Prediction Model
WMC Prediction Model: Normalized Data VS Raw Data
The best prediction model was the one modeling the data of package M-02
Correctness: %of classes correctly classified.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

66. Construction of Prediction Models[9/10] WMC Prediction Model
WMC Prediction Model: Normalized Data VS Raw Data
Sensitivity: % of MF classes correctly classified
Specificity: % of LF classes correctly classified
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

67. Construction of Prediction Models[10/10] Summary
Comparing the 16 datasets, using only code measures, we can say that the normalization procedure:
Greatly improved the results of the RFC model,
Benefits less to the CBO model,
Almost no improvement was obtained with the WMC model. Although, the NWMC model can discriminate MF classes better than the WMC model.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

68. STEP 2 STEP 1 D1 D2 C Analysis D Design B Implementation A Test UML
Design Complexity Metrics
CODE BASED PREDICTION MODEL
Faulty Code
Design
STEP 1
Approximation B
CODE
Design Complexity Metrics
Implementation
(*) Good Predictors:
The CK Metrics A
Test
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

69. UML Metrics Evaluation as Predictors of Faulty code
Evaluation using the JAIST Projects
P(y=1) x
x_code
P (MF class)
Comparison AND
Evaluation
(Using Normalized Data)
Code-based
P(y=1) x
x_UML
P (MF class)
Code-based
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

70. UML Metrics Evaluation as Predictors of Faulty code
Using normalized measures, we applied the code-based prediction models (from the MYL) to predict faulty classes in our JAIST projects → code metrics VS UML metrics.
Project
Correctness
Specificity
Sensitivity
NCBO
UML- NCBO
UML-NCBO
ECS
80%
75%
100%
83%
67%
CRS
88%
71%
33%
BNS
89%
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

71. UML Metrics Evaluation as Predictors of Faulty code
Project
Correctness
Specificity
Sensitivity
NRFC
UML- NRFC
UML-NRFC
UML-NURFC
ECS
80%
100%
67%
CRS
57%
64% 0%
25%
BNS
33%
50%
75%
Project
Correctness
Specificity
Sensitivity
NWMC
UML- NWMC
UML-NWMC
NRFC
ECS
55%
45%
25%
71%
57%
CRS
56%
38%
36%
50%
BNS
78%
44%
67%
33%
100%
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

72. UML Metrics Evaluation as Predictors of Faulty code Conclusion
Using normalized measures, the results of:
URFC → equal and sometimes better than those obtained by RFC.
UCBO → nearly equal results to those obtained by CBO.
UWMC → did not perform better than WMC.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

73. STEP 2 STEP 1 D1 D2 C Analysis D Design B Implementation A Test UML
Design Complexity Metrics
CODE BASED PREDICTION MODEL
Faulty Code
Design
STEP 1
Approximation B
CODE
Design Complexity Metrics
Implementation
(*) Good Predictors:
The CK Metrics A
Test
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

74. 5. Fault Prediction Multivariate Analysis [1/2]
Multivariate prediction models require uncorrelated variables (Linear or Logistic Regression)
MYL-P11 Data: Correlation Coefficients
Other studies findings conform our results, suggesting high correlation not only among the CK metrics, but also among other sets of design complexity metrics.
RFC
CBO
WMC NF 1
0.9
0.8
0.7
Number of Faults (NF)
0.5
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

75. 5. Fault Prediction Multivariate Analysis [2/2]NF
RFC
Shared Variance:
Variation amount of two variables that tend to vary together
CBO NF
Multiple Correlation coefficient
A measure of the fit of a multiple linear regression model → [0,1] NF
RFC
CBO
CBO NF
RFC
Shared variance :
* It is represented by the square of their correlation coefficient, r2.
* An easy way to visualize this is with a Venn diagram,
* For example, the amount of shared variance between CBO and NF is represented by their overlap of variation r2.
The multiple coefficient of correlation, R2,
is a measure of the fit of a multiple linear regression model.
It falls somewhere between zero and one;
1: indicates that all data points fall exactly on a line in multidimensional space and
0:no relationship at all between the independent variables collectively and the dependent variable.
The shared variance between CBO and NF likely contains the one between RFC and NF.
RFC → poor significance
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

76. JAIST – CAMARGO CRUZ Ana Erika
6. Related Work [1/3] ”Assessing UML design metrics for predicting fault-prone classes in a Java system” Nugroho et al., Proceedings of the MSR 2010, Cape Town, South Africa, 2010.
5 UML metrics (class and sequence diagrams) → to predict faulty classes.
A significant industrial Java system was used to build and test 3 logistic regression models: a UML-based model, a code-based model and an hybrid model
2 of their UML metrics yields a slightly better prediction accuracy than using SLOC metric.
Best Model: Their hybrid model with 78% accuracy.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

77. JAIST – CAMARGO CRUZ Ana Erika
6. Related Work [2/3] ” Quality prediction model for object-oriented software using UML metrics” In Proc. of the 4th World Congress for Softw.Quality, Bethesda, Maryland, USA, ASQ.
UML class and collaboration diagrams were used to approximate CBO and RFC CK metrics and three metrics of the QMOOD set.
2 JAIST projects were used to build and test several UML-based univariate prediction models.
The UML-based prediction models were compared with code-based prediction models:
Code-based models generally predict better than UML-based models
URFC model (84.61%) ~ RFC model (84.61%)
UCBO models (69.20%) ~ CBO model (92.3%)
UCIS model (90.9%) → from class diagrams (public number of methods) → (similar to our current proposed UWMC metric)
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

78. JAIST – CAMARGO CRUZ Ana Erika
6. Related Work [3/3] ” Quality prediction model for object-oriented software using UML metrics” In Proc. of the 4th World Congress for Softw.Quality, Bethesda, Maryland, USA, ASQ.
No normalization method was used
Prediction accuracy of UML-based prediction models 69% %
CONCLUSION:
UML metrics can be acceptable predictors of fault-proneness of code.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

79. JAIST – CAMARGO CRUZ Ana Erika
7. Conclusions
A simple approach to approximate CBO, RFC and WMC CK metrics using UML communication diagrams.
Our UML RFC and CBO metrics can predict faulty code almost with the same accuracy as their respective code metrics.
The normalization procedure was significantly efficient: for enabling our UML metrics to predict faulty code for improving the prediction ability of logistic models across different packages and projects.
Both the UML WMC and the code WMC metrics showed a poor fault-proneness prediction ability.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

80. JAIST – CAMARGO CRUZ Ana Erika
8. Future Work
Our plans for future work, mainly concern:
Study of other variables related to fault-proneness of code.
Data normalization and other pre-processing techniques, as well as other methodologies to predict fault-proneness of code.
The exploration of other areas of research in which our UML metrics can be applied.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

81. Thank you for listening!
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

82. JAIST – CAMARGO CRUZ Ana Erika
Publications
Camargo Cruz Ana E., Ochimizu Koichiro, A UML Approximation of Three Chidamber- Kemerer Metrics and Their Ability to Predict Faulty Code across Software Projects, IEICE TRANSACTIONS on Information and Systems, Vol.E93-D,No.11,Nov. 2010, pp , Japan.
Camargo Cruz Ana E., Exploratory Study of a UML Metric for Fault Prediction, Proceedings of the 32nd. International Conference on Software Engineering (ICSE 2010), Vol. 2 pp , Doctoral Symposium, May , Cape Town, South Africa.
Camargo Cruz Ana E., Ochimizu Koichiro, Towards Logistic Regression Models for Predicting Fault-prone Code across Software Projects, Proceedings of the Third International Symposium on Empirical Software Engineering and Measurement (ESEM 2009), IEEE Catalog Number: CFP09ENM, pp , Oct 15th- 16th 2009, Lake Buena Vista, USA.
Camargo Cruz Ana E., Ochimizu Koichiro, Quality prediction model for object oriented software using UML metrics, Proceedings of the 4th World Congress for Software Quality, September 2008, Bethesda, Maryland, USA.
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

83. Q3. How to enable our UML metrics be used with code-based models?
Project “A”
P(MF) x x1
design-complexity
metric
Can the model, built with Project “A”, work efficiently with other projects?
Code
Code
Code
Project “B”
Project “D”
Project “C”
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika

84. UML CK metrics Evaluation[9/11] Sensitivity of UML MetricsB2
UML CK metrics Evaluation[9/11] Sensitivity of UML Metrics
Same Design , Different Implementations
11/14/2018
JAIST – CAMARGO CRUZ Ana Erika