A UML Approximation of a Subset of the CK Metrics and Their Ability to Predict Faulty Classes CAMARGO CRUZ Ana Erika Advisor: Prof. OCHIMIZU Koichiro July 28th, 2011 Good Afternoon every one, thank you for being here. My name is Ana Erika Camargo Cruz, and today I am presenting my Thesis on “A UML …”, which has been done under the supervision of Prof. OCHIMIZU Koichiro here present in the audience

Achievements Major Achievements Journal IEICE, ICSE, ESEM, 4WCSQ UML metrics → to approximate the code Chidamber-Kemerer (CK) metrics. Fault prediction before coding → univariate logistic regression modeling Data Normalization → to reduce the systematic error, to improve fault prediction. Minor Achievements A methodology to report software implementation progress (Technical Report) Mathematical definition of our UML metrics (Not published) Most of my research work has been already published, and mainly concern three topics: First, We propose a set of UML metrics … Second, Third, Some minor achievements concern: A methodo … 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

JAIST – CAMARGO CRUZ Ana Erika Agenda Motivation Research Outline Background Our Approach Metrics Definition Code CK metrics vs UML metrics Normalization Fault Prediction Fault Prediction Multivariate Analysis Related Work Conclusions Future Work These are the different topics to be covered in this presentation during the following 50 min. Let me stat with my presentation … 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

JAIST – CAMARGO CRUZ Ana Erika 1. Motivation A basic software development process How to make earlier detections of low-quality elements within the life-cycle of the software? Analysis Design Implementation The greater the number of defects found, the greater the amount of resources needed to fix them back in the process Testing 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

2. Research Outline STEP 2 STEP 1 Analysis D C Design B Implementation CODE BASED PREDICTION MODEL UML Design Complexity Metrics Faulty Code Design STEP 1 B Approximation CODE BASED PREDICTION MODEL CODE Design Complexity Metrics Faulty Code Implementation (*) Good Predictors: The CK Metrics A Existing Research Works Test 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

3. Background : The CK Metrics What are they? CK and Fault proneness of Code The CK and UML artifacts Understanding the CK Metrics Conclusion 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

JAIST – CAMARGO CRUZ Ana Erika 3.1.What are the CK Metrics ? To measure complexity of the design Measured from the code Related to fault-proneness, maintenance effort, re-work effort for re-using classes, etc. Six metrics: DIT (Depth of Inheritance) NOC (Number of Children) WMC (Weighted Method per Class) RFC (Response For Class) CBO (Coupling Between Objects) LCOM (Lack of Cohesion in Methods) 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

3.2. The CK Metrics and Fault Prediction Study Used Metrics Good Predictors Basili et al. (1996) CK RFC, DIT, NOC → CBO → WMC Briand et al. (2000) 49 metrics of coupling, cohesion and inheritance metrics (CK among them) RFC and other coupling measures → NOC and DIT and other measures of inheritance Kanmani et al. (2004) CK among 64 metrics of inheritance, size, cohesion and coupling No details Nachiappan et al. (2005) STREW-J suite: 3 CK metrics and others Olague et al. (2007) CK, MOOD, QMOOD CK: RFC, WMC and QMOOD: NOM, CIS 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

JAIST – CAMARGO CRUZ Ana Erika 3.3. The CK Metrics and UML Research works found in the literature: Focus on UML class diagrams [Baroni, Mc Quillan et al., 2003, 2006]: CBO, RFC and LCOM measurement is not straightforwardly. HINT: Behavioral diagrams could provide more precise measures of the CBO and RFC metrics, but they do not indicate how Require very well detailed design information, such as that every non-abstract operation must be represented in one activity diagram and instance variable usages [Tang et al.,2002]. Mei-Huei Tang and Mei-Hwa Chen. Measuring oo design metrics from UML. In UML ’02: Proc. of the 5th Intl. Conference on The Unified Modeling Language, pages 368– 382, London, UK, 2002. Springer-Verlag. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

3.4. Understanding the CK Metrics [2/4] WMC is the sum of the complexities of every method of the given class. If the complexities are considered unity then WMC is equal to NOM (Number of Methods of a class). class A{ private C c; private int var; public int m1A(){ B b; b.m1B(); b.m2B(); } public int m2A(){ return (var+10); public void m3A(){ var = var*c.m2C; WMC(A) = 3 methods 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

3.4. Understanding the CK Metrics [3/4] RFC is measured as the number of methods of a given class, PLUS the number of different methods of other classes directly called by the given class. class A{ private C c; private int var; public int m1A(){ B b; b.m1B(); b.m2B(); } public int m2A(){ return (var+10); public void m3A(){ var = var*c.m2C; RFC(A) = 3 + 3 = 6 methods 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

3.4. Understanding the CK Metrics [1/4] CBO is the number of other classes to which a class is coupled. class A{ private C c; private int var; public int m1A(){ B b; b.m1B(); b.m2B(); } public int m2A(){ return (var+10); public void m3A(){ var = var*c.m2C; CBO(A) = 2 classes 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

3.4. Understanding the CK Metrics [4/4] LCOM measures the dissimilarity of methods in a class by instance variable or attributes. class A{ private C c; private int var; public int m1A(){ B b; b.m1B(); b.m2B(); } public int m2A(){ return (var+10); public void m3A(){ var = var*c.m2C; LCOM(A) = |P| − |Q|; IF |P| >= |Q| 0; IF |P| < |Q| |P| number of pairs of methods with NO variable in COMMON |Q| number of pairs of methods with AT LEAST ONE variable in COMMON m1A m2A = {} m1Am3A = {} m2Am3A= {var} |P| = 2 |Q| = 1 LCOM(A) = 2 − 1 = 1 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

JAIST – CAMARGO CRUZ Ana Erika 3.5. Conclusion CBO, RFC and LCOM metrics cannot be obtained precisely from UML class diagrams. Hypothesis: Because CBO and RFC are coupling measures, and UML communication diagrams describe how objects of classes interact to a method instance level, UML communication diagrams can provide good approximations of such metrics 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

STEP 2 STEP 1 Analysis D C E Design B B1 How? Implementation A Test CODE BASED PREDICTION MODEL UML Design Complexity Metrics Faulty Code Design STEP 1 B Approximation B1 How? CODE Design Complexity Metrics Implementation (*) Good Predictors: The CK Metrics A Test 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

JAIST – CAMARGO CRUZ Ana Erika B1 UML CK Metrics How to? Goal: Analyze UML Communication Diagrams for the purpose of obtaining an approximation of the CK RFC, CBO and WMC metrics with respect to their objects interactions . Q3. How can we approximate the CBO metric? Q1. How can we approximate the WMC metric? Q2. How can we approximate the RFC metric? 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML CK Metrics[4/11] Direct and Indirect Metrics B1 UML CK Metrics[4/11] Direct and Indirect Metrics In order to provide a UML approximation of the CBO, RFC and WMC CK metrics: Set of Direct Metrics for Objects Set of Indirect Metrics for Classes 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML CK Metrics Direct Metrics[1/3] B1 UML CK Metrics Direct Metrics[1/3] x being an object of a class X and x acting as a RECEIVER: [D1] NDRCMsgs(x) Number of Different Received Call Messages by x Example: NDRCMsgs(: Calculator) = 1 NDRCMsgs(: Display) = 1 The objects depicted in a communication diagram can play two basic roles: The can be acting as a SENDER and/or as a RECEIVER. Looking at the objects acting a receivers, we measure our first direct metric NDRCMsgs 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML CK Metrics Direct Metrics[2/3] B1 UML CK Metrics Direct Metrics[2/3] x being an object of a class X and x acting as a SENDER: [D2] NDSCMsgs(x) Number of the Different Sent Call Messages by x Example: NDSCMsgs(: Calculator) = 2 NDSCMsgs(: Display) = 0 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML CK Metrics Direct Metrics[3/3] B1 UML CK Metrics Direct Metrics[3/3] X being an object of a class X and x acting as a SENDER: [D3] NDRO(x) Number Different Receiving Objects of a Message from x Example: NDRO(: Calculator) = 2 NDRO(: Display) = 0 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML CK Metrics Indirect Metrics: UWMC B1 UML CK Metrics Indirect Metrics: UWMC class Calculator(){ private Display display = new Display(); public Multiply(int a, int b){ Multiplier multiplier = new Multiplier(); result = multiplier.Multiply(a,b); display.display(this.result); } public Sum(int a, int b){ Adder adder = new Adder(); this.result = adder.Sum(a,b) Diagram [D1] NDRCMsgs(:Calculator) A 1 = |{A1}| B 1 = |{B1}| Total 2 = |{A1, B1}| = NDIM(Calculator) Number of Different Instantiated Methods = NDIM(X) Now, Let us say we would like to approximate WMC using our direct metrics for objects. For example, for class Calculator WMC is equal two, because it has two methods … To approximate this metric we define a NEW indirect metric call NDIM (X), which uses our first metric defined as NDRSCM … We would have to measures thise metrics for ALL OBJECT OF CLASS CALCULATOR. WMC(Calculator) = 2 UWMC (X) = NDIM(X) = 2 11/14/2018

UML CK Metrics Indirect Metrics: URFC B1 UML CK Metrics Indirect Metrics: URFC class Calculator(){ private Display display = new Display(); public Multiply(int a, int b){ Multiplier multiplier = new Multiplier(); result = multiplier.Multiply(a,b); display.display(this.result); } public Sum(int a, int b){ Adder adder = new Adder(); this.result = adder.Sum(a,b) Diagram [D2] NDSCMsgs(:Calculator) A 2 = |{A4, A2 = A3 = A6}| B 2 = |{B4, B2 = B3 = B5}| Total 3 = |{A4, A2 = B2, B4}| = NDIMO(Calculator) Number of Different Instantiated Methods of Other classes = NDIMO (X) RFC(Calculator) = 2 + 3 = 5 URFC(X) = UWMC(X) + NDIMO(X) 11/14/2018

UML CK Metrics Indirect Metrics: UCBO class Calculator(){ private Display display = new Display(); public Multiply(int a, int b){ Multiplier multiplier = new Multiplier(); result = multiplier.Multiply(a,b); display.display(this.result); } public Sum(int a, int b){ Adder adder = new Adder(); this.result = adder.Sum(a,b) Diagram [D3] NDRO(:Calculator) A 2 = |{: Display, : Multipier}| B 2 = |{: Display, : Adder}| Total 3 = |{: Display, : Multiplier, : Adder}| = NDRC(Calculator) Number of Different Receiving Classes CBO(Calculator) = 3 UCBO(X) = NDRC(X) 11/14/2018

JAIST – CAMARGO CRUZ Ana Erika B1 UML CK Metrics Summary Goal: Analyze UML Communication Diagrams for the purpose of obtaining an approximation of the CK RFC, CBO and WMC metrics with respect to their objects interactions. Q1. How can we approximate the CK-WMC metric? Q2. How can we approximate the CK-RFC metric? Q3. How can we approximate the CK-CBO metric? UWMC(X) = NDIM(X) URFC(X) = NDIM(X) + NDIMO(X) UCBO(X) = NDRC(X) 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

STEP 2 STEP 1 Analysis D C E Design B B1 How? B2 Implementation CODE BASED PREDICTION MODEL UML Design Complexity Metrics Faulty Code Design STEP 1 B Approximation B1 How? CODE Design Complexity Metrics B2 Implementation How Good? (*) Good Predictors: The CK Metrics A Test 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML CK Metrics Evaluation [1/11] B2 UML CK Metrics Evaluation [1/11] Process: 1. Collection of code measures of 4 different small-size software projects written in Java by students of JAIST. They developed their programs based on the UML diagrams provided in Gomaa’s book* • An e-commerce system (ECS). • Two banking system (BNA, BNB). • A cruise control and monitoring system (CRS). 2. Measurement of the corresponding UML metrics from their UML communication diagrams provided in Gomaa’s book* 3. Comparison and Evaluation: Accuracy, Precision and Linearity. (*) Hassan Gomaa. Designing Concurrent, Distributed, and Real-Time Applications with UML. Addison Wesley-Object Technology Series Editors, Boston, MA, USA, 2000. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML CK Metrics Evaluation[3/11] Measures for Evaluation B2 UML CK Metrics Evaluation[3/11] Measures for Evaluation Accuracy: Degree of conformity with the true measure. Relative Error = |Code Metric– UML Approximation | |Code Metric| Precision: It refers to the degree of repeatability of measurements. It does not require to know the correct or true value → standard deviation. Linear Relationship between our UML approximations and code CK metrics → Pearson correlation coefficient. Precision: It refers to the degree of repeatability of measurements under unchanged conditions showing the same results. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML CK Metrics Evaluation [2/11] Accuracy and Precision B2 UML CK Metrics Evaluation [2/11] Accuracy and Precision Accuracy(+) Probability Density Systematic Error (-) Error approximation = Systematic Error + Random Error Systematic Error: Predictable, Proportional to the True Value Random Error : Unpredictable, Inconsistent measurements Assuming that our Relative Errors have a Normal distribution, we would have a curve like this. The distance between the mean of our relative errors and a reference value 0 (desirable) would represent how accurate is our measurement methodology. As for precision, it can be observed in the closeness of the measurements to each other. The closer the more precise our measurement method is. _________________________ The theory of measurements considers that all measurements have some error approximation, which is a sum of a systematic error and random error. Systematic Error, is considered to be predictable, and typically constant or proportional to the true value. In contrast Random Errors are unpredictable and lead to inconsistent measurements. Reference Value Relative Error = 0 Mean Precision (+) Random Error (-) Systematic Error (-) → Accuracy (+) Random Error (-) → Precision (+) 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML CK metrics Evaluation[4/11] UCBO Accuracy Mean(RE) 0.5 Precision  SD(RE) 0.36 Linearity 0.8 Accurate but not Precise Strong linear relationship 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML CK metrics Evaluation[5/11] URFC B2 UML CK metrics Evaluation[5/11] URFC Accuracy  Mean(RE) 0.7 Precision SD(RE) 0.15 Linearity 0.86 Precise but not Accurate Strong linear relationship 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML CK metrics Evaluation[6/11] UWMC B2 UML CK metrics Evaluation[6/11] UWMC Accuracy  Mean(RE) 0.65 Precision  SD(RE) 0.4 Linearity  0.46 Neither Accurate nor Precise Weak linear relationship 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML CK metrics Evaluation[7/11] Comparison UCBO, URFC and UWMC Probability Density Accuracy: UCBO <UWMC < URFC Precision: URFC<UWMC< UCBO URFC UWMC UCBO 0.5 0.7 11/14/2018 Average Relative Errors JAIST – CAMARGO CRUZ Ana Erika

The Gap [1/2] Proportional Differences B2 The Gap [1/2] Proportional Differences Metric Same level of detail in the design and 1 Developer or Developers with the same programming style. Classes Code Measures Systematic Type of Error UML Measures 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

The Gap [2/2] Non-Proportional Differences B2 The Gap [2/2] Non-Proportional Differences Metric Classes dealing with other requirements different from the ones specific to the domain problem, such as: GUIS, DB connectivity, IPC communication, etc. Random Type of Error Classes Code Measures Systematic Type of Error UML Measures 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML CK metrics Evaluation[8/11] Sensitivity of UML Metrics B2 UML CK metrics Evaluation[8/11] Sensitivity of UML Metrics Same Design , Different Implementations BNA entangles GUIs’ functionalities with those specific to the domain problem BNB was implemented different from the original design Use of Interfaces Decoupling GUIs’ BNB-00 , no considered in the design, Implements Interfaces BNB-05 and BNB-04 GUI’s Functionalities BNA-04 and BNA-05 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML CK metrics Evaluation[10/11] Sensitivity of UML Metrics B2 UML CK metrics Evaluation[10/11] Sensitivity of UML Metrics Same Design , Different Implementations UCBO [classes] BNA BNB Average(RE) 0.5 0.6 SD(RE) 0.37 0.33 Linearity 0.85 0.87 Different Implementations → Similar Evaluation Results Different Implementations → Similar Correlation URFC [methods] BNA BNB Average(RE) 0.85 0.6 SD(RE) 0.1 0.19 Linearity 0.7 0.8 UWMC [methods] BNA BNB Average(RE) 0.69 0.56 SD(RE) 0.15 0.41 Linearity 0.71 Different Implementations → Similar Precision Similar Correlation 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML CK metrics Evaluation[11/11] Conclusions B2 UML CK metrics Evaluation[11/11] Conclusions UCBO is the most accurate CBO Unity [Class] URFC is the most precise RFC Unity [Method] UWMC less accurate than UCBO and less precise than URFC WMC Unity [Method] UCBO and URFC show to have strong linear relationship to its code metric URFC and UCBO can approximate better their respective code metrics. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

STEP 2 STEP 1 Analysis D C E Design B B1 How? C1 B2 Implementation CODE BASED PREDICTION MODEL UML Design Complexity Metrics Faulty Code Design STEP 1 B Approximation B1 How? CODE Design Complexity Metrics C1 B2 Implementation How Good? (*) Good Predictors: The CK Metrics A Q1. How to build a code-based prediction model? Q2. Why not a UML-based prediction model? Q3 . How to enable our UML metrics to be used with a code-based prediction model? Test 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Q1. How to build a prediction model? Different techniques could be used to model data and predict faulty code: linear regression, Artificial Neural Networks, etc. Logistic Regression is preferred because: No assumptions on the distribution of variables are made. Use: When we have one dependent variable (y) with two values e.g. Most Faulty (MF) / Least Faulty (LF), 1/0; and one or more independent variables (xs). Goal: To predict the probability of getting a particular value of y (e.g. A Most Faulty class), given xs variables, through a logit model. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Q1. How to build a prediction model? x P(MF) Univariate Model: only one independent variable “x” Multivariate Model: more than one independent variable “x1, x2, …” 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Q2. Why NOT a UML-Based Prediction Model? Our previous work → Univariate UML-Based Prediction Models using JAIST projects Multivariate Analysis → Larger amount of data is required (~10 data points per independent variable) JAIST projects have a maximum number of 16 designed classes (BNS) → cannot be used for Multivariate Analysis P(MF) x x1, x2, … xn design-complexity metrics UML-based 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Q3. How to enable our UML metrics be used with code-based models? P (MF) LEAST FAULTY (LF) MOST FAULTY (MF) Smaller Size SW project 1 Prediction Model derived from the code of Project “A” OR UML Measures of Project “A” 0.5 10 X = Number of Methods 20 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Solution: Normalization C1 Solution: Normalization In order to apply efficiently the same prediction model across projects and to use our UML approximations using a code-based prediction model. Application of Data Normalization: A common scale Reduction of the systematic error of our approximations 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

STEP 2 STEP 1 Analysis D C E Design C1 CODE BASED PREDICTION MODEL UML Design Complexity Metrics Faulty Code Design STEP 1 Approximation C1 Q1. How to build a code-based prediction model? CODE Design Complexity Metrics Q2. Why not a UML-based prediction model? Implementation (*) Good Predictors: The CK Metrics A Q3 . How to enable our UML metrics to be used with a code-based prediction model? C2 Data Normalization: 2 candidates Test 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Preparation for Normalization [1/7] C2 Preparation for Normalization [1/7] Further Data Exploration: In addition to the JAIST projects, we analyzed 13 packages of the Mylyn from Eclipse (MYL) → used for building our code-based prediction models A total of 37 classes from the JAIST projects, and 602 classes from the MYL project are used for this analysis. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Preparation for Normalization [2/7] Boxplot C2 Preparation for Normalization [2/7] Boxplot Lower Fourth = FL (25%) Upper Fourth = FU (75%) Fourth-spread dF = FU − FL Outliers cutoffs: FL − 1.5dF and FU + 1.5dF OUTLIERS produces biased regression estimates and reduces the predictive power of regression models. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Preparation for Normalization [3/7] Results of Boxploting for CBO 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

JAIST – CAMARGO CRUZ Ana Erika Preparation for Normalization [4/7] Results of Boxploting for RFC and WMC 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Preparation for Normalization [5/7] General Observations C2 Preparation for Normalization [5/7] General Observations As the size of the dataset increases, the number of outliers do too. Most of the data are positively skewed. Tendency for data to spread as the level (Median) increases. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Preparation for Normalization [6/7] Spread VS Level C2 Preparation for Normalization [6/7] Spread VS Level 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Preparation for Normalization[7/7] C2 Preparation for Normalization[7/7] Can a prediction model built with MYL-P13 applied to BNA-COD or BNA-UML ? Can a prediction model built with MYL-P13 be applied to BNA-COD or BNA-UML ? 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Normalization [1/5] Candidate 1 Log Transformations: Especially useful to reduce the relationship between spread and level. Elimination of Outliers Improve Normality x’ is the new normalized x raw value 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Normalization [2/5] Candidate 2 Linear Scaling to Unit Variance: To acquire a common Scale [0..1] x’ is the new normalized value [0..1] x raw value x media s standard deviation It is based in the widely used z-score transformation, with transformed 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Normalization [3/5] Candidate 2 Approach 2: Linear Scaling to Unit Variance An additional l shift and re-scaling guarantees … Guarantees 99% of the normalized data fall [0..1] 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Normalization [4/5] Comparison: Candidate 1 vs Candidate 2 LOG Linear Scaling to Unit Variance 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Normalization [5/5] Conclusion: Candidate 1 vs Candidate 2 UCBO LOG(UCBO +1) LS(UCBO) Accuracy - Mean(RE) 0.5 0.35 0.17 Precision - SD(RE) 0.36 0.32 0.11 Linear Scaling to Unit Variance performed better than LOG transformations The systematic error was reduced. URFC LOG(URFC+1) LS(URFC) Accuracy -Mean(RE) 0.7 0.55 0.19 Precision - SD(RE) 0.17 0.14 UWMC LOG(UWMC+1) LS(UWMC) Accuracy -Mean(RE) 0.65 0.38 0.29 Precision - SD(RE) 0.4 0.2 0.28 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

STEP 2 STEP 1 D1 Analysis C D Design B C1 CODE BASED PREDICTION MODEL UML Design Complexity Metrics Faulty Code Design STEP 1 B Approximation CODE Design Complexity Metrics C1 Q1. How to build a prediction model? Implementation (*) Good Predictors: The CK Metrics A Q2. Why a code-based prediction model? Q3 . How to enable our UML metrics to be used with these models? C2 Data Normalization: 2 candidates Test 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Construction of Prediction Models[1/10] We use the code of 13 packages of the MYL system. To construct several univariate logistic models to predict Most Faulty classes: P(y=1) x design-complexity metric P(MF) Code-based 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Construction of Prediction Models[2/10] Procedure: Elimination of Outliers Normalization using Linear Scaling to Unit Variance. Construction of 3 prediction models (RFC, CBO, WMC): For every MYL package Selection of the best 3 resulting models. Test of the 3 best models using the 12 remained packages of the MYL and the 3 JAIST projects. Other 3 models were build using raw data for comparison. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Construction of Prediction Models[3/10] Evaluation Parameters To evaluate how well our prediction models can discriminate between MF and LF classes, we use the following indicators: 1. Correctness: It is the percentage of classes correctly classified. 2. Specificity: It is the percentage of LEAST FAULTY (LF) classes correctly classified. 3. Sensitivity: It is the percentage of MOST FAULTY (MF) classes correctly classified. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Construction of Prediction Models[4/10] CBO Prediction Model CBO Prediction Model: Normalized Data VS Raw Data The best prediction model was the one modeling the data of package M-11 Correctness: %of classes correctly classified. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Construction of Prediction Models[5/10] CBO Prediction Model CBO Prediction Model: Normalized Data VS Raw Data Sensitivity: % of MF classes correctly classified Specificity: % of LF classes correctly classified 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Construction of Prediction Models[6/10] RFC Prediction Model RFC Prediction Model: Normalized Data VS Raw Data The best prediction model was the one modeling the data of package M-12 Correctness: %of classes correctly classified. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Construction of Prediction Models[7/10] RFC Prediction Model RFC Prediction Model: Normalized Data VS Raw Data Sensitivity: % of MF classes correctly classified Specificity: % of LF classes correctly classified 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Construction of Prediction Models[8/10] WMC Prediction Model WMC Prediction Model: Normalized Data VS Raw Data The best prediction model was the one modeling the data of package M-02 Correctness: %of classes correctly classified. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Construction of Prediction Models[9/10] WMC Prediction Model WMC Prediction Model: Normalized Data VS Raw Data Sensitivity: % of MF classes correctly classified Specificity: % of LF classes correctly classified 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Construction of Prediction Models[10/10] Summary Comparing the 16 datasets, using only code measures, we can say that the normalization procedure: Greatly improved the results of the RFC model, Benefits less to the CBO model, Almost no improvement was obtained with the WMC model. Although, the NWMC model can discriminate MF classes better than the WMC model. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

STEP 2 STEP 1 D1 D2 C Analysis D Design B Implementation A Test UML Design Complexity Metrics CODE BASED PREDICTION MODEL Faulty Code Design STEP 1 Approximation B CODE Design Complexity Metrics Implementation (*) Good Predictors: The CK Metrics A Test 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML Metrics Evaluation as Predictors of Faulty code Evaluation using the JAIST Projects P(y=1) x x_code P (MF class) Comparison AND Evaluation (Using Normalized Data) Code-based P(y=1) x x_UML P (MF class) Code-based 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML Metrics Evaluation as Predictors of Faulty code Using normalized measures, we applied the code-based prediction models (from the MYL) to predict faulty classes in our JAIST projects → code metrics VS UML metrics. Project Correctness Specificity Sensitivity NCBO UML- NCBO UML-NCBO ECS 80% 75% 100% 83% 67% CRS 88% 71% 33% BNS 89% 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML Metrics Evaluation as Predictors of Faulty code Project Correctness Specificity Sensitivity NRFC UML- NRFC UML-NRFC UML-NURFC ECS 80% 100% 67% CRS 57% 64% 0% 25% BNS 33% 50% 75% Project Correctness Specificity Sensitivity NWMC UML- NWMC UML-NWMC NRFC ECS 55% 45% 25% 71% 57% CRS 56% 38% 36% 50% BNS 78% 44% 67% 33% 100% 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML Metrics Evaluation as Predictors of Faulty code Conclusion Using normalized measures, the results of: URFC → equal and sometimes better than those obtained by RFC. UCBO → nearly equal results to those obtained by CBO. UWMC → did not perform better than WMC. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

STEP 2 STEP 1 D1 D2 C Analysis D Design B Implementation A Test UML Design Complexity Metrics CODE BASED PREDICTION MODEL Faulty Code Design STEP 1 Approximation B CODE Design Complexity Metrics Implementation (*) Good Predictors: The CK Metrics A Test 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

5. Fault Prediction Multivariate Analysis [1/2] Multivariate prediction models require uncorrelated variables (Linear or Logistic Regression) MYL-P11 Data: Correlation Coefficients Other studies findings conform our results, suggesting high correlation not only among the CK metrics, but also among other sets of design complexity metrics. RFC CBO WMC NF 1 0.9 0.8 0.7 Number of Faults (NF) 0.5 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

5. Fault Prediction Multivariate Analysis [2/2] NF RFC Shared Variance: Variation amount of two variables that tend to vary together CBO NF Multiple Correlation coefficient A measure of the fit of a multiple linear regression model → [0,1] NF RFC CBO CBO NF RFC Shared variance : * It is represented by the square of their correlation coefficient, r2. * An easy way to visualize this is with a Venn diagram, * For example, the amount of shared variance between CBO and NF is represented by their overlap of variation r2. The multiple coefficient of correlation, R2, is a measure of the fit of a multiple linear regression model. It falls somewhere between zero and one; 1: indicates that all data points fall exactly on a line in multidimensional space and 0:no relationship at all between the independent variables collectively and the dependent variable. The shared variance between CBO and NF likely contains the one between RFC and NF. RFC → poor significance 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

JAIST – CAMARGO CRUZ Ana Erika 6. Related Work [1/3] ”Assessing UML design metrics for predicting fault-prone classes in a Java system” Nugroho et al., Proceedings of the MSR 2010, Cape Town, South Africa, 2010. 5 UML metrics (class and sequence diagrams) → to predict faulty classes. A significant industrial Java system was used to build and test 3 logistic regression models: a UML-based model, a code-based model and an hybrid model 2 of their UML metrics yields a slightly better prediction accuracy than using SLOC metric. Best Model: Their hybrid model with 78% accuracy. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

JAIST – CAMARGO CRUZ Ana Erika 6. Related Work [2/3] ” Quality prediction model for object-oriented software using UML metrics” In Proc. of the 4th World Congress for Softw.Quality, Bethesda, Maryland, USA, 2008. ASQ. UML class and collaboration diagrams were used to approximate CBO and RFC CK metrics and three metrics of the QMOOD set. 2 JAIST projects were used to build and test several UML-based univariate prediction models. The UML-based prediction models were compared with code-based prediction models: Code-based models generally predict better than UML-based models URFC model (84.61%) ~ RFC model (84.61%) UCBO models (69.20%) ~ CBO model (92.3%) UCIS model (90.9%) → from class diagrams (public number of methods) → (similar to our current proposed UWMC metric) 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

JAIST – CAMARGO CRUZ Ana Erika 6. Related Work [3/3] ” Quality prediction model for object-oriented software using UML metrics” In Proc. of the 4th World Congress for Softw.Quality, Bethesda, Maryland, USA, 2008. ASQ. No normalization method was used Prediction accuracy of UML-based prediction models 69% - 90.9% CONCLUSION: UML metrics can be acceptable predictors of fault-proneness of code. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

JAIST – CAMARGO CRUZ Ana Erika 7. Conclusions A simple approach to approximate CBO, RFC and WMC CK metrics using UML communication diagrams. Our UML RFC and CBO metrics can predict faulty code almost with the same accuracy as their respective code metrics. The normalization procedure was significantly efficient: for enabling our UML metrics to predict faulty code for improving the prediction ability of logistic models across different packages and projects. Both the UML WMC and the code WMC metrics showed a poor fault-proneness prediction ability. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

JAIST – CAMARGO CRUZ Ana Erika 8. Future Work Our plans for future work, mainly concern: Study of other variables related to fault-proneness of code. Data normalization and other pre-processing techniques, as well as other methodologies to predict fault-proneness of code. The exploration of other areas of research in which our UML metrics can be applied. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Thank you for listening! 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

JAIST – CAMARGO CRUZ Ana Erika Publications Camargo Cruz Ana E., Ochimizu Koichiro, A UML Approximation of Three Chidamber- Kemerer Metrics and Their Ability to Predict Faulty Code across Software Projects, IEICE TRANSACTIONS on Information and Systems, Vol.E93-D,No.11,Nov. 2010, pp. 3038-3050, Japan. Camargo Cruz Ana E., Exploratory Study of a UML Metric for Fault Prediction, Proceedings of the 32nd. International Conference on Software Engineering (ICSE 2010), Vol. 2 pp. 361-364, Doctoral Symposium, May 2-8 2010, Cape Town, South Africa. Camargo Cruz Ana E., Ochimizu Koichiro, Towards Logistic Regression Models for Predicting Fault-prone Code across Software Projects, Proceedings of the Third International Symposium on Empirical Software Engineering and Measurement (ESEM 2009), IEEE Catalog Number: CFP09ENM, pp. 460-463, Oct 15th- 16th 2009, Lake Buena Vista, USA. Camargo Cruz Ana E., Ochimizu Koichiro, Quality prediction model for object oriented software using UML metrics, Proceedings of the 4th World Congress for Software Quality, September 2008, Bethesda, Maryland, USA. 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

Q3. How to enable our UML metrics be used with code-based models? Project “A” P(MF) x x1 design-complexity metric Can the model, built with Project “A”, work efficiently with other projects? Code Code Code Project “B” Project “D” Project “C” 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika

UML CK metrics Evaluation[9/11] Sensitivity of UML Metrics B2 UML CK metrics Evaluation[9/11] Sensitivity of UML Metrics Same Design , Different Implementations 11/14/2018 JAIST – CAMARGO CRUZ Ana Erika