Evaluating Software Clustering Algorithms

The problem How do we know that a particular decomposition of a software system is good? What does ‘good’ mean anyway? We can compare against a Mental model Benchmark standard Can be done either manually or automatically 5/26/2019 COSC6431

Manual evaluation Have experts evaluate automatic decompositions Very time-consuming, impractical Also quite subjective Need an automatic, objective way of doing it 5/26/2019 COSC6431

Automatic evaluation Usually measures the similarity of an automatic decomposition A to an authoritative decomposition M (prepared manually) Major drawback: Assumes there exists one ‘correct’ decomposition Other evaluation approaches are possible, such as measuring the stability of an SCA 5/26/2019 COSC6431

Precision/Recall Standard Information Retrieval measures Were applied in a software clustering context by Anquetil Definitions: Intra pair: A pair of software entities in the same cluster Inter pair: A pair of entities in different clusters 5/26/2019 COSC6431

Precision/Recall Precision: Percentage of intra pairs in A which are also intra in M Recall: Percentage of intra pairs in M found also in A A good algorithm should exhibit high values in both measures 5/26/2019 COSC6431

Precision / Recall example Decomposition A Decomposition B 1 4 5 2 3 8 10 7 9 6 Pair 1-5 : Intra-pair in A but not in B Precision: 16/20 = 80% Recall: 16/21 = 76.2% 5/26/2019 COSC6431

Problems with P/R Sensitive to the size and number of clusters Differences are exaggerated if you have small/many clusters Two values makes comparison harder The two values are interchangeable if there is no “reference” decomposition 5/26/2019 COSC6431

Koschke – Eisenbarth measure Loosely based on Precision/Recall Attempts to be less strict Definitions: GOOD match: Two clusters (one in A, one in M) with both precision and recall larger than a threshold p (typical value 70%) OK match: Two clusters with only one of the measures larger than p 5/26/2019 COSC6431

Koschke – Eisenbarth metric B1B4 A2 A3 A4 A5 B1 B2 B3 A2A3 B5 B4 Good OK 5/26/2019 COSC6431

Koschke – Eisenbarth metric Does not take edges into account In extreme situations (each cluster contains only one element) may provide strange results (similarity of 100%) No penalty for joining clusters 5/26/2019 COSC6431

MoJo distance The distance between two different partitions of the same software system is defined as the minimum number of Move and Join operations to transform one to the other Move: Remove an object from a cluster and put it in a different cluster Join: Merge two clusters into one Split: Has to be simulated by Move operations 5/26/2019 COSC6431

MoJo example Decomposition A Decomposition B 1 2 1 2 3 4 3 5 7 4 5 6 6 8 7 8 5/26/2019 COSC6431

Why only Move and Join? Two clusters can be joined in only one way. One cluster can be split in two in an exponential number of ways Joining two clusters only means that we performed more detailed clustering than required Splitting is effectively assigned a weight equal to the cardinality of the smaller of the two resulting clusters 5/26/2019 COSC6431

Computing MoJo distance Give each object in partition A a tag j if it belongs in cluster Mj in partition M For each cluster in A, assign it to group Gk if it contains more objects with tag k than any other tag. To resolve ties, use the maximum bipartite matching method to maximize the number of groups. Join clusters within same group. Then move all objects with tag k to the clusters in group Gk. 5/26/2019 COSC6431

MoJo Distance Calculation MoJo Distance is M+J, where M is the number of Moves, and J is the number of Joins The moves for each cluster is , and the total moves is Assume the number of non-empty groups is g, from G1 to Gg. The joins of each group is|Gk|-1. Then the number of Joins is 5/26/2019 COSC6431

Outline of Proof Move & Join operations are commutative The algorithm uses the minimum number of Moves Any algorithm that also uses the minimum number of Moves can not beat us For any algorithm that does not use the minimum number of Moves, there exists another one which is better or equal and uses the minimum number of Moves 5/26/2019 COSC6431

MoJoFM Effectiveness Metric Assumes the existence of a ‘correct’ authoritative partition MoJo Distance can be used to measure the distance between two arbitrary partitions 5/26/2019 COSC6431

MeCl and EdgeSim Other measures do not consider edges Edges might convey important information EdgeSim: Rewards clustering algorithms for preserving the edge types Penalizes clustering algorithms for changing the edge types MeCl: Rewards the clustering algorithm for creating cohesive “subclusters” 5/26/2019 COSC6431

EdgeSim Inter-edge: Edge between clusters Intra-edge: Edge within a cluster Y: set of edges that are of the same type in both A and B 5/26/2019 COSC6431

EdgeSim example In this example, EdgeSim(A,B) = 52.6% 5/26/2019 COSC6431

EdgeSim counterexample 5/26/2019 COSC6431

EdgeMoJo philosophy A similarity measure cannot make assumptions as to what cluster a particular object should belong to Dissimilarity between decompositions should increase if the misplacement of an object results in the misplacement of a large number of edges 5/26/2019 COSC6431

EdgeMoJo calculation Apply MoJo and obtain a series of Move and Join operations Perform all Join operations The cost of each Move operations increases from 1 to 5/26/2019 COSC6431

EdgeMoJo example Decomposition A Decomposition B 1 2 1 2 3 4 5 3 4 5 7 6 8 6 8 m(5) = 1 + |4-1|/(4+1) = 1.6 5/26/2019 COSC6431

Real data experiments 5/26/2019 COSC6431

Synthetic data experiments 5/26/2019 COSC6431