Tracy Hall, Brunel University David Bowes, University of Hertfordshire Andrew Kerr, University of Hertfordshire

 Why are we interested in replicating slices?  What are slice-based coupling and cohesion metrics?  What did Meyers & Binkley do in their study?  What did we do in our replication of M&B’s study?  How do our results compare to M&B’s?  Do slice results matter?  What are the implications of our findings?

 Aimed to investigate whether sliced-based metrics can predict fault-prone code.  We needed to validate that we were collecting slice-based metrics data correctly.  Tried to identically re-produce Meyers and Binkley’s (2004, 2007) metrics values  Our replication highlights many ways in which the identification of program slices can vary.  Our results identify a need for consistency and/or full specification of slicing variables.

 Original set of cohesion metrics proposed by Weiser in 1981 and extended by Ott et al in the 1990’s  Harman et al. (1997) introduced slice-based coupling.  Green et al (2009) present a detailed overview showing the evolution of slice-based coupling and cohesion metrics.

Meyers and Binkley (2007, p.8), use Harman et al.’s (1997) definition of coupling to define the coupling of a function f to be a weighted average of its coupling to all other functions in the program:

Cohesion metric definition (Ott & Thuss, 1993) Slice-based cohesion metrics

 Meyers and Binkley (2004, 2007) first to collect and analyse large scale slice-based metrics data  Collected slice-based metrics data on 63 open source C projects.  Produced a longitudinal study showing the evolution of coupling and cohesion over many releases of Barcode and Gnugo projects  Used CodeSurfer to slice  Wrote scripts to collect slice-based metrics data

“I have discovered a truly marvelous proof that it is impossible to separate a cube into two cubes, or a fourth power into two fourth powers, or in general, any power higher than the second into two like powers. This margin is too narrow to contain it.” (1637) Replicated Wiles A (1995) The problem in replicating studies Insufficient space in a published paper to describe the methods to allow for replication….

 Replicated only M&B’s longitudinal results for the evolution of cohesion in Barcode  Barcode has 65 functions & 49 releases  The highest preset build option was used on CodeSurfer  We tried to replicate the method reported by M&B.  We discussed with Dave Binkley methodological issues that were unclear.  We wrote our own Scheme scripts (and were provided with scripts from CREST (Youssef))

Barcode - M&B ResultsBarcode - Our results

Longitudinal cohesion Barcode - M&B Results Barcode – Our results (full vertex removal)

Trying to understand where we were going wrong…  Looked in detail at one data point (release 0.98)  Tried to examine all variations in the way that this data point could be calculated.  We sliced both on files and on projects  We varied the way lines of code are included in slices using: 1.Formal Ins: Input parameters for the function specified in the module declaration. 2.Formal Outs: Return variables. 3.Globals: Variables used by or affected by the module. 4.Printf: Variables which appear as Formal Outs in the list of parameters in an output statement. (based on the variations reported in previous studies analysed by Green et al 2009)

NB all these settings were sliced both on a file and project basis Not possible

I = Formal Ins, O = Formal Out, G = Globals, pF=printf; NB: Both forward and backward slices were used in all cases. Meyers & Binkley results: O=0.51 T=0.26 cov=0.54 min=0.30 max=0.71

 Only use pdgs which are 'user-defined‘ and remove pdgs with zero vertices  Keep globals identified n times?  String constants considered as output variables (?)  Slices are based on both data and control edges  Slices of length zero are removed (would have a significant impact on tightness)  Intersect all slices with the pdg vertices to remove vertices found outside of the pdg  Remove vertex indices with an identifier <1  Remove vertices associated with body '{' and '}'  Declaration vertices removed as not consistently included with forward and back slices  Return has auto generated value so if a variable is output via a global or written as well as returned the script may catch the same (source code) variable twice.  Global outputs from a function f include globals modified transitively by calls from f ("outgoing variables"), resulting in numerous slices.  Selection of actual inputs to output functions is naïve; sometimes we may want format string in printf statements  Dealing with placeholder functions: if they have size zero after vertices are pruned they are ignored  Should only some types of variables not be included in slicing criteria, e.g. string type?  Should forward slices use may-kill or declaration vertices? Time for variant performance analysis? Slide 19

 For slice-based metrics: ◦ Specifying precisely all parameters of a slice and a metric is important but difficult. ◦ Identifying the ‘best’ variant of a metric may be useful.  For replicating studies: ◦ Studies need to publish basic information that allows replication  For Software Engineering ◦ We need to build bodies of evidence and this must include replicated studies.

1. Green, P., Lane, P., Rainer, A., Scholz, S.-B. (2009). An Introduction to Slice-Based Cohesion and Coupling Metrics. Technical Report No. 488, University of Hertfordshire, School of Computer Science. 2. Harman, M., Okunlawon, M., Sivagurunathan, B., Danicic, S. (1997). Slice-Based Measurement of Coupling. IEEE/ACM ICSE workshop on Process Modelling and Empirical Studies of Software Evolution, (pp ). Boston, Massachusetts. 3. Meyers, T. M., Binkley, D. (2004) A Longitudinal and Comparative Study of Slice-Based Metrics. International Software Metrics Symposium, Chicargo, USA, IEEE Procs 4. Meyers, T. M., Binkley, D. (2007). An Empirical Study of Slice- Based Cohesion and Coupling Metrics. ACM Transactions on Software Maintenance, 17(1), pp Ott, L. M., &Thuss, J. J. (1993). Slice Based Metrics for Estimating Cohesion. In Proceedings of Internationl Software Metrics Symposium, Proceedings of the IEEE-CS, 71—81

Tracy Hall Reader in Software Engineering Brunel University Uxbridge, UK David Bowes Senior Lecturer in Computing University of Hertfordshire Hatfield, UK

 Some variants have a better relationship with fault-prone code than other varients…

 Another Cohesion metric: ◦ Proposed by Counsel et al 2006 Adapted for program slices : l = number of slices k = number of vertices in the module c = is the number of vertices for the slice based on j