Martin Hoffman Monica Hoffman

Goal To determine if a correlation exists between specified internal and dependability (external) metrics.

The Experiment UVa Online Judge Informal specifications in natural language Results of Online Judge were not used 59 specifications were selected

Sample Specification Problem E - Ones Given any integer 0 <= n <= not divisible by 2 or 5, some multiple of n is a number which in decimal notation is a sequence of 1's. How many digits are in the smallest such a multiple of n? Time limit – seconds Sample input Output for sample input

Selection of Programs Programs must compile and run within predetermined time and memory resources Programs written in C/C++ Must succeed for at least one demand Only the first successful submission of each author was used Program size smaller than 40kB No lookup tables allowed 71,917 submissions (avg.1219 per specification)

Internal Metrics Lines of Code (LOC) Number of comment lines Halstead Volume Describes the size of the implementation. V = N * log2(n) N is the total number of operators and operands n is the total number of unique operators and operands McCabe’s Cyclomatic Complexity (CC) Measures complexity based on program paths. M = N+1 where N is the number of binary decisions. Three way decisions count as 2 binary decisions N-way case statements count as n-1 binary decisions

Dependability Metrics Number of defects (D) D = 0 for the correct submission D = 1 for the penultimate submission There afterwards, each prior submission which displays a different behavior from the one after it increases D by 1. Probability of Failure on Demand (PFD) For some specifications a complete test was used where possible. For the remaining specifications, part of the demand space was tested completely or a random test was performed. # of demands = 2,500 or 10,000 for all specifications PFD = fraction of demands which program fails.

Correlations Correlation between internal software metrics were derived for each of 59 specifications.

Correlation Histograms

Findings Strong correlations CC vs. LOC (mean=0.78) Halstead Volume vs. CC (mean =0.75) Halstead Volume vs. LOC (mean =0.82)

Density Plots

Findings (cont.) Strong relationship between LOC and Volume/CC Mean Volume vs. mean LOC (correlation = 0.97) Mean CC vs. mean LOC (correlation = 0.95) Formulas to determine mean Volume = 45 x LOC – 428 CC = 0.22 x LOC Informally, every five lines of C/C++ code contains a branch

Findings (cont.) Internal software metrics of incorrect vs. correct programs Lines of Code On average, no difference Comment lines Correct programs have an average of 10% more comments Halstead Volume On average, no difference Cyclomatic Complexity On average, no difference

Findings (cont.) Correlations between internal metrics and dependability metrics Lines of Code Correlation between LOC and number of defects and probability of failure on demand (PFD) is close to zero Number of comment lines Correlation between number of comment lines and number of defects and (PFD) is close to zero Cyclomatic Complexity and Halstead Volume No correlation between these and number of defects or PFD (Probability of Failure on Demand)

Disagreements Ambiguities Histograms labels were unclear. Lacked a clear definition of a specification/demand. Definition of a defect in the dependability metrics. Size of programs (< 1000 LOC) Definition of LOC

References M. van der Meulen and M. A. Revilla. Correlations between Internal Software Metrics and Software Dependability in a Large population of Small C/C++ Programs.  2007 S.H. Kan. Metrics and Models in Software Quality Engineering, Second Edition. Pearson Education, Inc Pages