1. Software Metrics

2. Engineering
Engineering is a discipline based on quantitative approaches
All branches of engineering rely upon the use measurements
Software Engineering is not an exception
SE uses measurements to evaluate the quality of software products

3. Software Metrics
Software metrics are an attempt to quantify the quality of the software
Measures are expressed in terms of computable formulas or expressions
Elements of these expressions are derived from source code, design and other artifacts used in software development

4. Quality Measurement Direct measurement Indirect measurement
Are values calculated directly from known sources (source code, design diagram etc.)
E.g. number of software failures over some time period
Indirect measurement
Cannot be so accurately defined nor directly realized
E.g. reliability, maintainability, usability

5. Advantages of Metrics
Metrics are a tool for comparing certain aspects of software
Sometimes metrics provide guidance for evaluating complexity, time and cost of a future product
Metrics can be used to evaluate software engineers
Metrics can even be applied to the creation of development teams

6. Example A general rule in allocating project time:
25%: Analysis & Design
50%: Coding & Unit Test
25%: Integration/System/Acceptance Test

7. Calculation of ‘Lack of Cohesion’ metrics
Lack of Cohesion (LCOM) for a class is defined as follows:
Let ‘C’ denote the class for which LCOM is computed.
Let ‘m’ denote the number of methods in C.
Let ‘a’ denote the attributes of C.
Let ‘m(a)’ denote the methods of C that access the attribute ‘a’ in C.
Let ‘Sum(m(a))’ denote the sum of all ‘m(a)’ over all the attributes in C.
Now, LCOM( C ) is defined as ( m – Sum (m(a)) / a) / (m – 1)
C-S 446/546

8. Cohesion Metrics
If class C has only one method or no method, LCOM is undefined. If there are no attributes in C, then also LCOM is undefined. In both situations, for computational purposes, LCOM is set to zero.
Normally, LCOM value is expected to be between 0 and 2. A value greater than 1 is an indication that the class must be redesigned. The higher is the value of LCOM, the lower is the cohesion and hence the need for redesign.
C-S 446/546

9. Example
C-S 446/546

10. Calculation
The class ‘Account’ (without constructor) ‘Account’ (with constructor)
m = 4, a = m = 5, a = 3
m(Account#) = m(Account#) = 2
m(UserID) = m(UserID) = 2
m(Balance) = m(Balance) = 3
sum(m(a)) = sum(m(a)) = 7
LCOM (Account) = (4 –( 5 / 3)) / (3 – 1) = LCOM(Account) = (5 – (7 / 3))/ (5
– 1) = 0.667
C-S 446/546

11. Calculation m = 1, a = 2 m = 2, a = 2 m(UserID) = 0 m(UserID) = 1
The class ‘LoginAccount’ (without constructor) ‘LoginAccount’ (with constructor)
m = 1, a = m = 2, a = 2
m(UserID) = m(UserID) = 1
m(Password) = m(Password) = 2
sum(m(a)) = sum(m(a)) = 3
LCOM (LoginAccount) = (1 – (1 / 2)) / (1- 1) LCOM(LoginAccount) = (2
= 0 (set to zero because m = 1) – (3 / 2)) (2 – 1) = 0.5
C-S 446/546

12. Calculation m = 4, a = 1 m = 5, a = 1 m(Accounts) = 4 m(Accounts) = 5
The class ‘AccountsDatabase’ (without constructor) ‘AccountsDatabase’ (with constructor)
m = 4, a = m = 5, a = 1
m(Accounts) = m(Accounts) = 5
sum(m(a)) = sum(m(a)) = 5
LCOM (AccountsDatabase) = (4 – 4 / 1) / (4 – 1) LCOM(AccountsDatabase) = (5 – 5/1)(5-1) = 0
= 0 / 3 = 0
C-S 446/546

13. Calculation
The class ‘Login Accounts Database’ Exactly similar to ‘Accounts Database’ and hence LCOM (Login Accounts Database) = 0, both for with constructor and without constructor.
C-S 446/546

14. Control and Information Flow Metrics

15. Control Flow Define the calling sequence of modules
Defined by a call-graph A B C D

16. Fan-in metrics
Defines the number of modules that call a given a module M
Fi denotes the actual number of calls made to module M : determined at run-time
fi denotes the unique number of calls made to module M : determined from code
For the modules in the previous slide,
Fi (C) = X, X may be more than 2.
fi (C) = 2

17. Fan-out metrics
Defines the number of modules that are called by a given a module M
Fi denotes the actual number of calls made by module M : determined at run-time
fi denotes the unique number of calls made by module M : determined from code
For the modules in the previous slide,
Fi (A) = X, X may be more than 3.
fi (A) = 3

18. Henry and Kafura’s complexity metrics
Based on fan-in and fan-out metrics
Complexity (M) = (Fan-in (M) * Fan-out (M))2
Justification
Complexity increases as the square of connections between modules increases
Modules with large fan-out are relatively small and simple
Large and complex modules will have a low fan-in value
Modules with high values for fan-in and fan-out indicate a poor design  requires re-design

19. Henry and Selig’s complexity metrics
Complexity (M) = Ci * (Fan-in (M) * Fan-out (M))2
where Ci refers to internal complexity evaluated using Cyclomatic complexity or other measures.
This version of complexity is called “Hybrid complexity” because of the combined analysis of program structure and control flow.

20. System complexity by Card and Glass
System complexity CSys =
Structural complexity CStruct +
Data Complexity CData
Structural complexity CStruct =
where ‘n’ denotes the number of modules in the system n
∑ fan-out2 (i)
i=1 n

21. System complexity by Card and Glass (continued)
Data complexity CData = n
∑ D(i)
i=1 n
V(i)
of module i
where D(i) =
Fan-out(i) + 1
‘V(i)’ denotes the I/O variables used in module i

22. Justification by Card and Glass
Fan-in values do not provide SIGNIFICANT contribution to module complexity or system complexity
More I/O variables used in a module means more functionalities required to manipulate data and hence increased complexity
Data complexity is inversely proportional to fan-out values because fan-out values indicate how the current module delegates the functionalities to lower level modules

23. Error rate and system complexity
Card and Glass’s findings
Error rate = * System complexity
This indicates that each unit increase in system complexity increases the error rate by 0.4
Error rate in this case is defined as the number of errors per 1000 lines of code

24. Refinement of Card and Glass metrics
Refine Data complexity by including the complexities due to data structures and data types
Refine Structural complexity by including
Cyclomatic complexity along with fan-out values
Henry and Kafura’s structural complexity metrics
Henry and Selig’s structural complexity metrics

25. Metrics for Object-Oriented Design

26. Some OO Design metrics proposed by IBM (1993)
Rules of Thumb
Average method size (measured in terms of Lines Of Code)
Should be < 8 for Smalltalk and < 24 for C++
Average number of methods per class
Should be < 20
Average number of instance variables per class
Should be < 6
Depth of inheritance tree (DIT)
Number of class relationships in each subsystem
Should be relatively high.

27. Discussion
Lower LOC and lower DIT values support understandability and easy maintenance
Lower number of methods supports easy modification of a class; otherwise, the class must be split
More relationships between classes in a subsystem indicates high cohesion (??) of the subsystem
Marc Lorenz, IBM (1993)

28. OO metrics from six IBM projects
Project A(C++)
Project B(C++)
Project C(C++)
Project D(Sm)
Project E(Sm)
Project F(Sm)
# of classes
5741
2513
3000
100
566
492
# of methods per class 8 3 7 17 36 21
LOC per method 19 15
5.3
5.2
5.7
LOC per class
207 60 97
188
117
Max. DIT 6
N/A 5
Ave. DIT
4.8
2.8
© Stephen Kan, 2003

29. Chidamber & Kemerer’s OO Design metrics (1994)
Weighted Methods per Class (WMC)
Sum of complexities of all methods in the class; the complexity of each method can be its cyclomatic complexity
Depth of Inheritance Tree (DIT)
Length of the maximum path of the class hierarchy from the node to the root of the inheritance tree (see picture on the next slide)
Number of Children of a Class (NOC)
Number of immediate subclasses of a class

30. … Class A Level 0 Level 1 Class B Class C Class D Class E Level 2
Class F
Class G
Level 3
Class H
Class I
Level 4
DIT(Class I) = 4 for this inheritance hierarchy

31. Chidamber & Kemerer’s OO Design metrics (1994) (continued)
Coupling Between Object classes (CBO)
A class C1 is coupled with a class C2 if C1 invokes a method in C2 or uses instance variables of C2
Response For a Class (RFC)
Number of methods within a class C that are executed in response to a message received by C
Lack of Cohesion on Methods (LCOM)
Number of disjoint sets of local methods
Measures the dissimilarity of methods in the class by usage of instance variables
Indirectly, LCOM reflects how methods in a class are related to its instance variables

32. View points on C&K metrics
Weighted Methods per Class (WMC)
The development time and cost of an application depends on the number of classes and their complexities.
The larger the number of methods (and so the higher the complexities of methods) is, the greater the impact these methods have when this class is inherited.
The larger the number of methods is, the lower potential for reuse of this class.

33. View points on C&K metrics (continued)
Depth of Inheritance Tree (DIT)
The deeper a class is in the inheritance hierarchy, the more chances for reuse of methods from parent classes.
The deeper a class is in the inheritance hierarchy, the more methods inherited into the class. It increases the complexity of this class.
Deeper inheritance hierarchy also increases overall design complexity which is a significant factor for maintenance.

34. View points on C&K metrics (continued)
Number of Children (NOC)
The greater is the number of children, the greater are the chances for reuse.
The greater is the number of children, the least chances of maintaining abstraction from parent class.
The more children a class has, the more testing involved especially when polymorphism is used.

35. View points on C&K metrics (continued)
Coupling Between Object classes (CBO)
Excessive coupling between classes causes more difficulties in reuse and maintenance.
Excessive coupling also indicates high dependency and so more testing efforts are required.

36. View points on C&K metrics (continued)
Response For a Class (RFC)
If larger number of methods are invoked in response to a single message, testing and debugging become harder.
The larger is the number of methods invoked for a response, the greater is the complexity of the class.

37. View points on C&K metrics (continued)
Lack of Cohesion of Methods (LCOM)
Cohesion of a method within a class promotes encapsulation.
Lack of cohesion is an indication of why the class needs to be redesigned.
Lack of cohesion is an indirect measure of increase in complexity of the class. Hence it also indicates difficulties during design and maintenance.

38. Application of C&K metrics
Project A (C++)
Project B (Smalltalk)
WMC 5 10
DIT
1 (Max = 8)
3 (Max = 10)
NOC
RFC 6 29
CBO 9
LCOM
0 (Max = 200)
2 (Max = 17)
© Chidamber & Kemerer 1993, 1994; Brian Hendersen-Sellers 1995

39. Validation of C & K Metrics by Victor Basili and his team @ U of Maryland (1996)
The six C & K metrics are relatively independent
Lack of use of inheritance is reflected by low values of DIT and NOC
LCOM was not a proper metric because it lacked discrimination power in predicting faulty classes
DIT, RFC, NOC and CBO are significantly correlated with faulty classes
The metrics were superior to code metrics in predicting faulty classes

40. NASA Requirements Metrics

41. Seven Measures proposed by NASA
Number of lines of text
Used to measure the size of the document
Imperatives
Words and phrases that command something or indicate what must be provided
E.g., “shall”, “must”, “must not”, “will”, “should”, “required”, “are applicable”, “responsible for”
Indicate the testability of the functionalities in the requirements document

42. Seven Measures (continued)
Continuances
Phrases that introduce specification at a lower level
E.g., “following”, “as follows”, “below”, “listed”, “in particular”, “support”
Indicate contribution of the requirements document for tractability and maintenance
Also indicates how well the document is structured and organized

43. Seven Measures (continued)
Directives
Words and phrases that point to (or refer to) illustrative information within the requirements document.
E.g., “figure”, “table”, “for example”, “note”
Indicates the extent to which the document explains requirements in more detail
Contribution to the understandability of requirements

44. Seven Measures (continued)
Weak Phrases
Category of clauses that are apt to cause uncertainty and leave room for multiple interpretations
E.g., “adequate”, “as applicable”, as a minimum”, “as appropriate”, “easy”, “be capable of”, “capability of”, “effective”, “if practical”, “timely”, “but not limited to”
Indicate the extent to which the requirements document is ambiguous and incomplete

45. Seven Measures (continued)
Options
Category of words that give the developer latitude in satisfying the specification
E.g., “can”, “may”, “optionally”
Indicate the extent to which the designer has control over the specification
Indicate potential cost and schedule risks because of the control left for design and implementation stage

46. Seven Measures (continued)
Incompleteness
Words and phrases that deliberately leave portions of specification left unspecified
E.g., “TBD”
Indicate the looseness and incompleteness in the specification

47. Derived Measures Size Text Structure Specification depth Readability
Based on lines of text, words per line, words per functional requirement, etc.
Text Structure
Specification depth
Readability

48. Indicators of quality attributes – Table 1
Complete
Consistent
Correct
Modifiable
Imperatives X
Continuances
Directives
Options
Weak Phrases
Size
Text Structure
Specification Depth
Readability

49. Indicators of quality attributes – Table 2
Tractable
Unambiguous
Understandable
Imperatives X
Continuances
Directives
Options
Weak Phrases
Size
Text Structure
Specification Depth
Readability

50. Indicators of quality attributes – Table 3
Validatable
Verifiable
Imperatives X
Continuances
Directives
Options
Weak Phrases
Size
Text Structure
Specification Depth
Readability

51. Reference NASA web site http://satc.gsfc.nasa.gov/support
SATC – Software Assurance Technology Center
GSFC – Goddard Space Flight Center
Developed a tool called Automated Requirements Measurement (ARM)

52. Software Quality Metrics

53. Impact of Defects
Most organizations assume that a quality product is the one that has relatively less number of defects
Goals
Identification of defects
Fixing defects
Ensure that future products do not have such defects that were previously identified

54. Classifications Product metrics Process metrics Project metrics
Describe the characteristics of the product in terms of its size, complexity, performance, and defects (found, fixed, not-yet-fixed, ….) and so on.
Process metrics
Describe the characteristics of a software development process in terms of resources used, defects found, time to complete and so on. The goal is to improve the process for future projects.
Project metrics
Describe the characteristics of a software development project as a whole in order to compare and to improve the products.

55. Product Quality Metrics
Defect density
Mean-time to failure
Customer problems count
Customer satisfaction

56. Number of defects found
Defect density
The term Defect is used synonymously with Fault, Failure and Error
Defect density =
Product size can be measured in terms of Lines of Code (LOC), Function points, memory required, and so on.
Number of defects found
Product size

57. Mean time to failure (MTTF)
Based on the measurement of time intervals between failures
Example
Suppose there are three failures of a software product after it was released – once after the 1st month, once at the 6th month and once at the 9th month.
So, the durations between failures are 1, 5 and 3 and the mean of these durations is (1+5+3)/3 = 3.
MTTF for this product is 3 months.

58. Mean time to failure (MTTF) (continued)
MTTF is calculated after the product is released.
MTTF varies for different products and depends on customers. It also depends on the duration in which it is measured (for one year, for three years, and so on)
E.g., IBM, Rochester uses 4 years for Operating Systems and 2 years for application software to calculate MTTF.

59. Mean time to failure (MTTF) (continued)
Generally, MTTF is difficult to measure and it does not follow a predictable pattern.
For safety-critical systems, MTTF is a useful metric
E.g., US government mandates that its air traffic control system cannot unavailable for more than three seconds per year!!†
CMM level 4 or level 5 organizations use MTTF for application software systems.
† From Stephen Kan’s book

60. Customers problem count
Measures the number of problems reported by customers after the product is released.
One such metric is Problems per User Month (PUM) defined as
PUM = Total number of problems reported in a given duration DIVIDED BY Total number of license-months of the software
where
Total number of license-months = number of installed licenses for the products * number of months in the duration of PUM calculation

61. Customers problem count (continued)
Number of problems reported by customers include the defects found by customers as well as general problems such as not-understanding something, missing information, usability problems, etc.
PUM metric is not stable and is generally not predictable.

62. Customer satisfaction
Counting the results of feedback from customers
Generally done through surveys
Reported in counts, ratios, percentages and so on.

63. Process metrics
Typically called “in-process metrics” with respect to software quality measurement
Includes
Defect inspection efficiency
Defect removal efficiency

64. Defects Classification
Defects before release
Defects found until and including alpha testing
Post-release failures
Defects found during beta testing
Residual faults
Defects found during customer usage
Total faults
Every defect found from the initial development stage until now
Defects in a particular stage/phase of life cycle
E.g., requirement defects

65. Defect injection and identificationRQ AD DD CD UT CT ST
FLD
Total
RQR
ADR 49
681
730
DDR 6 42
729
CDR 12 28
114
941
1095 21 43
223 2
332 20 41 61
261 - 4
387 8 24 72 1
111 16 40 81
122
859
939
1537
3465 D E F C T O U N
© Stephen Kan, 2003

66. Legend RD Requirements RDR Requirements Review AD Architectural Design
ADR
Architectural Design Review DD
Detailed Design
DDR
Detailed Design Review CD
Code
CDR
Code Review UT
Unit Testing CT
Component Testing ST
System Testing
FLD
Field Testing (Beta)

67. Defect Inspection Efficiency
A Sample calculation
Architectural Design Inspection Efficiency =
Number of defects found during Arch. Des. Review
* 100
Number of defects injected until and including Arch. Des. stage
730
* 100 = % (approximately)
Do the calculations for each stage and find out its defect inspection efficiencies

68. Defect Removal Efficiency
A sample calculation
Code stage defect removal efficiency =
Number of defects removed until and including code stage
* 100
Number of defects injected until and including code stage
* 100 = % (approximately)
Do the calculations for the other stages

69. Software quality and defect removal efficiency
Number of defects found by customers
1 -
* 100
Total number of defects found by development team 81
1 -
* 100 = %
3465

70. Defect injection and identification
Requirements
Design
Code
Total
Requirements Review
Design Review 14
Code Review 3 21 24
Unit Testing 1 48 50
System Testing 17 18
Acceptance Testing 6 2 92
112 D E F C T O U N
© Pankaj Jalote, 2002