Software Test Metrics When you can measure what you are speaking about and express it in numbers, you know something about it; but when you cannot measure, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind: it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science. (reference: unknown)

Measure - quantitative indication of extent, amount, dimension, capacity, or size of some attribute of a product or process. Metric - quantitative measure of degree to which a system, component or process possesses a given attribute. “A handle or guess about a given attribute.”

Why We need Metrics? “You cannot improve what you cannot measure.” “You cannot control what you cannot measure” AND TEST METRICS HELPS IN – Take decision for next phase of activities – Evidence of the claim or prediction – Understand the type of improvement required – Take decision on process or technology change – Improve software quality – Anticipate and reduce future maintenance needs

Metric Classification

Products – Explicit results of software development activities – Deliverables, documentation, by products Processes – Activities related to production of software Resources – Inputs into the software development activities – hardware, knowledge, people

Product vs. Process Process Metrics – Insights of process paradigm, software engineering tasks, work product, or milestones – Lead to long term process improvement Product Metrics – Assesses the state of the project – Track potential risks – Uncover problem areas – Adjust workflow or tasks – Evaluate teams ability to control quality

Types of Measures Direct Measures (internal attributes) – Cost, effort, LOC, speed, memory Indirect Measures (external attributes) – Functionality, quality, complexity, efficiency, reliability, maintainability

Size-Oriented Metrics Size of the software produced LOC - Lines Of Code KLOC Lines Of Code SLOC – Statement Lines of Code (ignore whitespace) Typical Measures: – Errors or Defects/KLOC, Cost/LOC, Documentation Pages/KLOC

McCabe’s Complexity Measures McCabe’s metrics are based on a control flow representation of the program. A program graph is used to depict control flow. Nodes represent processing tasks (one or more code statements) Edges represent control flow between nodes

Steps for CC: 1: Draw a control flow graph 2: Calculate Cyclomatic complexity (CC) 3: Choose a “basis set” of paths – A basis path is a unique path through the software where no iterations are allowed - all possible paths through the system are linear combinations of them. 4: Generate test cases to exercise each path

Control Flow Graph – Any procedural design can be translated into a control flow graph – Lines (or arrows) called edges represent flow of control – Circles called nodes represent one or more actions – Areas bounded by edges and nodes called regions – A predicate node is a node containing a condition

Control Flow Graph Structures Basic control flow graph structures:

Cyclomatic Complexity Set of independent paths through the graph (basis set) V(G) = E – N + 2 – E is the number of flow graph edges – N is the number of nodes V(G) = P + 1 – P is the number of predicate nodes

Example One int example1 (int value, boolean cond1, boolean cond2) { 1 if ( cond1 ) 2 value ++; 3 if ( cond2 ) 4 value --; 5 return value; } V(G) = E – N + 2; 6 – 5+2 Complexity = 3 Basis PathsTest Data false false true false true true

Cyclomatic Complexity - Exercise - Calculate CC for the following Control flow graphs

Solution Cyclomatic complexity for the two control flow graphs; V(G) = edges - nodes + 2 – 12 edges – 9 nodes – =5 – 13 edges – 10 nodes – =5

Predicate Nodes Formula One possible way of calculating V(G) is to use the predicate nodes formula: V(G) = Number of Predicate Nodes + 1 Thus for this particular graph: V(G)=1+1=2

A limitation on the predicate nodes formula is that it assumes that there are only two outgoing flows for each of such nodes.

Basis Path Testing During Integration Basis path testing can also be applied to integration testing when units/components are integrated together McCabe’s Design Predicate approach: Draw a “structure chart” Calculate integration complexity Select tests to exercise every “type” of interaction, not every combination

Types of Interactions Types of unit/component interactions:

Unconditional – Unit/component A always calls unit/component B. A calling tree always has an integration complexity of at least one. The integration complexity is NOT incremented for each occurrence of an unconditional interaction. Conditional – Unit/component A calls unit/component B only if certain conditions are met. The integration complexity is incremented by one for each occurrence of a conditional interaction in the structure chart. Mutually Exclusive both) – Unit/component A calls either unit/component B or unit/component C (but not based upon certain conditions being met.) The integration complexity is incremented by one for each occurrence of a mutually exclusive conditional interaction in the structure chart. Iterative – Unit/component A calls unit/component B one or more times based upon certain conditions being met. The integration complexity is incremented by one for each occurrence of an iterative interaction in the structure chart.

Integration Basis Path Test Cases Basis path testing can also be applied to integration testing when units/components are integrated together Complexity: 5 Basis set of paths: – 1. A,B,C,E & then A calls I – 2. A,B,C,F & then A calls I – 3. A,B,D,G (only once) & then A calls I – 4. A,B,D,G (more than once) & then A calls I – 5. A,B,D,G,H & then A calls I

Integration Basis Path Testing Interaction types: – 2 conditional – 1 mutually exclusive conditional – 1 iteration – 1 simply because we have a graph Basis set of paths: 1. A,C,F (only once) 2. A,C,F (more than once) 3. A,B,D,G & then B calls E & then A calls C 4. A,B,D,H & then B calls E & then A calls C,F 5. A,B,D,H & then B calls E, I & then A calls C,F

Test Case defect density Total number of errors found in test scripts v/s developed and executed. (Defective Test Scripts /Total Test Scripts) * 100 Example: Total test script developed 1360, total test script executed 1280, total test script passed 1065, total test script failed 215 So, test case defect density is 215 X = 16.8% 1280 This 16.8% value can also be called as test case efficiency %, which depends upon total number of test cases which uncovered defects ***( =215)

Review Efficiency The Review Efficiency is a metric that offers insight on the review quality and testing Some organization also use this term as “Static Testing” efficiency and they are aiming to get min of 30% defects in static testing Review efficiency=100*Total number of defects found by reviews/Total number of project defects Example: A project found total 269 defects in different reviews, which were fixed and test team got 476 defects which were reported and valid So, Review efficiency is [269/( )] X 100 = 36.1%

Defect Removal Effectiveness DRE= Defects removed during development phase x100% Defects latent in the product Defects latent in the product = Defects removed during development phase+ defects found later by user The Defect Removal Effectiveness for each of the phases would be as follows: Requirements DRE = 10 / ( ) x 100% = 63% Design DRE = (3+18) / ( ) x 100% = 60% Coding DRE = (0+4+26) / ( ) x 100% = 55% Testing DRE = (2+5+8) / ( ) x 100% = 60% Development DRE = (Pre-release Defect) / (Total Defects) x 100% ( ) / ( ) x 100 = 88% Example:

According to Capers Jones, world class organizations have Development DRE greater than 95%. The longer a defect exists in a product before it is detected, the more expensive it is to fix. Knowing the DRE for each phase can help an organization target its process improvement efforts to improve defect detection methods where they can be most effective.