1. When is testing sufficient?
Test coverage criteria I

2. We will learn… How to measure test coverage?
Different types of Control flow criteria
Statement
Branch
Path
Condition
Multiple condition adequacy
Excercise

3. Only 1 week left!
Presentation topic must be agreed with the lecturer (by on in the seminar)
Deadline for agreeing the topic and presentation time – March 4th (6th week)
Presentations starting from March 11th (7th week)

4. Finishing testing Usually When the project runs out of time/money
Infinite set of possible tests (exhaustive testing)
Difficulties of objective measurement of what has been tested
The need for criteria for choosing finite amount of tests
Test coverage criteria
Shows, what parts of the SUT/IUT have been excecuted on tested
The set of tests based on coverage criteria is finite (mostly) - >
Possibility to measure the amount of tests (%) excecuted during testing
Eg. We have 100 possible tests, we have run 75 of them (75%)

5. Coverage criterion … aka adequacy criterion
Must include a measurable element – coverage item
Statement coverage – all statements in the SUT must be excecuted at least once to achieve 100% statement coverage
Some parts of the program may be infeasible (no possible inputs for excecuting a certain path)
Some criteria still define infinite set of test (path-criteria)

6. Different criteria
Black-box – test cases are based on functionality analysis
Input/output equivalence partitions
Boundary values
...
White-box – test cases are based on SUT program structure
Statement, branch, path adequay

7. Equivalence partitions
Equivalence partition is a set of values, so that for every x in this partition P(x) is the same
In other words: the SUT behaves exactly the same with every value in one equivalence partition
Example
Input triangle edges a, b, c
Program answers, whether or not it is possible to form a triangle and the triangle what type

8. Triangle example: equivalence partitions
Input: “success story” (main flow)
All inputs are numeric AND ]0; ∞}
Input: “incorrect values” (alternative flow)
At least one edge is not numeric (incl. NULL)
]-∞;0[
Boundary values: 0, 0.01, -0.1
Output:
Isoceles (a==b || a==c || b ==c)
Equilateral (a==b & b==c)
Scalene (a!=b!=c)
A triangle can not be formed (Heron’s formula)

9. Combining equivalence partitions
Main flow equivalence partitions can be tested all at once:
P(3,4,5), P(1,2,3)
Alternative flows must be tested each class separately
P(-1, 2, 3), P(1, -2, 3), P(1, 2, -3) s.o
Combining boundary values depends on the situation
NB! Equivalence partitions depend on the technology (web, desktop, mobile…)
CAPITAL/small letters, diacritical marks, special symbols (&?%)
Minimal and maximal length/value
Is NULL allowed?
Memory constraints (mobile)
Connectivity issues (mobile)
...

10. White box structural testing
Control-flow coverage
Based on program control structures – logical expressions, branches, cycles
IF x=2 THEN ... ELSE
Data-flow coverage
Based on assigning values to variables and how values affect the excecution of parts of the program
X=2 -> do_this(), X!=2 -> do_that()
The difference is related to choosing tests only! Because control structure and data can not be separated!
Control flow – kattekriteeriumid juhinduvad programmi juhtstruktuuridest (loogikaavaldised, millest tekivad hargnemised ja tsüklid)
Andmevoog – lähtuvad muutujatest ning nende väärtuste omistamistest ning sellest, kuidas need omistamised mõjutavad programmi täitmist

11. White box structural coverage criteria
Control flow coverage
Statement adequacy
Branch adequacy
Path adequacy
Condition adequacy
Multiple-condition adequacy
...
Data flow coverage
All-definitions,
All-uses
All-p-uses, All-c-uses
All Def-use paths

12. White box structural coverage
Based on program formalisation (~program)
Graph ~
FSM (Finite State Machine) ~
EFSM (Extended FSM) ~
State machine diagram, activity diagram ~
...

13. Statement adequacy Statement coverage, All Nodes coverage
Every excecutable program statement is excecuted at least once
A=2, B=0, X=3

14. Branch adequacy Branch coverage, All edges or decision coverage
A=3,B=0,X=3
A=2,B=1,X=1

15. Cyclomatic complexity
Cyclomatic complexity V(g) (McCabe, 1976)
Based on branches in program structure (graph)
V(g) = E-N+2 (edges, nodes)
V(g) = P+1 (predicates)
V(g) is equal to recommended count of branch adequacy tests
Helps to estimate complexity of constructing a program
Recommended cyclomatic complexity of a module is ≤ 10

16. Path adequacy Path coverage All paths in the program are traversed
Mostly too many (infinitely) paths to be used in reality
Infinite number of paths when loops are included
5 paths in a loop
5n paths! (n – number of excecuting the loop)

17. Condition coverage IF (A>1)&(B=0) THEN X=X/A; END;
A set of tests, where in every decision all conditions have all possible values
A=1, B=0, X=3
FALSE, TRUE
A=2, B=1, X=0
TRUE, FALSE
IF (A>1)&(B=0) THEN X=X/A; END;
IF (A=2)|(X>1) THEN X=X+1; END;

18. Multiple-condition coverage
Combinations of all possible outcomes of elementary conditions
A>1,B=0
A>1,B≠0
A1,B=0
A1,B≠0
A=2,X>1
A=2,X1
A≠2,X>1
A≠2,X1
A=2,B=0,X=4
A=2,B=1,X=1
A=1,B=0,X=2
A=1,B=1,X=1
IF (A>1)&(B=0) THEN X=X/A; END;
IF (A=2)|(X>1) THEN X=X+1; END;

19. Excercise For a given program, find Cyclomatic complexity V(g)
How many tests are needed for branch coverage?
What tests (which data, eg values for a, b, c) are needed for branch coverage tests?

20. Answers V(g) = (Edges-Nodes+2) = 6-6+2 = 2
Thus, 2 tests are needed for branch coverage
Data:
b>a (“down” transition) -> (1,2,3)
b≤a (“right” transition) -> (2,1,3) OR (1,1,3)
Value of c does not matter in this concext