1. Simulation based verification: coverage
Verification coverage estimates the quality of simulation
Digitaalsüsteemide verifitseerimise kursus

2. Digitaalsüsteemide verifitseerimise kursus
Coverage metrics
System can’t be simulated with all possible stimuli
Question: how many stimuli are needed?
Coverage metrics used
Coverage shows our confidence in how thoroughly we verified
Digitaalsüsteemide verifitseerimise kursus

3. Digitaalsüsteemide verifitseerimise kursus
Coverage metrics
Three types:
Code coverage
Parameter coverage
Functional coverage
Also will talk about
Domain coverage, cross coverage, stuck-at fault coverage
Digitaalsüsteemide verifitseerimise kursus

4. Coverage metrics: code coverage
Code coverage shows how thoroughly is the code examined during simulation
Based on implementation only
Easy to compute
Digitaalsüsteemide verifitseerimise kursus

5. Coverage metrics: parameter coverage
How well parameters and dimensions covered in functional blocks
Based on implementation, spec or both
Digitaalsüsteemide verifitseerimise kursus

6. Coverage metrics: functional coverage
Based on behaviors derived from the spec
Shows the number of covered behaviors
Problem: how to formally generate the behaviors and guarantee that the full functionality is covered…
Digitaalsüsteemide verifitseerimise kursus

7. Comparison: coverage metrics
None of the three types is universally best
Advantages of code coverage are simplicity and formalisation
Advantage of functional coverage is high verification power
Parameter coverage is a trade-off between the two
Coverage metrics may be used for guiding test generation
To reach 100 % coverage is very much harder than to reach 90 %...
Digitaalsüsteemide verifitseerimise kursus

8. Code coverage: statement coverage
All lines except begin, end, else
Example: a > x, x = y
Statement coverage 8 out of 10 = 80%
Digitaalsüsteemide verifitseerimise kursus

9. Code coverage: block coverage
Digitaalsüsteemide verifitseerimise kursus

10. Digitaalsüsteemide verifitseerimise kursus
Instrumented code
Instrumented code applied in code coverage analysis. Thus, simulation slowed down considerably.
Turn off the code already covered by simulation
In that case almost no extra time penalty towards the end of simulation
Digitaalsüsteemide verifitseerimise kursus

11. Digitaalsüsteemide verifitseerimise kursus
Instrumented code
Digitaalsüsteemide verifitseerimise kursus

12. Code coverage: branch and path
More stringent than statement coverage
Path coverage does not scale
Digitaalsüsteemide verifitseerimise kursus

13. Code coverage: expression coverage
Again, more stringent than statement coverage
We analyze which expressions were evaluated
Expressions divided into layers:
E = (x1x2 + x3x4)
layer 1: E = y1 + y2
layer 2: y1 = x1x2, y2 = x3x4
With multi-layer expressions, either the top layer or all layers analyzed
Digitaalsüsteemide verifitseerimise kursus

14. Code coverage: expression coverage
Minimum input tables applied:
f = x1 & x2 x1 x2 - 1
f = (y ? x1 : x2) y
f = (x > y)
x > y
meaning
x <= y
Expression x1 & x2 and two input stimuli: (x1=1,x2=0) and (x1=1,x2=1)
In that case the expression coverage for & is 66.67%
Digitaalsüsteemide verifitseerimise kursus

15. Code coverage: state coverage
Sequence b,a,b,a,b,a,b,a,... gives 66.67% of state coverage, as S1 and S2 traversed (2 out of 3).
S4 is illegal state, thus max state coverage is 75%!
Digitaalsüsteemide verifitseerimise kursus

16. Code coverage: transition coverage
Sequence b,a,b,a,b,a,b,a,... gives 40% of transition coverage as 2 out of 5 transitions traversed.
Transition S2/b not defined!
Calculated by transitions specified (5) or all possible transitions (6)
Digitaalsüsteemide verifitseerimise kursus

17. Code coverage: sequence coverage
User can specify state sequences at the basis of the coverage calculation
Normally essential functionality or corner cases specified
Possible to keep track on which desired (or undesired) sequences were simulated
Digitaalsüsteemide verifitseerimise kursus

18. Code coverage: toggle coverage
Measures how many bits in the signals changed their state between 0 and 1
E.g. a signal for a 4-bit bus and values 0000 and 1101 would give a toggle coverage of 75%
Digitaalsüsteemide verifitseerimise kursus

19. Digitaalsüsteemide verifitseerimise kursus
Parameter coverage
Parameter
Range
Server configuration
{(S1,S2): S1, S2 passive or active}
Task processing time
{3,...,6}
Length of the queue
{(m1,m2,m3): 1   mi  8, 1  i   3 }
Digitaalsüsteemide verifitseerimise kursus

20. Parameter coverage: example
Range
{(S1,S2): S1, S2 passive or active}
{3,...,6}
{(m1,m2,m3): 1   mi  8, 1  i   3 }
Server configuration
Task processing time
Length of the queue
If simulation provides 2 server configurations: (S1 active, S2 passive) ja (S1 active, S2 active); processing times 3, 4 and 5; maximum que lengths (6,4,8), then ...
... Parameter coverages: server conf. 50 %, processing times 75 %, queue lengths 75%,50%,100%
Digitaalsüsteemide verifitseerimise kursus

21. Digitaalsüsteemide verifitseerimise kursus
Functional coverage
Functional coverage based on behaviors derived from the specification
Functional coverage reports the ratio of covered behaviors
Problem: how to formally generate behaviors and guarantee that the entire functionality is covered
Digitaalsüsteemide verifitseerimise kursus

22. Digitaalsüsteemide verifitseerimise kursus
Functional coverage
User defined mapping of each functional feature to a ‘cover point’ 
Cover points have certain conditions (ranges, defined transitions or cross etc.)
Conditions for a cover point are defined in form of ‘bins’.
Digitaalsüsteemide verifitseerimise kursus

23. Digitaalsüsteemide verifitseerimise kursus
Functional coverage
During simulation, when conditions of a cover point hit, those bins are covered.
A number of cover points can be captured under one cover group’
Collection of cover groups is usually called a ‘functional coverage model’. 
Digitaalsüsteemide verifitseerimise kursus

24. Digitaalsüsteemide verifitseerimise kursus
Cross coverage
Cartesian product of 2 or more coverages
More stringent than using just a single coverage metric
Cartesian product of several coverages may represent a huge space
Digitaalsüsteemide verifitseerimise kursus

25. Domain coverage (boundary coverage)
Every condition defines a partition of the input domain
Small design error may slightly shift the partitioning
in1
in2
out=0
out=1
foo (in1, in2)
if (in1 < in2)
out = 1;
else
out = 0;
What if in1 < in2 should be rather in1 <= in2?
What if the value of in1 or in2 is slightly wrong?
Digitaalsüsteemide verifitseerimise kursus

26. Domain coverage: Setting up a test
Testing with a value at the boundary and with a value immediately off the boundary
in1
in2
out=0
out=1 1 2
foo (in1, in2)
if (in1 < in2)
out = 1;
else
out = 0;
Test 1: in1, in2 = 1,1 – false, at the boundary
Test 2: in1, in2 = 1,2 – true, immediately off the boundary
Digitaalsüsteemide verifitseerimise kursus

27. Domain coverage: example
Per each condition (predicate) two tests:
True and immediately off the boundary
False and immediately off the boundary
Test 1: in1, in2 = 0,0
c = 0, and a = 0
false and at the boundary for c < a
false and at the boundary for c < in2
a = in1 + in2;
b = 0; c = 0;
while (c < a)
c = c + in1;
if (c < in2)
out = a + b;
else
out = a + c;
Test 2: in1, in2 = 1,0
c = 0, and a = 1
true and at the boundary for c < a
c set to 1
false and at the boundary for c < in2
3 conditions out of 4 satisfied, 75% domain coverage
Digitaalsüsteemide verifitseerimise kursus

28. Issues with domain coverage
Non-linear boundaries
Partly linear boundaries: test each part separately
in1
in2 2
(in2 > in1) and (in1 > 2)
Higher order boundaries: no standard solutions
in1
in2
(in2 > in12)
Digitaalsüsteemide verifitseerimise kursus

29. Digitaalsüsteemide verifitseerimise kursus
Observation coverage
Code coverage ignores the observability aspect
It may happen that the covered code items have no influence to any observable point
It has been showed that 100% code coverage may result in 60-80% observation coverage in the worst case [Fallah, Devadas, Keutzer DAC’98].
Thus, we should consider propagating the bug effect to an observable output
Digitaalsüsteemide verifitseerimise kursus

30. Other metrics: stuck-at coverage
Assume that a bug manifests itself as a signal line being constantly stuck to 0 (or 1)
Assune single faults, i.e. only one signal line is faulty at a time
Stuck-at coverage indicates how well we have excercised the structure
Digitaalsüsteemide verifitseerimise kursus

31. Digitaalsüsteemide verifitseerimise kursus
Issues with coverage
None of the coverage metrics provides 100% of confidence in correct functionality
E.g. 100 % stuck-at coverage test for OR gate does not discover the design error OR XOR x1 x2
x1 V x2 1 x1 x2
x1  x2 1 =
Digitaalsüsteemide verifitseerimise kursus

32. Digitaalsüsteemide verifitseerimise kursus
Issues with coverage
Similar problem with expression coverage
100 % expression coverage for OR gate does not discover the error OR XOR x1 x2
x1 V x2 1 x1 x2
x1  x2 1 =
Thus, we need to consider a set of coverages!
Digitaalsüsteemide verifitseerimise kursus