Simulation based verification: coverage Verification coverage estimates the quality of simulation Digitaalsüsteemide verifitseerimise kursus

Digitaalsüsteemide verifitseerimise kursus Coverage metrics System can’t be simulated with all possible stimuli Question: how many stimuli are needed? Coverage metrics used Coverage shows our confidence in how thoroughly we verified Digitaalsüsteemide verifitseerimise kursus

Digitaalsüsteemide verifitseerimise kursus Coverage metrics Three types: Code coverage Parameter coverage Functional coverage Also will talk about Domain coverage, cross coverage, stuck-at fault coverage Digitaalsüsteemide verifitseerimise kursus

Coverage metrics: code coverage Code coverage shows how thoroughly is the code examined during simulation Based on implementation only Easy to compute Digitaalsüsteemide verifitseerimise kursus

Coverage metrics: parameter coverage How well parameters and dimensions covered in functional blocks Based on implementation, spec or both Digitaalsüsteemide verifitseerimise kursus

Coverage metrics: functional coverage Based on behaviors derived from the spec Shows the number of covered behaviors Problem: how to formally generate the behaviors and guarantee that the full functionality is covered… Digitaalsüsteemide verifitseerimise kursus

Comparison: coverage metrics None of the three types is universally best Advantages of code coverage are simplicity and formalisation Advantage of functional coverage is high verification power Parameter coverage is a trade-off between the two Coverage metrics may be used for guiding test generation To reach 100 % coverage is very much harder than to reach 90 %... Digitaalsüsteemide verifitseerimise kursus

Code coverage: statement coverage All lines except begin, end, else Example: a > x, x = y Statement coverage 8 out of 10 = 80% Digitaalsüsteemide verifitseerimise kursus

Code coverage: block coverage Digitaalsüsteemide verifitseerimise kursus

Digitaalsüsteemide verifitseerimise kursus Instrumented code Instrumented code applied in code coverage analysis. Thus, simulation slowed down considerably. Turn off the code already covered by simulation In that case almost no extra time penalty towards the end of simulation Digitaalsüsteemide verifitseerimise kursus

Digitaalsüsteemide verifitseerimise kursus Instrumented code Digitaalsüsteemide verifitseerimise kursus

Code coverage: branch and path More stringent than statement coverage Path coverage does not scale Digitaalsüsteemide verifitseerimise kursus

Code coverage: expression coverage Again, more stringent than statement coverage We analyze which expressions were evaluated Expressions divided into layers: E = (x1x2 + x3x4) layer 1: E = y1 + y2 layer 2: y1 = x1x2, y2 = x3x4 With multi-layer expressions, either the top layer or all layers analyzed Digitaalsüsteemide verifitseerimise kursus

Code coverage: expression coverage Minimum input tables applied: f = x1 & x2 x1 x2 - 1 f = (y ? x1 : x2) y f = (x > y) x > y meaning x <= y Expression x1 & x2 and two input stimuli: (x1=1,x2=0) and (x1=1,x2=1) In that case the expression coverage for & is 66.67% Digitaalsüsteemide verifitseerimise kursus

Code coverage: state coverage Sequence b,a,b,a,b,a,b,a,... gives 66.67% of state coverage, as S1 and S2 traversed (2 out of 3). S4 is illegal state, thus max state coverage is 75%! Digitaalsüsteemide verifitseerimise kursus

Code coverage: transition coverage Sequence b,a,b,a,b,a,b,a,... gives 40% of transition coverage as 2 out of 5 transitions traversed. Transition S2/b not defined! Calculated by transitions specified (5) or all possible transitions (6) Digitaalsüsteemide verifitseerimise kursus

Code coverage: sequence coverage User can specify state sequences at the basis of the coverage calculation Normally essential functionality or corner cases specified Possible to keep track on which desired (or undesired) sequences were simulated Digitaalsüsteemide verifitseerimise kursus

Code coverage: toggle coverage Measures how many bits in the signals changed their state between 0 and 1 E.g. a signal for a 4-bit bus and values 0000 and 1101 would give a toggle coverage of 75% Digitaalsüsteemide verifitseerimise kursus

Digitaalsüsteemide verifitseerimise kursus Parameter coverage Parameter Range Server configuration {(S1,S2): S1, S2 passive or active} Task processing time {3,...,6} Length of the queue {(m1,m2,m3): 1   mi  8, 1  i   3 } Digitaalsüsteemide verifitseerimise kursus

Parameter coverage: example Range {(S1,S2): S1, S2 passive or active} {3,...,6} {(m1,m2,m3): 1   mi  8, 1  i   3 } Server configuration Task processing time Length of the queue If simulation provides 2 server configurations: (S1 active, S2 passive) ja (S1 active, S2 active); processing times 3, 4 and 5; maximum que lengths (6,4,8), then ... ... Parameter coverages: server conf. 50 %, processing times 75 %, queue lengths 75%,50%,100% Digitaalsüsteemide verifitseerimise kursus

Digitaalsüsteemide verifitseerimise kursus Functional coverage Functional coverage based on behaviors derived from the specification Functional coverage reports the ratio of covered behaviors Problem: how to formally generate behaviors and guarantee that the entire functionality is covered Digitaalsüsteemide verifitseerimise kursus

Digitaalsüsteemide verifitseerimise kursus Functional coverage User defined mapping of each functional feature to a ‘cover point’  Cover points have certain conditions (ranges, defined transitions or cross etc.) Conditions for a cover point are defined in form of ‘bins’. Digitaalsüsteemide verifitseerimise kursus

Digitaalsüsteemide verifitseerimise kursus Functional coverage During simulation, when conditions of a cover point hit, those bins are covered. A number of cover points can be captured under one cover group’ Collection of cover groups is usually called a ‘functional coverage model’.  Digitaalsüsteemide verifitseerimise kursus

Digitaalsüsteemide verifitseerimise kursus Cross coverage Cartesian product of 2 or more coverages More stringent than using just a single coverage metric Cartesian product of several coverages may represent a huge space Digitaalsüsteemide verifitseerimise kursus

Domain coverage (boundary coverage) Every condition defines a partition of the input domain Small design error may slightly shift the partitioning in1 in2 out=0 out=1 foo (in1, in2) if (in1 < in2) out = 1; else out = 0; What if in1 < in2 should be rather in1 <= in2? What if the value of in1 or in2 is slightly wrong? Digitaalsüsteemide verifitseerimise kursus

Domain coverage: Setting up a test Testing with a value at the boundary and with a value immediately off the boundary in1 in2 out=0 out=1 1 2 foo (in1, in2) if (in1 < in2) out = 1; else out = 0; Test 1: in1, in2 = 1,1 – false, at the boundary Test 2: in1, in2 = 1,2 – true, immediately off the boundary Digitaalsüsteemide verifitseerimise kursus

Domain coverage: example Per each condition (predicate) two tests: True and immediately off the boundary False and immediately off the boundary Test 1: in1, in2 = 0,0 c = 0, and a = 0 false and at the boundary for c < a false and at the boundary for c < in2 a = in1 + in2; b = 0; c = 0; while (c < a) c = c + in1; if (c < in2) out = a + b; else out = a + c; Test 2: in1, in2 = 1,0 c = 0, and a = 1 true and at the boundary for c < a c set to 1 false and at the boundary for c < in2 3 conditions out of 4 satisfied, 75% domain coverage Digitaalsüsteemide verifitseerimise kursus

Issues with domain coverage Non-linear boundaries Partly linear boundaries: test each part separately in1 in2 2 (in2 > in1) and (in1 > 2) Higher order boundaries: no standard solutions in1 in2 (in2 > in12) Digitaalsüsteemide verifitseerimise kursus

Digitaalsüsteemide verifitseerimise kursus Observation coverage Code coverage ignores the observability aspect It may happen that the covered code items have no influence to any observable point It has been showed that 100% code coverage may result in 60-80% observation coverage in the worst case [Fallah, Devadas, Keutzer DAC’98]. Thus, we should consider propagating the bug effect to an observable output Digitaalsüsteemide verifitseerimise kursus

Other metrics: stuck-at coverage Assume that a bug manifests itself as a signal line being constantly stuck to 0 (or 1) Assune single faults, i.e. only one signal line is faulty at a time Stuck-at coverage indicates how well we have excercised the structure Digitaalsüsteemide verifitseerimise kursus

Digitaalsüsteemide verifitseerimise kursus Issues with coverage None of the coverage metrics provides 100% of confidence in correct functionality E.g. 100 % stuck-at coverage test for OR gate does not discover the design error OR XOR x1 x2 x1 V x2 1 x1 x2 x1  x2 1 = Digitaalsüsteemide verifitseerimise kursus

Digitaalsüsteemide verifitseerimise kursus Issues with coverage Similar problem with expression coverage 100 % expression coverage for OR gate does not discover the error OR XOR x1 x2 x1 V x2 1 x1 x2 x1  x2 1 = Thus, we need to consider a set of coverages! Digitaalsüsteemide verifitseerimise kursus