Structural Coverage Verilog code is available to help generate tests o Code can be analyzed statically and/or simulated Easier to detect “additive” design errors o Tests what the code actually does, not what it is supposed to do Possible to automate test generation

Structural Coverage Metrics Coverage metric is a numerical measurement of the adequacy of an arbitrary test set for verification. High coverage is a goal for test generation The measure is based on code execution 1.Toggle Coverage - Fraction of signals which have toggled 2.Statement coverage - Fraction of statements executed 3.Branch coverage - Fraction of branch directions executed 4.Path coverage - Fraction of control-flow paths executed

Toggle Coverage Counts the fraction of signals/variables whose bits have toggled ex. x = 4b0000 and x=4b0111, 75% toggle coverage Easy to compute Little functional meaning (came from manufacturing/memory test)

Statement Coverage Ensure that the “code” is completely covered during testing Statement Coverage: Requires that all statements are executed a = in1 + in2; b = 0; c = 0; while (c < a) c = c + in1; if (c < in2) out = a + b; else out = a + c; a = in1 + in2; b = 0; c = 0; c<a c<in2 c = c + in1; out = a + c;out = a + b; Test 1: in1, in2 = 0,0 Test 2: in1, in2 = -1, 1 TF T F 0,0 -1,1

Branch Coverage Branch Coverage: Requires that all branch directions are executed 2 branches, 4 possibilities, 3 are executed, 75% branch coverage a = in1 + in2; b = 0; c = 0; while (c < a) c = c + in1; if (c < in2) out = a + b; else out = a + c; a = in1 + in2; b = 0; c = 0; c<a c<in2 c = c + in1; out = a + c;out = a + b; Test 1: in1=0, in2=0 Test 2: in1=-1, in2=1 TF T F 0,0 -1,1

Test Sequence for Higher Coverage 100% statement and branch coverage High coverage goal directs test generation a = in1 + in2; b = 0; c = 0; while (c < a) c = c + in1; if (c < in2) out = a + b; else out = a + c; a = in1 + in2; b = 0; c = 0; c<a c<in2 c = c + in1; out = a + c;out = a + b; Test 1: in1,in2 = 1,0 Test 2: in1,in2 = -1,1 TF T F -1,1 1,0

Path Coverage Each pattern covers a different control path Infinite number of paths with a loop a = in1 + in2; b = 0; c = 0; c<a c<in2 c = c + in1; out = a + c;out = a + b; TF T F -1,1 1,0 0,0

Domain (Boundary) Coverage Each conditional predicate defines partitions the input space A small error may move the partition foo (in1, in2) if (in1 < in2) out = 1; else out = 0; in1 in2 out=0 out=1 What if in1 < in2 should be in1 <= in2? What if in1 or in2 has a slightly incorrect value?

Domain Test Goal Test with input data on the boundary and just on either side of the boundary foo (in1, in2) if (in1 < in2) out = 1; else out = 0; in1 in2 out=0 out=1 Test 1: in1, in2 = 1,1 - False, on the boundary Test 2: in1, in2 = 1,2 - True, just off the boundary 1 1 2

Domain Test Example a = in1 + in2; b = 0; c = 0; while (c < a) c = c + in1; if (c < in2) out = a + b; else out = a + c; Test 1: in1, in2 = 0,0 Sets c = 0, and a = 0 False and on-boundary for c < a False and on-boundary for c < in2 Test 2: in1, in2 = 1,0 Sets c = 0, and a = 0 True and near-boundary for c < a c set to 1 in loop False and on-boundary for c < in2 Need to satisfy 2 conditions for each predicate:  True and on/near-boundary  False and on/near-boundary 3 of 4 conditions satisfied, 75% coverage

Domain Test Complications Non-standard Domain Boundaries Piecewise Linear Boundaries: Test each linear piece separately (in2 > in1) and (in1 > 2) in1 in2 2 Higher Degree Boundaries: No standard solution in1 in2 (in2 > in1 2 )

Finite State Machine Metrics These metrics assume that the HDL is written in an FSM style o A ‘case’ statement which switches on a single ‘state’ var o Each control path assigns the state variable case state is when 0 => x = 0; state = 3; when 1 => x = 1; state = 2;. State Coverage - All states are entered Transition Coverage = All transitions are traversed

a = in1 + in2; b = 0; c = 0; while (c < a) c = c + in1; if (c < in2) out = a + b; else out = a + c; out = a - c; Erroneous statement Test 0,0 executes erroneous statement but does not detect the error Detection constraint includes a - c != a + c Optimism in a Coverage Metric 100% statement coverage does not guarantee error detection Test constraints may not be the same as detection constraints

Pessimism in a Coverage Metric 100% coverage may not be possible  Fault detection conditions may not be satisfiable  Undetectable faults are redundant, can be ignored  Identifying redundant faults in NP-complete a = in1 + in2; b = 0; c = 0; c<a c<in2 c = c + in1; out = a + c;out = a + b; TF T F This path cannot be executed Sub-100% coverage is misleading

Functional Coverage Test goals are defined based on the features in the specification The implementation (HDL) is not used to create tests Coverage events are “interesting” events which might reveal errors Coverage events must be defined manually because they are extracted from the specification Ex. Did overflow occur? Was x=17? Did an add follow a subtract?

Functional Coverage Results add sub shl shr noop illegal port1 port2port3port4 X X X X Status report indicates which events have been covered (and how often) Can compute the cross product of events to get complex events Many such events may be impossible or uninteresting