Lecture 9: Testbench and Division UCSD ECE 111 Prof. Farinaz Koushanfar Fall 2018 UCSD ECE 111, Prof. Koushanfar, Fall 2018 Some slides are courtesy of Prof. Lin

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Testbench UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Testbench A testbench is an HDL module that is used to test another module, called device under test (DUT). Testbench contains statements to apply inputs to the DUT and, ideally, to check that the correct outputs are produced. The input and desired output patterns are called test vectors UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 DUT module sillyfunction ( input logic a, b, c, output logic y); assign y = ~a & ~b & ~c | a & ~b & ~c | a & ~b & c; endmodule Create a module to be tested This is a simple module, so we can perform exhaustive testing by applying all eight possible test vectors UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Simple Testbench module testbench1() logic a, b, c, y; // instantiate device under test sillyfunction dut(a, b, c, y); // apply inputs one at a time intial begin a = 0; b = 0; c = 0;#10; $display("a=%b, b=%b, c=%b, y=%b", a, b, c, y); c = 1; #10; b = 1; c = 0; #10; a = 1; b = 0; c = 0;#10; b = 1; c = 0; #10; end endmodule First applies the input pattern 000 and waits for 10 time units Then applies 001 and waits for 10 more units So forth until all eight possible inputs have been applied UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Simple Testbench module testbench1() logic a, b, c, y; // instantiate device under test sillyfunction dut(a, b, c, y); // apply inputs one at a time intial begin a = 0; b = 0; c = 0;#10; $display("a=%b, b=%b, c=%b, y=%b", a, b, c, y); c = 1; #10; b = 1; c = 0; #10; a = 1; b = 0; c = 0;#10; b = 1; c = 0; #10; end endmodule Instantiates the DUT Applies the inputs Blocking assignments and delays are used to apply the inputs in the appropriate order User must view the results of the simulation and verify by inspection that the correct outputs are produced. UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 initial statement The initial statement executes the statements in its body at the start of simulation. initial statement should be used only in testbenches for simulation, not in modules intended to be synthesized into actual hardware Hardware has no way of magically executing a sequence of special steps when it is first turned on UCSD ECE 111, Prof. Koushanfar, Fall 2018

Self-checking Testbench Checking for correct outputs is tedious and error-prone Determining the correct outputs is much easier when the design is fresh in your mind Making minor changes and need to retest weeks later is troublesome Write a self-checking testbench UCSD ECE 111, Prof. Koushanfar, Fall 2018

Self-checking Testbench In SystemVerilog, comparison using == or != is effective between signals that do not take on the values of x and z Testbenches use the === and !== operators for comparisons of equality and inequality, respectively, because these operators work correctly with operands that could be x or z module testbench2(); logic a, b, c, y; // instantiate device under test sillyfunction dut (a, b, c, y); // apply inputs one at a time and check results initial begin a = 0; b = 0; c = 0; #10; assert (y === 1) else $error(“000 failed.”); c = 1; #10; assert (y === 0) else $error(“001 failed.”); b = 1; c = 0; #10; assert (y === 0) else $error(“010 failed.”); assert (y === 0) else $error(“011 failed.”); a = 1; b = 0; c = 0; #10; assert (y === 1) else $error(“100 failed.”); assert (y === 1) else $error(“101 failed.”); assert (y === 0) else $error(“110 failed.”); assert (y === 0) else $error(“111 failed.”); end endmodule UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 assert statement assert statement checks if a specified condition is true If not, it executes the else statement The $error system task in the else statement prints an error message describing the assertion failure assert is ignored during synthesis UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Test Vectors Writing code for each test vector also becomes tedious, especially for modules that require a large number of vectors. Place the test vectors in a separate file example.tv 000_1 001_0 010_0 011_0 100_1 101_1 110_0 111_0 UCSD ECE 111, Prof. Koushanfar, Fall 2018

Testbench with Test Vector module testbench3(); logic clk, reset; logic a, b, c, y, yexpected; logic [31:0] vectornum, errors; logic [3:0] testvectors [10000:0] // instantiate device under test sillyfunction dut (a, b, c, y); // generate clock always begin clk = 1; #5; clk = 0; #5; end // at start of test, load vectors // and pulse reset initial begin $readmemb("example.tv", testvectors); vectornum = 0; errors = 0; reset = 1; #27; reset = 0; // apply test vectors on rising edge of clk always @(posedge clk) begin #1; (a, b, c, yexpected) = testvectors[vectornum]; // check results on falling edge of clk always @(negedge clk) if (~reset) begin // skip during reset if (y !== yexpected) begin // check result $display("Error: inputs=%b", {a, b, c}); $display(" outputs=%b (%b expected)", y, yexpected); errors = errors + 1; vectornum = vectornum + 1; if (testvectors[vectornum] === 4'bx) begin $display("%d tests completed with %d errors", vectornum, errors); $finish; endmodule UCSD ECE 111, Prof. Koushanfar, Fall 2018

Testbench with Test Vector // generate clock always begin clk = 1; #5; clk = 0; #5; end Generates a clock using an always/process statement with no sensitivity list Continuously reevaluated (clock is not needed for this combinational module but included for the sake of understanding) UCSD ECE 111, Prof. Koushanfar, Fall 2018

Testbench with Test Vector // at start of test, load vectors // and pulse reset initial begin $readmemb("example.tv", testvectors); vectornum = 0; errors = 0; reset = 1; #27; reset = 0; end Beginning of the simulation, reads the test vectors from a text file and pulses reset for two cycles $readmemb reads a file of binary numbers into the testvectors array. $readmemh is similar but reads a file of hexadecimal numbers. UCSD ECE 111, Prof. Koushanfar, Fall 2018

Testbench with Test Vector // at start of test, load vectors // and pulse reset initial begin $readmemb("example.tv", testvectors); vectornum = 0; errors = 0; reset = 1; #27; reset = 0; end Beginning of the simulation, reads the test vectors from a text file and pulses reset for two cycles $readmemb reads a file of binary numbers into the testvectors array. $readmemh is similar but reads a file of hexadecimal numbers. UCSD ECE 111, Prof. Koushanfar, Fall 2018

Testbench with Test Vector // apply test vectors on rising edge of clk always @(posedge clk) begin #1; (a, b, c, yexpected) = testvectors[vectornum]; end Waits 1 time unit after the rising edge of the clock To avoid any confusion if clock and data change simultaneously Sets the three inputs (a, b, c) and the expected output, (yexpected) based on the four bits in the current test vector UCSD ECE 111, Prof. Koushanfar, Fall 2018

Testbench with Test Vector // check results on falling edge of clk always @(negedge clk) if (~reset) begin // skip during reset if (y !== yexpected) begin // check result $display("Error: inputs=%b", {a,b,c}); $display(" outputs=%b (%b expected)", y, yexpected); errors = errors + 1; end vectornum = vectornum + 1; if (testvectors[vectornum] === 4'bx) begin $display("%d tests completed with %d errors", vectornum, errors); $finish; Compares the generated output, y, with the expected output, yexpected prints an error if they don’t match Repeats until no more valid test vectors in the testvectors array. $finish terminates the simulation SystemVerilog supports upto 10,001 test vectors UCSD ECE 111, Prof. Koushanfar, Fall 2018

Testbench with Test Vector Summary Reads the test vectors from the file Applies the input test vector to the DUT Wait, checks that the output values from the DUT match the output vector Repeat until reaching the end of the test vector file New inputs are applied on the rising edge of the clock Output is checked on the falling edge of the clock Errors are reported as they occur Total number of test vectors applied and number of errors occurred are printed at the end UCSD ECE 111, Prof. Koushanfar, Fall 2018

Testbench (Verification) UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Introduction Verification is the process of experimenting our design with possible test scenarios. Verification contains many phases that includes Testcase generation, coverage, monitor, …, etc. Before tape-out our design should be verified more than 90% successfully. UCSD ECE 111, Prof. Koushanfar, Fall 2018

Verification Methodology Key Features The key features associated with the verification methodology are: Assertion Based Verification Functional Coverage Driven Verification Constrained Random Verification Scoreboard & Checker UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Key Features… Assertion Based Verification Assertion Based Verification helps to detect more functional bugs earlier in the verification process and enables short verification cycle times and faster debugging. Functional Coverage Driven Verification Functional Coverage Driven Verification is the process in which the result of functional coverage is used to drive the verification to completion. It also gives a more structured approach to verification, instead of writing more testcases. UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Key Features… Constrained Random Verification It is well suited for designs that have diverse set of operations and sequences that are difficult to cover completely. Constrained random tests are faster to write and enables faster verification of the design. Scoreboard & Checker Scoreboard and Checker is a mechanism used to dynamically predict the response of the design and check the observed response against the predicted response. Usually it refers to the entire dynamic response-checking structure. UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Verification Flow UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Verification Flow Design Specification Verification flow starts with understanding the architecture specification of the design under verification. Once the architecture is well understood then comes the verification plan. Verification Plan The verification plan comprises the preparation of test case scenarios and testbench environment UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Testcase Scenarios The testcase scenarios document includes all the possible combinations to test the functionality of the design under test UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Self Checking Self checking test cases are written in such a way that is tests itself. UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Self Checking… Corner Corner cases covers the scenarios which are prone to errors and non-trivial, where there are more possibilities of uncovering bugs. Directed Directed cases covers all the scenarios related to the general features and they are straight forward cases which are written in an orderly manner. Coverage Coverage cases are based on the coverage report and they contain the scenarios which are missed out in the other normal categories. UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Non-Self Checking Non - self checking test cases has no self checking code and they are coded such that if the program execution reaches the last line of the test case properly then a success code will be moved into a specific register. After the completion of execution that register can be checked for the success code to ensure the correctness of the design. UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Non-Self Checking… Random Random cases cover all the possible combinations of data as well as scenarios in a random manner. Constrained Random Constrained Random cases cover the all possible combinations of data as well as scenarios in a random manner, constrained for valid combinations. Stress Stress cases exercise the logic with more number of combinations of scenarios as well as data for a long time to ensure the reliability of the design under stress conditions. UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Non-Self Checking Negative Negative cases covers the scenarios which are not possible under normal cases. They are used to analyze the behavior of the design under error conditions. Application Specific High level cases targeted for a particular application normally written in “C” language. UCSD ECE 111, Prof. Koushanfar, Fall 2018

Testbench Environment The testbench environment includes different types of environments to be developed for effectively verifying the design under verification. The testbench environments are of two types based on the testing strategies adopted. Top Level Block Level UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Top Level The top level testing includes the testing of the design as a whole and the stimulus forced in this case is the memory image generated from the assembly level or high level test files. Based on the requirement for the verification, there are three types of environments. UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Top Level Static Performs functional verification in a static manner using functional model. The results of DUT and model are dumped and compared for equivalence once the execution is over. Dynamic Performs timing verification in a dynamic manner using the cycle accurate model. The results of DUT and model are compared for equivalence at each and every cycle dynamically. Coverage The following are considered : Line coverage Branch coverage FSM coverage UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Block Level Block level verification includes the verification of the different inner modules of the design for checking the scenarios which cannot be covered using the top level verification. It includes both static and dynamic environments and the stimuli in this case are inputs generated using tasks. The testbench provides different debugging options and also uses sentinels to ensure the error free flow of data. UCSD ECE 111, Prof. Koushanfar, Fall 2018

SystemVerilog Based Verification Environment UCSD ECE 111, Prof. Koushanfar, Fall 2018

SystemVerilog Based Testbench Development Sequencer Sequencer is an object that defines a set of transactions to be executed and controls the execution of other sequences. Driver It is the component responsible for executing or processing transactions and provides stimuli to the design-under-test (DUT). Monitor This block continuously monitors the DUT signals and bus functions. Scoreboard Driver requests are transferred to the scoreboard via monitor block. UCSD ECE 111, Prof. Koushanfar, Fall 2018

Scoreboard Components Functional Coverage Accumulation Functional coverage is a measure of which design features have been exercised by the tests. Dynamic response checking mechanism The comparison of golden data with DUT data is performed dynamically. This block controls the overall verification environment such as reporting, violations and changing of the stimulus during run-time. Reporting mechanism Compiler directives to issue messages throughout the simulation. (Error, Info, Warning) UCSD ECE 111, Prof. Koushanfar, Fall 2018

Integration, Simulation Once the RTL coding is over, it is hooked up in the testbench environment and then the verification process can be started. Simulation Design is simulated using the simulation tool. Any error in the simulation triggers an error message which is dumped into an error log. TestSuite Regression / Coverage The Regression Environment is developed using perl and shell scripts, to automate the regression run. UCSD ECE 111, Prof. Koushanfar, Fall 2018

UCSD ECE 111, Prof. Koushanfar, Fall 2018 Bug Report Mechanism Bug reports are maintained in Google docs for all the projects in the verification phase. Once the bug is encountered it is updated in the bug report with detailed information. Status of the open bugs is updated regularly. UCSD ECE 111, Prof. Koushanfar, Fall 2018