HW/SW- Codesign Verification and Debugging. HW versus SW Ondrej Cevan

Outline what is testing embedding testing in design process of HW & SW testing tools in HW, SW Simulator, Emulator, Monitor

HW/SW Codesign Verification and Debugging3 What is Testing? “to place a product or piece of equipment under everyday and/or extreme conditions and examine it for its durability, etc.” (wiktionary.org) with testing we want to achieve:  the object meets its specification  estimation for reliability  save cost (maintenance)  improve quality debugging: the activity of locating, analyzing and removing a bug (an error state) from the system.

HW/SW Codesign Verification and Debugging4 Design Flow and Testing V-model in SW design assigns testing activities to development activities (figure from Yilin's slide)

HW/SW Codesign Verification and Debugging5 Design Flow and Testing model in HW- design extension of every test process in V-model Simulations are checks for the designer to make sure correct behavior is being implemented we need to simulate the HW-clock

HW/SW Codesign Verification and Debugging6 Abstraction Levels SW:  System Specification Language  High-level Language  Assembler  Computer Instructions HW:  System Level  Algorithmic Level  Register Transfer Level  Logic Level  Circuit Level  Chip

HW/SW Codesign Verification and Debugging7 HW-model Behavior Simulation  verifies the correctness of the HDL description  not every code is synthesis-able Functional Simulation  verification after synthesis, design with internal components Pre Layout Simulation  after technology mapping, design with components of the target technology Post Layout Simulation  to verify the results of place and route process  timing and functionality of the design checked

HW/SW Codesign Verification and Debugging8 Verification Methods (HW/SW) Review (of documents & code) Formal Verification (model checking, theorem proving) Dynamic Verification:  functional testing, structural testing, performance, reliability, security testing  simulations, black-box, white-box, code coverage, path coverage, throughput, resource usage, response time, stress/load testing, fault injection, simulation of attack...

HW/SW Codesign Verification and Debugging9 Verification Methods/Tools HW Hardware Implemented Fault Injection (HIFI) Hardware in the Loop Remote debugging tools (in-circuit emulator (ICE) = on-circuit debugger (OCD)= background debug module (BDM)) Logic Analyzer Oscilloscope (manual I/O testing) Simulation-based fault injection (inject faults in a simulation model- on electrical level, gate level, register level, VHDL level, system level) FPGA Prototyping

HW/SW Codesign Verification and Debugging10 SW/HW-Simulator simulator resembles real-world environment Hardware in the Loop, Software in the Loop, Model in the Loop Testbench (generates stimuli- input vector, emulates test cases) test case created from the specification document

HW/SW Codesign Verification and Debugging11 SW/HW-Debugger an analyze tool for developers to test and debug other programs program or fraction of code is processed line by line one can stop the program, read the values of variables and set new values to the variables certain test environments, test situations can be enforced also good for failure reproduction and fault analysis examples: GNU debugger (gdb), dbx, vhdldbx, Eclipse

HW/SW Codesign Verification and Debugging12 SW/HW- Emulator ability of a program or device to imitate other program or device in theoretical sense, Church-Turing thesis implies that any operating environment can be emulated within any other (in practice it can be quite difficult to achieve it) enables to test code for specific HW on a common PC we can emulate HW faults in the design phase (e.g. with fault injection) also SW faults can be emulated (Ballista tool) other ex.: wine, Commodore 64 emulator,...

HW/SW Codesign Verification and Debugging13 HW-Emulator Simulation: executes RTL code serially, good debug environment, user interface, easy, flexible, low cost, BUT: not fast enough for large designs Prototype: executes fully in parallel, fast, BUT needs logic analyzer to analyze limited number of signals, little to no debug capability HW-Emulator: rich debug environment, parallel execution, provides many features that can be found in logic simulators, a specific case of HW Emulator: In-Circuit Emulator

HW/SW Codesign Verification and Debugging14 In-Circuit Emulator (ICE) provides a “window” (UI) into the embedded system often emulates the CPU recent ICEs enable a programmer to access the on chip debug circuit that is integrated into the CPU of the embedded system computer accessed via JTAG or BDM (background debug mode) offers to the developer a full control over the system under test (code loading, debugging, memory analysis, monitoring of signals...)

HW/SW Codesign Verification and Debugging15 Monitor software tool or hardware unit acts concurrent with the system or component observes /monitors, verifies, analyses and records the execution of the system  e.g. measures the response time depending on the amount of user transactions (performance testing)

HW/SW Codesign Verification and Debugging16 When do we have tested enough? exhaustive testing is infeasible in practice we define coverage criteria that have to be met by a testing functional coverage (black-box view) structural coverage (white-box view)  code coverage: path coverage, state coverage, statement coverage, decision coverage, MCDC...  data flow coverage in industry safety standards are used that describe the testing procedure and also the system development (RTCA/DO-178b, IEC 61508), certification is also used in some domains

HW/SW Codesign Verification and Debugging17 Effort and Time many factors influence the time spent on testing  know-how, experience, test-object (is it new to the test team?), test strategy, requirements on the object: is it safety critical?, how intensive must the object be tested?, what quality of the object should be achieved?... in theory in SW the test effort should be approximately 50% of the total development effort (rule of thumb according to the book Basiswissen Softwaretest) in HW it should be approximately 70-80% (according to the slides from VL Hardware Modeling) in practice: much much less 10-20% (according to the slides VL Hardware Modeling)

HW/SW Codesign Verification and Debugging18 Summary HW:  verification needs HW an SW tools  more effort is needed: no possibility for patches, the whole system/component must be replaced in case of failure SW:  SW tools for verification are sufficient  no need for extra testing hardware (oscilloscope..)

HW/SW Codesign Verification and Debugging19 List of used sources slides: VO Testing of Embedded Systems (Raimund Kirner, Astrid Ademaj) slides: VL Hardware Modeling (Martin Delvai) slides: VO Embedded System Engineering (Wilfried Elmenreich) book: Basiswissen Softwaretest, Andreas Spillner and Tilo Linz wikipedia.org wiktionary.org