ELN5622 Embedded Systems Class 10 Spring, 2003 Aaron Itskovich

Outlook  Testing, & Verification, Reliability  Test Plan Creation & Execution.  Relaiability & Correctness  Design for Test & Debugging  Testing (Yield, Field Return, Golden System, Coverage), JTAG

Why test?  To reduce risk to both user and company  To reduce development and maintenance cost  To improve performance  To find bugs in software and hardware

To Find the Bugs  Halting Theorem proves it’s impossible to prove that an arbitrary program is correct  Given the right test you can prove that a program is incorrect  Testing is not about proving “correctness” of a program but about finding bugs  The only way to “know” how many bugs left is to test it with carefully designed test plan  Known bug is already “half bug”

To reduce the risk and costs  Minimize risk to yourself, your company and your customers  Earlier you detect the problem cheaper the fix

Costs of bugs  In 1990 HP sampled the cost of errors in software development during the year. The answer $400 million, shocked HP into a completely new effort to eliminate mistakes in writing software. The $400 waste, half of ot spent in the labs on rework and half in the field to fix the mistakes that escaped from lab amounted to 1/3 of the company total R&D budget

How to make bug fixing cheaper?  If we can’t ensure correctness of the released system how to make bug fix cheaper?  Design system to be field upgradeable –Re-configurable hardware (FPGA) –Separate application software from boot –Use Flash or EEPROM as application storage

When to test?  As early as possible  Statistically about 70% of the bugs found during the integration phase of the project were generated by code that had never been exercised before

What tests?  Every time the program is modified, it should be retested to assure that the changes didn’t break some unrelated behavior - REGRESSION TESTING  Individual developers test at the module level by writing stub code to substitute for the rest of the system hardware and software – UNIT TESTINGH

Test case design  Functional testing (black box) –Can and should be written in parallel with the requirements document  Coverage testing (white box) –Coverage test implies that your code is stable  Both kind of testing are necessary to test rigorously your embedded design

A bit of history  The first known computer bug came about in 1946 when a primitive computer used by Navy to calculate the trajectories of artillery shells shut down when a moth got stuck in one of it’s computing elements, a mechanical relay. Hence, the name bug for computer error.

When to stop testing  When the boss says  When the new iteration of the test cycle founds fewer than X new bugs  When a certain coverage threshold has been met without uncovering any new bugs  In case your system is mission critical look into DO-178B specification.

Choosing Test Cases  Functional tests –Stress tests: test that intentionally overload input input channels, memory buffers… –Boundary value tests: Inputs that represent “boundaries”within particular rangeand input values that should case the output to transition across a similar boundary in the output range –Exception test: Test that should trigger a failure mode or exception mode –Error guessing: Test based on the prior experience with testing similar products –Random tests: Usually the least productive form of testing –Performance tests: Test that performance expectation from the requirements are met

Choosing Test Cases  Coverage tests –Statement coverage: Test cases selected because they execute every statement in the program at least once. –Decision or branch coverage: Test cases chosen because they cause every branch (both true and false path) to be executed at least once. –Condition coverage: Test cases chosen to force each condition (term) in decision to take on all possible logic values.

Practical alternatives  Gray box testing-what it is? –White box tests – expensive to maintain need to be reengineered every time code is changed –Gray box – exploit knowledge of implementation without being intimately tied to the coding details

Some distinguishers of the embedded system  Embedded system must run reliably without crashing for long periods of time.  Embedded software must often compensate for problems with the embedded hardware  Real world events are usually asynchronous and non deterministic, making simulations tests difficult and unreliable  Did you read software “license agreement”?

Measuring test coverage  Code instrumentation methods - aka Software logging –Printf: intrusive - slows down system –Low intrusion Printf –Usage of logic analyzer to measure coverage –Decision coverage (DC): measures results of decision points in the code –Modified decision coverage (MDC): One step farther than DC evaluates the terms that make up decision point.  Hardware instrumentation methods (logic analyzer, trace,

How to test performance

Manufacturing tests  Build in self test (BIST)  Test bed  Golden system concept  JTAG boundary scan  Yield  Field return  Fault correlation and root cause analyzes