1 Test & Verification Mark E. Sampson UPDATED 10/20/04 EMIS 8390 Systems Engineering Tool—applying tools to engineering systems

2 Design for Testability… Test & Validation begins with considering test from the start, including: 1.Partitioning decisions-functions & physical to give you the observability you need. 2.Fault-isolation 3.Built-in Test 4.HW vs. SW Some guidelines: Standard connectors & orientation  Functions exist wholly within a partition Get drivers/probes to the right spot quickly/easily Some Tools: Electronics—Test Vector generators, managers …Labview Software—Regression, UI, Function/Condition coverage…TestDirector,… System-- [SE Handbook 12] [Lacy 1992]

3 Verfication & Validation… Verification: Are we building the right product? Systems Engineering ensures this happens at each level of the product Validation: Did we build it right? Customers validate products. Verification happens at each level of a product…a necessary input to the next level design [SE Handbook 12] [Lacy 1992] System Sub-System Component System Sub-System Component Customer needs Customer Validation Verification The Product Development “V”

4 Verification Process Verification: performed throughout the lifecycle to assure the system is on track (one of the definitions of a good requirement includes “is verifiable”). A distinct verification test is appropriate for each level of the spec…with the goal to surface problems early while it’s cheap to fix/correct. [SE Handbook 12] [Lacy 1992] Ricoh did not find a problem with one of their copy machines until the copiers were in the field. They quantified the cost—it cost them $590,000 to do field upgrades. If they had caught the problem before production start it would have cost them $368, if they caught it before they started procuring parts it would have cost them $177, if they had corrected it during design, it would have cost them $35. “We’ve gotten pretty good at chip design. 90% work as designed, 50% of them fail when you plug them in the system” IEEE

5 Verification Process…continued Test decompositions …60k test cases on phone switches Tests tied to requirements, functions,… …of 5 types… 1.Inspection (I)…examination against applicable documents 2.Analysis (A)…of analytical/simulated data to show theoretical compliance 3.Demonstration (D))…qualitative exhibition of performance 4.Test (T)…capability of an item verified, usually through the use of special test equipment or test data analysis 5.Certification… verification against standards/legal by outside authority (UL) Verification Matrix [SE Handbook 12] [Lacy 1992]