1. 1 Agenda for verify activity r1. Objective r2. Verification methods r3. Test documentation r4. Test execution r5. Test results r6. Reviews r7. Special tests

2. 2 1. Objective rVerify activity rVerify tasks rCompletion criteria rPseudo-completion criteria 1. Objective

3. 3 Verify activity rVerifies that the product complies with all its requirements, and documents these test results 1. Objective

4. 4 Verify tasks verify product by ins, demo, test write test plan write test spec verify product by analysis write test procedure product spec test readiness review (TRR) verification review (VR) test results test plan product test spec test proc ins demo test sta ana product design 1. Objective

5. 5 Completion criteria rCustomer agrees testing is complete 1. Objective

6. 6 Pseudo-completion criteria rTesting complete rProduct verification review complete 1. Objective

7. 7 2. Verification methods rInspection rAnalysis rDemonstration rTest rDistinction among methods 2. Verification methods

8. 8 Inspection (INS) (1 of 2) rDefinition: Verification without the use of special laboratory appliances or procedures. Generally non-destructive. Includes sight, hearing, smell, touch, and taste May include simple physical manipulation May include mechanical & electrical measurement May include other forms of investigation 2. Verification methods

9. 9 Inspection (INS) (2 of 2) rDone when convenient, normally when no further changes to the configuration are planned rExamples Weighs less than 10 pounds Surface free of visible defects 2. Verification methods

10. 10 Analysis (ANA) (1 of 2) rDefinition: Verification by technical or mathematical models, simulations, algorithms, charts, circuit diagrams, or other scientific principals 2. Verification methods

11. 11 Analysis (ANA) (2 of 2) Verification through the technical evaluation of data Data may may include lower-level product test results, historical data, simulation data, engineering drawings, flow-down studies, or other documentation MTBF and MTBCF analysis may include failures from testing

12. 12 Demonstration (DEM) rDefinition: Verification using actual operation, adjustment, or reconfiguration. Products may be instrumented and quantitative limits of performance monitored. rExamples Closed caption text appears when commanded Light goes off when safety switch activated 2. Verification methods

13. 13 Test (TST) rDefinition: Verification by technical means, including operation, and involves application of scientific principles and procedures. rMany opinions say that test involves exercising the product and collecting data that needs to be reduced rTest examples Signal to noise ratio > 2 Vibration power spectral density <0.2 g 2 /Hz at 20 Hz 2. Verification methods

14. 14 Distinctions among methods (1 of 2) rSubjective rAgreement between customer &contractor rCrude distinction Analysis -- Verification that can occur without the product being present Inspection -- Simple verification without exercising the product Demonstration -- Simple verification while exercising the product Test -- Everything else 2. Verification methods

15. 15 Distinctions among methods (2 of 2) r Importance of distinguishing among methods Distinction is not terribly important Inspection represents less risk Inspection and demonstration are easier than test Many methods change when test spec written 2. Verification methods

16. 16 3. Test documentation rTest plan rTest specification rTest procedure 4. Test documentation

17. 17 Test plan (1 of 3) rGenerated by the contractor rDefines approach to testing rEstablishes approach to test methods Demonstration and test -- defined in test spec Analysis -- defined in verification by analysis plan Inspection -- defined as inspection directives 4. Test documentation

18. 18 Test plan (2 of 3) rDefines roles of test specs and test procedures rDefines how to capture test results rDefines reviews 4. Test documentation

19. 19 Test plan (3 of 3) rTest and evaluation master plan (TEMP) Special case of test plan Generated by customer Defines customer plan for evaluating product Applies to the highest level government product 3. Test documentation

20. 20 Test specification (1 of 2) rGenerated by contractor rDefines agreement between customer and contractor about what each demonstration and test must do rIncludes the pass/fail criteria rIs not a substitute for the product specification 3. Test documentation

21. 21 Test specification (2 of 2) rAnalysis plan Generated by contractor Defines agreement between customer and contractor about what each analysis must do Includes the pass/fail criteria Is not a substitute for the product specification 3. Test documentation

22. 22 Test procedure (1 of 3) rGenerated by the contractor rDefines specific test set up rDefines specific test process for running test rApplies to demonstration, test, and selected test and analysis 3. Test documentation

23. 23 Test procedure (2 of 3) rInspection directive Generated by contractor Defines agreement between customer and contractor about what each inspection must do Includes the pass/fail criteria Usually a single sheet that also documents results of test Is not a substitute for the product specification 3. Test documentation

24. 24 Test procedure (3 of 3) RequirementsDesignBuild Test specTest descriptionTest procedure rTest description Expands detail between the test spec & procedures Analogous to design Allows people to agree on details of test before starting to write test procedure is similar to 3. Test documentation

25. 25 4. Test execution rSimulation rConfiguration rTests rRegression tests 4. Test execution

26. 26 Simulation (1 of 2) rA test tool that reduces cost rVerifies extremes that would be hard to produce in actual tests rSupports performance and environmental testing 4. Test execution

27. 27 Simulation (2 of 2) rSimulation verification Simulations need to be validated for accuracy Customer may not have confidence in simulation if the phenomena being simulated is complicated; e.g. software throughput 4. Test execution

28. 28 Configuration rTests should be done on configured hardware and software to ensure repeatability 4. Test execution

29. 29 Tests rMay be run in a lab, in the manufacturing environment, when installed, or when used in the field rImportant tests may warrant a quick-look report giving test results within 24 hours of the test 4. Test execution

30. 30 Regression tests rDefinition: Tests that are run to confirm that a product performance is OK after a change to the product rRun after originally scheduled verification test rFor example, if a new device driver is added to an already tested operating system, tests would be run to confirm that the operating system still opiates with the added device driver 4. Test execution

31. 31 5. Test results Test method Analysis (ANA) Inspection (INS) Demonstration (DEMO) Test (TEST) Pass/fail VDB Analysis Study n/a Study Test Sheet Report Pass/fail is recorded in single place like a verification data base (VDB). Studies and single-sheet reports provide supporting data Pass/fail is recorded in single place like a verification data base (VDB). Studies and single-sheet reports provide supporting data 5. Test results

32. 32 6. Reviews rReadiness review rProduct verification review rFailure review board 6. Reviews

33. 33 Readiness review rObtains customer concurrence that contractor is ready to perform test rConfirms that test procedures are satisfactory, that product to be tested will be configured, and that correct people will witness the test rPresents schedule of testing, analysis, and review 6. Reviews

34. 34 Product verification review (1 of 2) rObtains customer concurrence that testing is satisfactory rHeld after test rExamines results of tests, including analysis 6. Reviews

35. 35 Product verification review (2 of 2) rList tests that pass rExplains exceptions, problems, and failures; addresses deviations and waivers rMay be executed as a series of reviews with each review associated with one or more tests 6. Reviews

36. 36 Failure review board rExamines each hardware failure rDefines failure mechanism such as electrostatic discharge, manufacturing defect, or manufacturing rDefines impact of failure rDefines options for preventing future failures 6. Reviews

37. 37 7. Special tests rReliability testing rDesign of experiments rAccelerated life testing rMaintainability and testability rEnvironmental stress screening (ESS) rElectromagnetic testing rFailure report and corrective action system 7. Special tests