1. 9. Verify and sell-off1 Agenda for verify activity r1. Objective r2. Verification Vs validation r3. Verification methods r4. Test documentation r5. Test execution r6. Test results r7. Reviews r8. Special tests

2. 9. Verify and sell-off2 1. Objective rVerify activity rVerify tasks rCompletion criteria rPseudo-completion criteria 1. Objective

3. 9. Verify and sell-off3 Verify activity rVerifies that the product complies with all its requirements, and documents these test results 1. Objective

4. 9. Verify and sell-off4 Verify tasks Write test plan Write test spec Write test procedure product spec Review test readiness Verify product by analysis Review verification results Verify product by ins, demo, test, sta test results test plan product test spec test proc ins demo test sta ana product design 1. Objective

5. 9. Verify and sell-off5 Completion criteria rCustomer agrees testing is complete 1. Objective

6. 9. Verify and sell-off6 Pseudo-completion criteria rTesting complete rProduct verification review complete 1. Objective

7. 9. Verify and sell-off7 2. Verification vs validation rVerification rValidation rHeuristic rGovernment vs civilian validation 2. Verification vs validation

8. 9. Verify and sell-off8 Verification rFormal collection of test results that show compliance to requirements rCan we prove that the product meets the requirements of the specification 2. Verification vs validation

9. 9. Verify and sell-off9 Validation rAssessment of the correctness of a product for its end-use. rCan we show that the requirements reflect the needs of the end user 2. Verification vs validation

10. 9. Verify and sell-off10 Heuristic rVerification -- Have we built the system right rValidation -- Have we built the right system 2. Verification vs validation

11. 9. Verify and sell-off11 Government vs civilian validation rGovernment -- Contractor might not participate directly in formal validation or operational evaluation (OPEVAL) rCivilian -- Contractor often validates needs 2. Verification vs validation

12. 9. Verify and sell-off12 3. Verification methods rInspection rAnalysis rDemonstration rTest rSelected test and analysis rDistinction among methods 3. Verification methods

13. 9. Verify and sell-off13 Inspection (INS) (1 of 2) rDefinition: Verification without the use of special laboratory appliances or procedures. Generally non-destructive. Includes sight, hearing, smell, touch, and taste May include simple physical manipulation May include mechanical & electrical measurement May include other forms of investigation 3. Verification methods

14. 9. Verify and sell-off14 Inspection (INS) (2 of 2) rDone when convenient, normally when no further changes to the configuration are planned rExamples Weighs less than 10 pounds Surface free of visible defects 3. Verification methods

15. 9. Verify and sell-off15 Analysis (ANA) (1 of 3) rDefinition: Verification by technical or mathematical models, simulations, algorithms, charts, circuit diagrams, or other scientific principals Verification through the technical evaluation of data Data may may include lower-level product test results, historical data, simulation data, engineering drawings, flow-down studies, or other documentation MTBF and MTBCF analysis may include failures from testing 3. Verification methods

16. 9. Verify and sell-off16 Analysis (ANA) (2 of 3) rSubtypes Math analysis (ANA/M) -- Uses a model of formula to verify a design meets requirements Design analysis (ANA/D) -- Analysis of the design of a component to verify a design meets requirements Extrapolation analysis (ANA/T) -- Extrapolates the results of lower level tests to verify a design meets requirements 3. Verification methods

17. 9. Verify and sell-off17 Analysis (ANA) (3 of 3) Analysis based on test data (ANA_T) -- Verifies a design based on testing of lower products. This approach is often used with environmental tests. rExamples Test coverage is greater than 95 percent Design conforms to best commercial practices 3. Verification methods

18. 9. Verify and sell-off18 Demonstration (DEM) rDefinition: Verification using actual operation, adjustment, or reconfiguration. Products may be instrumented and quantitative limits of performance monitored. rExamples Closed caption text appears when commanded Light goes off when safety switch activated 3. Verification methods

19. 9. Verify and sell-off19 Test (TST) rDefinition: Verification by technical means, including operation, and involves application of scientific principles and procedures. rMany opinions say that test involves exercising the product and collecting data that needs to be reduced rTest examples Signal to noise ratio > 2 Vibration power spectral density <0.2 g 2 /Hz at 20 Hz 3. Verification methods

20. 9. Verify and sell-off20 Selected test and analysis (STA) rDefinition: Verifies a quantitative requirement by selected or sample tests when it is not feasible to perform comprehensive testing or when statistical significance cannot be obtained through a reasonable test program rConducted under controlled conditions rAnalysis needed to interpret test results and to extend test results to points not tested rModels need to be validated rSTA example Performance shall be met at all altitudes from 1000 meters to 5000 meters 3. Verification methods

21. 9. Verify and sell-off21 Distinctions among methods (1 of 2) rSubjective rAgreement between customer &contractor rCrude distinction Analysis -- Verification that can occur without the product being present Inspection -- Simple verification without exercising the product Demonstration -- Simple verification while exercising the product Test -- Everything else 3. Verification methods

22. 9. Verify and sell-off22 Distinctions among methods (2 of 2) r Importance of distinguishing among methods Distinction is not terribly important Analysis represents less risk Inspection and demonstration are easier than test Many methods change when test spec written 3. Verification methods

23. 9. Verify and sell-off23 4. Test documentation rTest plan rTest specification rTest procedure 4. Test documentation

24. 9. Verify and sell-off24 Test Plan (1 of 3) rGenerated by the contractor rDefines approach to testing rEstablishes approach to test methods Demonstration, test, & STA -- defined in test spec Analysis -- defined in verification by analysis plan Inspection -- defined as inspection directives 4. Test documentation

25. 9. Verify and sell-off25 Test Plan (2 of 3) rDefines roles of test specs and test procedures rDefines how to capture test results rDefines reviews 4. Test documentation

26. 9. Verify and sell-off26 Test plan (3 of 3) rTest and evaluation master plan (TEMP Special case of test plan Generated by customer Defines customer plan for evaluating product Applies to the highest level government product 4. Test documentation

27. 9. Verify and sell-off27 Test specification (1 of 6) rGenerated by contractor rDefines agreement between customer and contractor about what each demonstration, test, and STA must do rIncludes the pass/fail criteria rIs not a substitute for the product specification 4. Test documentation

28. 9. Verify and sell-off28 Test specification (2 of 6) rOutline 1.0 Purpose -- gives high level purpose 2.0 Scope -- covers DEM, TST, STA test methods 3.0 Objectives -- states to verify requirements 4.0 Document organization -- describes each section 5.0 Overview -- describes test approach and strategy 6.0 Detailed tests -- defines tests - 4. Test documentation

29. 9. Verify and sell-off29 Test specification (3 of 6) 7.0 Verification map -- assigns requirements to tests 8.0 Applicable documents -- lists cited documents 9.0 Acronyms -- lists acronyms used 4. Test documentation

30. 9. Verify and sell-off30 Test specification (4 of 6) rDetailed tests outline 1. Test objectives -- requirements verified by test group 2. Test prerequisites -- tests or activities that must be completed prior to this test 3. Test requirements -- what is required to execute the test group; e.g. facilities, environment, test equipment including accuracy, configuration, repetitions 4. Test documentation

31. 9. Verify and sell-off31 Test specification (5 of 6) 4. Test conditions -- test conditions and ordering 5. Analysis requirements -- analysis techniques and tools needed to determine pass/fail 6. Pass/fail criteria -- exit criteria for test DEM -- what should be observed TST, STA -- what should be measured ALL -- margin of error, tolerance 4. Test documentation

32. 9. Verify and sell-off32 Test specification (6 of 6) rAnalysis plan Generated by contractor Defines agreement between customer and contractor about what each analysis must do Includes the pass/fail criteria Is not a substitute for the product specification 4. Test documentation

33. 9. Verify and sell-off33 Test procedure (1 of 3) rGenerated by the contractor rDefines specific test set up rDefines specific test process for running test rApplies to demonstration, test, and selected test and analysis 4. Test documentation

34. 9. Verify and sell-off34 Test procedure (2 of 3) rInspection directive Generated by contractor Defines agreement between customer and contractor about what each inspection must do Includes the pass/fail criteria Usually a single sheet that also documents results of test Is not a substitute for the product specification 4. Test documentation

35. 9. Verify and sell-off35 Test procedure (3 of 3) RequirementsDesignBuild Test specTest descriptionTest procedure rTest description Expands detail between the test spec & procedures Analogous to design Allows people to agree on details of test approach before starting to write test procedure is similar to 4. Test documentation

36. 9. Verify and sell-off36 5. Test Execution rSimulation rConfiguration rTests rRegression tests 5. Test execution

37. 9. Verify and sell-off37 Simulation (1 of 2) rA test tool that reduces cost rVerifies extremes that would be hard to produce in actual tests rSupport s performance and environmental testing 5. Test execution

38. 9. Verify and sell-off38 Simulation (2 of 2) rSimulation verification Simulations need to be validated for accuracy Customer may not have confidence in simulation if the phenomena being simulated is complicated; e.g. software throughput 5. Test execution

39. 9. Verify and sell-off39 Configuration rTests should be done on configured hardware and software to ensure repeatability 5. Test execution

40. 9. Verify and sell-off40 Tests rMay be run in a lab, in the manufacturing environment, when installed, or when used in the field rImportant tests may warrant a quick-look report giving test results within 24 hours of the test 5. Test execution

41. 9. Verify and sell-off41 Regression tests rDefinition: Tests that are run to confirm that a product performance is OK after a change to the product rRun after originally scheduled verification test rFor example, if a new device driver is added to an already tested operating system, tests would be run to confirm that the operating system still opiates with the added device driver 5. Test execution

42. 9. Verify and sell-off42 6. Test Results Test method Analysis (ANA) Inspection (INS) Demonstration (DEMO) Test (TEST) STA Pass/fail VDB Analysis Study n/a Study Test Sheet Report Pass/fail is recorded in single place like a verification data base (VDB). Studies and single-sheet reports provide supporting data Pass/fail is recorded in single place like a verification data base (VDB). Studies and single-sheet reports provide supporting data 6. Test results

43. 9. Verify and sell-off43 7. Reviews rReadiness review rProduct verification review rFailure review board 7. Reviews

44. 9. Verify and sell-off44 Readiness review rObtains customer concurrence that contractor is ready to perform test rConfirms that test procedures are satisfactory, that product to be tested will be configured, and that correct people will witness the test rPresents schedule of testing, analysis, and review 7. Reviews

45. 9. Verify and sell-off45 Product verification review (1 of 2) rObtains customer concurrence that testing is satisfactory rHeld after test rExamines results of tests, including analysis 7. Reviews

46. 9. Verify and sell-off46 Product verification review (2 of 2) rList tests that pass rExplains exceptions, problems, and failures; addresses deviations and waivers rMay be executed as a series of reviews with each review associated with one or more tests 7. Reviews

47. 9. Verify and sell-off47 Failure review board rExamines each hardware failure rDefines failure mechanism such as electrostatic discharge, manufacturing defect, or manufacturing rDefines impact of failure rDefines options for preventing future failures 7. Reviews

48. 9. Verify and sell-off48 8. Special tests rReliability testing rDesign of experiments rAccelerated life testing rMaintainability and testability rEnvironmental stress screening (ESS) rElectromagnetic testing rFailure report and corrective action system 8. Special tests

49. 9. Verify and sell-off49 Agenda for sell-off activity r1. Objective r2. Sell off vs close-out r3. Audits r4. Importance of sell-off r5. Examples r5. Homework

50. 9. Verify and sell-off50 1. Objective rSell-off activity rSell-off tasks rCompletion criteria rPseudo-completion criteria 1. Objective

51. 9. Verify and sell-off51 Sell-off activity rConfirms that all elements of contract have been met and obtains written agreement from the customer that the customer accepts the product 1. Objective

52. 9. Verify and sell-off52 Sell-off tasks Conduct PCA Conduct FCA Collect contract deliverables and test results Agree product FCA audit results contract deliverables and test results agreement PCA audit results 1. Objective

53. 9. Verify and sell-off53 Completion criteria rSell off agreement complete 1. Objective

54. 9. Verify and sell-off54 Pseudo-completion criteria rProduct delivered rFCA and PCA complete 1. Objective

55. 9. Verify and sell-off55 2. Sell-off vs close-out rSell off occurs when the product has completed verification rContract close out comes later when all contractual issues are closed 2. Sell-off vs close-out

56. 9. Verify and sell-off56 3. Audits rDefinition rFCA rPCA 3. Audits

57. 9. Verify and sell-off57 Definition rSurveys to confirm delivered product is as it should be rMajor government audits Functional configuration audit (FCA) Physical configuration audit (PCA) rMay alternately be considered part of verification 3. Audits

58. 9. Verify and sell-off58 FCA rVerifies that the each hardware and software configuration item complies with its requirements rConfirms that product passed its tests rExample items in audit Test procedures agree with spec & test spec Test results Configuration control of testing Action items 3. Audits

59. 9. Verify and sell-off59 PCA (1 of 2) rEstablishes the product baseline, ensures as- built product matches documentation, verify that acceptance testing is adequate rConfirms we built what we designed rExample items in audit CI design documentation History of changes made and pending Operating and support manuals Parts selection list 3. Audits

60. 9. Verify and sell-off60 PCA (2 of 2) CI as-built specifications and interfaces Manufacturing and assembly instructions Acceptance test procedures and data rTypes of PCAs Build up -- Perform PCA as product built up Tear down - Perform PCA after product built 3. Audits

61. 9. Verify and sell-off61 4. Importance of sell-off rImportant for completing contract rNot a difficult concept rRequires planning to assemble documentation Suggestions Make the customer a team member in sell off so that the customer will know what to expect at the sell off Begin at CDR to monitor the percentage of all sell-off items that are complete 4. Importance of sell off

62. 9. Verify and sell-off62 5. Examples rExample 1 rExample 2 5. Examples

63. 9. Verify and sell-off63 Example 1 rGovernment product Contract items Product Test results Warranty Spare parts Support agreement Government furnished equipment Facilities Contract data Sell-off agreement DD 250 for each contract line item 5. Examples

64. 9. Verify and sell-off64 Example 2 rHouse Sell-off items House Title policy Warranty deed Sell-off agreement Bill of sale 5. Examples

65. 9. Verify and sell-off65 6. Homework r1. For buying a car for personal use, list three types of tests that might apply for each of the four test methods. r2. If we said that we were going to verify the weight of a car by adding up the weights from the spec sheets for each component of the car, what test method would we use? r3. In building a house, what is an example of a verification test that we might require before the house is complete? 6. Homework