6. Verify and sell-off 1 Agenda for Verify Activity r1. Objectives r2. Verification Vs validation r3. Verification methods r4. Documentation of tests r5. Test execution r6. Test results r7. Reviews r8. Special tests

6. Verify and sell-off 2 1. Objectives rVerify activity rProduct-based activities rProducts used to manage rCompletion criteria 1. Objectives

6. Verify and sell-off 3 Verify Activity rVerifies that the product complies with all its requirements, and documents these test results 1. Objectives

6. Verify and sell-off 4 Verify Tasks Write test plan Write test spec Write test procedure product spec Review test readiness Verify product by analysis Review verification results Verify product by ins, demo, test, sta test results test plan product test spec test proc ins demo test sta ana product design 1. Objectives

6. Verify and sell-off 5 Completion Criteria rCustomer agrees testing is complete 1. Objectives

6. Verify and sell-off 6 Pseudo- Completion Criteria rTesting complete rProduct verification review complete 1. Objectives

6. Verify and sell-off 7 2. Verification Vs Validation rVerification rValidation rHeuristic rGovernment Vs civilian validation 2. Verification vs validation

6. Verify and sell-off 8 Verification rFormal collection of test results that show compliance to requirements rCan we prove that the product meets the requirements of the specification 2. Verification vs validation

6. Verify and sell-off 9 Validation rAssessment of the correctness of a product for its end-use. rCan we show that the requirements reflect the needs of the end user 2. Verification vs validation

6. Verify and sell-off 10 Heuristic rVerification -- Have we built the system right rValidation -- Have we built the right system 2. Verification vs validation

6. Verify and sell-off 11 Government Vs Civilian Validation rGovernment -- Contractor might not participate directly in formal validation or operational evaluation (OPEVAL) rCivilian -- Contractor often validates needs 2. Verification vs validation

6. Verify and sell-off Verification Methods rInspection rAnalysis rDemonstration rTest 3. Verification methods

6. Verify and sell-off 13 Inspection (INS) rVerification without the use of special laboratory appliances or procedures. l Generally non-destructive. l Includes sight, hearing, smell, touch, and taste l May include simple physical manipulation l May include mechanical & electrical measurement l May include other forms of investigation rDone when convenient, normally when no further changes to the configuration are planned 3. Verification methods

6. Verify and sell-off 14 Analysis (ANA) rVerification by technical or mathematical models, simulations, algorithms, charts, circuit diagrams, or other scientific principals rVerification through the technical evaluation of data rData may may include lower-level product test results, historical data, simulation data, engineering drawings, flow-down studies, or other documentation rMTBF and MTBCF analysis may include failures from testing 3. Verification methods

6. Verify and sell-off 15 Analysis Categories rMath analysis (ANA/M) -- Uses a model of formula to verify a design meets requirements rDesign analysis (ANA/D) -- Analysis of the design of a component to verify a design meets requirements rExtrapolation analysis (ANA/T) -- Extrapolates the results of lower level tests to verify a design meets requirements rAnalysis based on test data (ANA_T) -- Verifies a design based on testing of lower products. This approach is often used with environmental tests. 3. Verification methods

6. Verify and sell-off 16 Demonstration (DEM) rVerification using actual operation, adjustment, or reconfiguration. Products may be instrumented and quantitative limits of performance monitored. 3. Verification methods

6. Verify and sell-off 17 Test (TST) rVerification by technical means, including operation, and involves application of scientific principles and procedures. 3. Verification methods

6. Verify and sell-off 18 Selected Test and Analysis (STA) rVerifies a quantitative requirement by selected or sample tests when it is not feasible to perform comprehensive testing or when statistical significance cannot be obtained through a reasonable test program rConducted under controlled conditions rAnalysis needed to interpret test results and to extend test results to points not tested rModels need to be validated 3. Verification methods

6. Verify and sell-off 19 Distinctions of Methods rSubjective rAgreement between customer &contractor 3. Verification methods

6. Verify and sell-off 20 Crude Distinction rAnalysis -- Verification that can occur without the product being present rInspection -- Simple verification without exercising the product rDemonstration -- Simple verification while exercising the product rTest -- Everything else 3. Verification methods

6. Verify and sell-off 21 Test rMany opinions say that test involves exercising the product and collecting data that needs to be reduced 3. Verification methods

6. Verify and sell-off 22 Importance of Distinguishing among Methods rDistinction is not terribly important rAnalysis represents less risk rInspection and demonstration are easier than test rMany methods change when test spec written 3. Verification methods

6. Verify and sell-off 23 Analysis Examples rTest coverage is greater than 95 percent rDesign conforms to best commercial practices 3. Verification methods

6. Verify and sell-off 24 Inspection Examples rWeighs less than 10 pounds rSurface free of visible defects 3. Verification methods

6. Verify and sell-off 25 Demonstration Examples rClosed caption text appears when commanded rLight goes off when safety switch activated 3. Verification methods

6. Verify and sell-off 26 Test Examples rSignal to noise ratio > 2 rVibration power spectral density <0.2 g 2 /Hz at 20 Hz 3. Verification methods

6. Verify and sell-off Documentation of Tests rTest plan rTest specification rTest procedure 4. Documentation of tests

6. Verify and sell-off 28 Test Plan (1 of 2) rGenerated by the contractor rDefines approach to testing rEstablishes approach to test methods l Demonstration, test, & STA -- defined in test spec l Analysis -- defined in verification by analysis plan l Inspection -- defined as inspection directives 4. Documentation of tests

6. Verify and sell-off 29 Test Plan (2 of 2) rDefines roles of test specs and test procedures rDefines how to capture test results rDefines reviews 4. Documentation of tests

6. Verify and sell-off 30 Test and Evaluation Master Plan rGenerated by customer rDefines customer plan for evaluating product rApplies to the highest level government product 4. Documentation of tests

6. Verify and sell-off 31 Test Specification rGenerated by contractor rDefines agreement between customer and contractor about what each demonstration, test, and STA must do rIncludes the pass/fail criteria rIs not a substitute for the product specification 4. Documentation of tests

6. Verify and sell-off 32 Test Specification Outline r1.0 Purpose -- gives high level purpose r2.0 Scope -- covers DEM, TST, STA test methods r3.0 Objectives -- states to verify requirements r4.0 Document organization -- describes each section r5.0 Overview -- describes test approach and strategy r6.0 Detailed tests -- defines tests - r7.0 Verification map -- assigns requirements to tests r8.0 Applicable documents -- lists cited documents r9.0 Acronyms -- lists acronyms used 4. Documentation of tests

6. Verify and sell-off 33 Detailed Test Group Outline (1 of 2) r1. Test objectives -- requirements verified by test group r2. Test prerequisites -- tests or activities that must be completed prior to this test r3. Test requirements -- what is required to execute the test group; e.g. facilities, environment, test equipment including accuracy, configuration, repetitions 4. Documentation of tests

6. Verify and sell-off 34 Detailed Test Outline (2 of 2) r4. Test conditions -- test conditions and ordering r5. Analysis requirements -- analysis techniques and tools needed to determine pass/fail r6. Pass/fail criteria -- exit criteria for test l DEM -- what should be observed l TST, STA -- what should be measured l ALL -- margin of error, tolerance 4. Documentation of tests

6. Verify and sell-off 35 Analysis Plan rGenerated by contractor rDefines agreement between customer and contractor about what each analysis must do rIncludes the pass/fail criteria rIs not a substitute for the product specification 4. Documentation of tests

6. Verify and sell-off 36 Inspection Directive rGenerated by contractor rDefines agreement between customer and contractor about what each inspection must do rIncludes the pass/fail criteria rUsually a single sheet that also documents results of test rIs not a substitute for the product specification 4. Documentation of tests

6. Verify and sell-off 37 Test Description RequirementsDesignBuild Test specTest descriptionTest procedure rExpands detail between the test spec & procedures rAnalogous to design rAllows people to agree on details of test approach before starting to write test procedure is similar to 4. Documentation of tests

6. Verify and sell-off 38 Test Procedure rGenerated by the contractor rDefines specific test set up rDefines specific test process for running test rApplies to demonstration, test, and selected test and analysis 4. Documentation of tests

6. Verify and sell-off Test Execution rSimulation rConfiguration rTests rRegression tests 5. Test execution

6. Verify and sell-off 40 Simulation rA test tool that reduces cost rVerifies extremes that would be hard to produce in actual tests rSupport s performance and environmental testing 5. Test execution

6. Verify and sell-off 41 Simulation Verification rSimulations need to be validated for accuracy rCustomer may not have confidence in simulation if the phenomena being simulated is complicated; e.g. software throughput 5. Test execution

6. Verify and sell-off 42 Configuration rTests should be done on configured hardware and software to ensure repeatability 5. Test execution

6. Verify and sell-off 43 Tests rMay be run in a lab, in the manufacturing environment, when installed, or when used in the field rImportant tests may warrant a quick-look report giving test results within 24 hours of the test 5. Test execution

6. Verify and sell-off 44 Regression Tests rTests that are run to confirm that a product performance is OK after a change to the product rRun after originally scheduled verification test rFor example, if a new device driver is added to an already tested operating system, tests would be run to confirm that the operating system still opiates with the added device driver 5. Test execution

6. Verify and sell-off Test Results Test method Analysis (ANA) Inspection (INS) Demonstration (DEMO) Test (TEST) STA Pass/fail VDB Analysis Study n/a Study Test Sheet Report Pass/fail status for each type of test is recorded in a single place such as a verification data base (VDB), Analysis is documented in a study. Tests having methods ANA, INS, or DEMO are documented in a single sheet report. Test and STA are documented in a separate report 6. Test results

6. Verify and sell-off Reviews rReadiness review rProduct verification review rFailure review board 7. Reviews

6. Verify and sell-off 47 Readiness Review rObtains customer concurrence that contractor is ready to perform test rConfirms that test procedures are satisfactory, that product to be tested will be configured, and that correct people will witness the test rPresents schedule of testing, analysis, and review 7. Reviews

6. Verify and sell-off 48 Product Verification Review (1 of 2) rObtains customer concurrence that testing is satisfactory rHeld after test rExamines results of tests, including analysis 7. Reviews

6. Verify and sell-off 49 Product Verification Review (2 of 2) rList tests that pass rExplains exceptions, problems, and failures; addresses deviations and waivers rMay be executed as a series of reviews with each review associated with one or more tests 7. Reviews

6. Verify and sell-off 50 Failure Review Board rExamines each hardware failure rDefines failure mechanism such as electrostatic discharge, manufacturing defect, or manufacturing rDefines impact of failure rDefines options for preventing future failures 7. Reviews

6. Verify and sell-off Special Tests rReliability testing rDesign of experiments rAccelerated life testing rMaintainability and testability rEnvironmental stress screening (ESS) rElectromagnetic testing rFailure report and corrective action system 8. Special tests

6. Verify and sell-off 52 Agenda for Sell Off Activity r1. Objectives r2. Sell off Vs close out r3. Audits r4. Importance of sell off r5. Homework

6. Verify and sell-off Objectives rSell-off activity rProduct-based activities rProducts used to manage rCompletion criteria 1. Objectives

6. Verify and sell-off 54 Sell-Off Activity rConfirms that all elements of contract have been met and obtains written agreement from the customer that the customer accepts the product 1. Objectives

6. Verify and sell-off 55 Sell-Off Tasks Conduct PCA Conduct FCA Collect SOW deliverables and test results Agree product FCA audit results SOW deliverables and test results agreement PCA audit results 1. Objectives

6. Verify and sell-off 56 Completion Criteria rSell off agreement complete 1. Objectives

6. Verify and sell-off 57 Pseudo-Completion Criteria rProduct delivered rFCA and PCA complete 1. Objectives

6. Verify and sell-off Sell Off Vs Close Out rSell off occurs when the product has completed verification rContract close out comes later when all contractual issues are closed 2. Sell off vs close out

6. Verify and sell-off Audits rSurveys to confirm delivered product is as it should be rMajor government audits l Functional configuration audit (FCA) l Physical configuration audit (PCA) rMay alternately be considered part of verification 3. Audits

6. Verify and sell-off 60 FCA rVerifies that the each hardware and software configuration item complies with its requirements rConfirms that product passed its tests rExample items in audit l Test procedures agree with spec & test spec l Test results l Configuration control of testing l Action items 3. Audits

6. Verify and sell-off 61 PCA rEstablishes the product baseline, ensures as-built product matches documentation, verify that acceptance testing is adequate rConfirms we built what we designed rExample items in audit l CI design documentation l History of changes made and pending l Operating and support manuals l Parts selection list l CI as-built specifications and interfaces l Manufacturing and assembly instructions l Acceptance test procedures and data 3. Audits

6. Verify and sell-off 62 Types of PCAs rBuild up -- Perform PCA as product built up rTear down - Perform PCA after product built 3. Audits

6. Verify and sell-off Importance of Sell Off rImportant for completing contract rNot a difficult concept rRequires planning to assemble documentation Suggestions l Make the customer a team member in sell off so that the customer will know what to expect at the sell off l Begin at CDR to monitor the percentage of all sell-off items that are complete 4. Importance of sell off

6. Verify and sell-off 64 Example 1 -- Government Product rContract items l Product l Test results l Warranty l Spare parts l Support agreement l Government furnished equipment l Facilities l Contract data rSell-off agreement l DD 250 for each contract line item 4. Importance of sell off

6. Verify and sell-off 65 Example 2 -- House rSell-off items l House l Title policy l Warranty deed rSell-off agreement l Bill of sale 4. Importance of sell off

6. Verify and sell-off Homework r1. For buying a car for personal use, list three types of tests that might apply for each of the four test methods. r2. If we said that we were going to verify the weight of a car by adding up the weights from the spec sheets for each component of the car, what test method would we use? r3. In building a house, what is an example of a verification test that we might require before the house is complete? 5. Homework