Product Validation Adapted from the NASA Systems Engineering Handbook for CSULB EE 400D by Alia Bonetti

Reading NASA Systems Engineering Handbook Chapter 5.4: Product Validation Appendix E: Creating the Validation Plan Appendix I: Verification and Validation Plan Sample Outline

What is Product Validation? “Validation is performed for the benefit of the customers and users to ensure that the system functions in the expected manner when placed in the intended environment” Recall that Validation proves whether “the right product was built” Whereas Verification proves whether “the product was built right.” Generate evidence as necessary to confirm that the product meets the capability and operational expectations of the customer and any other stakeholder. For EE400D a detailed mission definition takes the place of the Validation plan.

How to Avoid Major Validation Upsets This is where your requirements come in. Well-defined requirements are the keys to a successful project from start to finish. If validation is left to be done too near to the end of a project’s life, major discoveries can result in serious cost, performance, and schedule issues. In the case of EE 400D, major upsets will be extremely difficult to resolve. So how is this process mitigated? “[Ensure] that each end product in the system structure was correctly realized in accordance with its specified requirements before conducting validation.” i.e. Use ConOps as a driving force throughout requirements definition and product verification processes.

Inputs PRODUCT VALIDATION PROCESS INPUTS OUTPUTS Verified product Sample(s) Verification Report with Matrix Validation Mission Plan Customer expectations (ConOps, Mission Objective/Profile, etc.) Any products necessary to perform validation (i.e., the Mission Plan) For EE400D a detailed mission definition takes the place of the Validation plan.

Outputs PRODUCT VALIDATION PROCESS INPUTS OUTPUTS Validated product Discrepancy reports/notes and necessary corrective actions Validation reports For EE400D a detailed mission definition takes the place of the Validation plan.

Steps in the product validation process Validation Planning Validation Preparation Conduct Validation Analyze Results

Product Validation Planning Determine the type of validation that must be used for the particular aspect of the project that is being validated On the day of the Mission this will be “Demonstration.” Although the type names are the same as the verification type names, the purposes are very different Review the validation Mission plan with the customer in order to confirm that the proper end product is being validated in the required environment and under the specified operating instructions For example, which aspects are controlled by the user and which are intrinsic operations of the robot (done automatically via sensors, software, etc.)

True or False Validation can be performed recursively throughout the project life cycle and on a wide variety of product forms.

True! Wait, what?! I thought validation was done to confirm the end product?? Yes and no… Think of verification as a method for testing whether or not a system works properly. Now think of validation as a method for testing whether that system works as it was intended to in the operational environment. If a perfectly operational vehicle was developed in a lab, but did not work once operated on the street, that vehicle is an example of successful verification but failed validation. Should the engineers wait until the vehicle is completed to confirm that it works on the street? No! This is why validation is done throughout the project life cycle.* The Drop Physics Module (DPM) was a JPL managed instrument operated on the SpaceLab, a Space Shuttle module. The validation of the design and operation (ConOps) of DPM, was the responsibility of the Marshall Space Flight Center (MSFC). Validation began by briefing Astronauts on instrument operations, provide MSFC with high fidelity instrument for training astronauts, and during the first and subsequent flights of the DPM instrument. * There is “final validation,” which should be done on the end product, but this does not mean that validation is not done throughout the life of the project.

Product Validation Preparation In preparation for product validation, gather the following: Validation plan Procedures for each type of validation Purpose and objectives of steps within the validation process Any necessary pre and post test actions Criteria for success vs. failure Well-defined requirements to compare against validation Product to be validated Set of customer expectations Any supporting resources Any measurement and recording tools For EE400D a detailed mission definition takes the place of the Validation plan.

Outputs Validated product VALIDATION PROCESS INPUTS OUTPUTS Validated product Discrepancy reports/notes and necessary corrective actions Validation reports

Conduct Product Validation When conducting product validation, follow the procedures from the validation plan, take any necessary measurements, and record resulting data needed to determine whether or not validation was successful The following should be determined from conducting the product validation: Supporting information to confirm that appropriate results were obtained Whether the products comply with customer expectations Whether the product was properly integrated with validation environment according to customer expectations Whether the product functions with any required interfaces

ANALYZE Product Validation Results Product validation is a wasted effort if the results are not analyzed. Analyze data for quality, correctness, integrity and consistency Compare the actual validation results to the expected results Identify any deficiencies that arose in the validation process Determine corrective actions needed to address any deficiencies Record all analysis data and prepare to repeat validation as needed

Notes on Validation Each product in the system (think about the PBS) needs to be validated to confirm that it meets customer expectations before it is integrated into a higher level product. If a deficiency is discovered during validation, be careful that its correction does not create a new issue with a product that previously operated without issue. “Regression testing” is a method used to deal with this issue. Lack of regression testing resulted in repair mission to the Solar Maximum Mission (SMM) in 1984 by Space Shuttle Challenger. Documentation is essential throughout the validation process in order to have proof that customer expectations have been met.

Validation Matrix* *This is a simplified version of the verification matrix from Professor Hill’s EE400D Lectures. A more detailed version is in Appendix E of the NASA Systems Engineering Handbook