1. Product Validation
Adapted from the NASA Systems Engineering Handbook for CSULB EE 400D by Alia Bonetti

2. Reading NASA Systems Engineering Handbook
Chapter 5.4: Product Validation
Appendix E: Creating the Validation Plan
Appendix I: Verification and Validation Plan Sample Outline

3. What is Product Validation?
“Validation is performed for the benefit of the customers and users to ensure that the system functions in the expected manner when placed in the intended environment”
Recall that Validation proves whether “the right product was built”
Whereas Verification proves whether “the product was built right.”
Generate evidence as necessary to confirm that the product meets the capability and operational expectations of the customer and any other stakeholder.
For EE400D a detailed mission definition takes the place of the Validation plan.

4. How to Avoid Major Validation Upsets
This is where your requirements come in.
Well-defined requirements are the keys to a successful project from start to finish.
If validation is left to be done too near to the end of a project’s life, major discoveries can result in serious cost, performance, and schedule issues.
In the case of EE 400D, major upsets will be extremely difficult to resolve.
So how is this process mitigated?
“[Ensure] that each end product in the system structure was correctly realized in accordance with its specified requirements before conducting validation.”
i.e. Use ConOps as a driving force throughout requirements definition and product verification processes.

5. Inputs
PRODUCT
VALIDATION PROCESS
INPUTS
OUTPUTS
Verified product
Sample(s) Verification Report with Matrix
Validation Mission Plan
Customer expectations (ConOps, Mission Objective/Profile, etc.)
Any products necessary to perform validation (i.e., the Mission Plan)
For EE400D a detailed mission definition takes the place of the Validation plan.

6. Outputs
PRODUCT
VALIDATION PROCESS
INPUTS
OUTPUTS
Validated product
Discrepancy reports/notes and necessary corrective actions
Validation reports
For EE400D a detailed mission definition takes the place of the Validation plan.

8. Steps in the product validation process
Validation Planning
Validation Preparation
Conduct Validation
Analyze Results

9. Product Validation Planning
Determine the type of validation that must be used for the particular aspect of the project that is being validated
On the day of the Mission this will be “Demonstration.”
Although the type names are the same as the verification type names, the purposes are very different
Review the validation Mission plan with the customer in order to confirm that the proper end product is being validated in the required environment and under the specified operating instructions
For example, which aspects are controlled by the user and which are intrinsic operations of the robot (done automatically via sensors, software, etc.)

10. True or False
Validation can be performed recursively throughout the project life cycle and on a wide variety of product forms.

11. True!
Wait, what?! I thought validation was done to confirm the end product??
Yes and no…
Think of verification as a method for testing whether or not a system works properly. Now think of validation as a method for testing whether that system works as it was intended to in the operational environment.
If a perfectly operational vehicle was developed in a lab, but did not work once operated on the street, that vehicle is an example of successful verification but failed validation. Should the engineers wait until the vehicle is completed to confirm that it works on the street? No! This is why validation is done throughout the project life cycle.*
The Drop Physics Module (DPM) was a JPL managed instrument operated on the SpaceLab, a Space Shuttle module. The validation of the design and operation (ConOps) of DPM, was the responsibility of the Marshall Space Flight Center (MSFC). Validation began by briefing Astronauts on instrument operations, provide MSFC with high fidelity instrument for training astronauts, and during the first and subsequent flights of the DPM instrument.
* There is “final validation,” which should be done on the end product, but this does not mean that validation is not done throughout the life of the project.

12. Product Validation Preparation
In preparation for product validation, gather the following:
Validation plan
Procedures for each type of validation
Purpose and objectives of steps within the validation process
Any necessary pre and post test actions
Criteria for success vs. failure
Well-defined requirements to compare against validation
Product to be validated
Set of customer expectations
Any supporting resources
Any measurement and recording tools
For EE400D a detailed mission definition takes the place of the Validation plan.

13. Outputs Validated product
VALIDATION PROCESS
INPUTS
OUTPUTS
Validated product
Discrepancy reports/notes and necessary corrective actions
Validation reports

14. Conduct Product Validation
When conducting product validation, follow the procedures from the validation plan, take any necessary measurements, and record resulting data needed to determine whether or not validation was successful
The following should be determined from conducting the product validation:
Supporting information to confirm that appropriate results were obtained
Whether the products comply with customer expectations
Whether the product was properly integrated with validation environment according to customer expectations
Whether the product functions with any required interfaces

15. ANALYZE Product Validation Results
Product validation is a wasted effort if the results are not analyzed.
Analyze data for quality, correctness, integrity and consistency
Compare the actual validation results to the expected results
Identify any deficiencies that arose in the validation process
Determine corrective actions needed to address any deficiencies
Record all analysis data and prepare to repeat validation as needed

16. Notes on Validation
Each product in the system (think about the PBS) needs to be validated to confirm that it meets customer expectations before it is integrated into a higher level product.
If a deficiency is discovered during validation, be careful that its correction does not create a new issue with a product that previously operated without issue.
“Regression testing” is a method used to deal with this issue.
Lack of regression testing resulted in repair mission to the Solar Maximum Mission (SMM) in by Space Shuttle Challenger.
Documentation is essential throughout the validation process in order to have proof that customer expectations have been met.

17. Validation Matrix*
*This is a simplified version of the verification matrix from Professor Hill’s EE400D Lectures. A more detailed version is in Appendix E of the NASA Systems Engineering Handbook