1 Simulation Validation H Plan: –Discuss verification and validation –Define concepts and terminology –Example: Weasley Barber Shop simulator u Effect of arrival rate u Effect of service rate(s) u Effect of waiting room size u Effect of dispatch policy

2 Verification and Validation H Simulation is a widely used (and abused!) technique for systems performance eval H It is important to establish trust and confidence in the results generated and reported via simulation H Two (slightly different) aspects: –Verification: results are correct –Validation: results match reality

3 Verification H Verification is mostly about establishing correctness (i.e., veracity, truthfulness) H Does it do what it is supposed to do? –Meets requirements specifications (if any) H Does it produce correct results? –Check results on known cases (input/output) –Handles special “corner” cases –Results are mathematically plausible

4 Validation H Validation is more about the practical applicability of the simulator (usefulness) H Does it give answers that make sense? H Are the results intuitively plausible? H Can it predict results for unknown cases? H Can results be corroborated in some way? –Analytical model –Experimental measurement data H Relationship to reality is key!

5 Validation Example H Suppose you have a simulator for the Weasley Barber Shop from Assignment 1 H How to do “simulation validation”? –Manual tracing with printf statements –Try boundary cases (extreme and/or special) –Numerical comparisons to analytical model –Comparison to M/M/1/K analytical model –Comparison to M/M/2 analytical model

6 Summary H Simulation validation is an important step in establishing the correctness of results H Rarely discussed in papers themselves H Important “behind the scenes” effort H Should be part of your science “checklist” H If results don’t make sense, then find simplest possible example showing the problem, so you can debug and fix it!