1. Chap. 5 Building Valid, Credible, and Appropriately Detailed Simulation Models

2. 5.1 Introduction and Definitions (1) Verification is concerned with determining whether the conceptual simulation model (model assumptions) has been correctly translated into a computer program, i.e., debugging the simulation computer program.

3. OJOJ '.'. I Validation is the process of determining whether a simulation model (as opposed to the computer program) is an accurate representation of the system, for the particular objectives of the study. 5.1 Introduction and Definitions(2)

4. Validation A valid model can be used to make decisions. A validation process depends on the complexity of the system and on whether a version of the system currently exists. A model can only be an approximation. A model is valid for one purpose. The measures of performance used to validate the model should include those that the decision maker will actually use for evaluating system design.

5. A simulation model and its results have credibility if the manager and other project personnel accept them as "correct“. A credible model is not necessarily valid, and vice versa. 5.1 Introduction and Definitions(3)

6. System Conceptual model Simulation program “Correct” results available Results used in decision- making process ValidationVerification ValidationEstablish credibility Establish credibility Analysis and data 1, 2, 3 Programming 4 Make model runs 5,6,7,8,9 Sell results to management 10 Figure 5.1 Timing and relationships of validation, verification, and establishing credibility

7. 5.2 Guidelines for Determining the Level of Model Detail (1) Carefully define the specific issues to be investigated by the study and the measures of performance that will be used for evaluation. The entity moving through the simulation model does not always have to be the same as the entity moving through the corresponding system. Use subject-matter experts (SMEs) and sensitivity analyses. “Moderately detailed“ model. Regular interaction.

8. 5.2 Guidelines for Determining the Level of Model Detail (2) Do not have more detail in the model than is necessary to address the issues of interest, subject to the proviso that the model must have enough detail to be credible. The level of model detail should be consistent with thetype of data available. In all simulation studies, time and moneyconstraints are a major factor in determining the amount of model detail. If the number of factors (aspects of interest) for the study is large, then use a "coarse" simulation model or analytic model to identify what factors have a significant impacton system performance.

9. 5.3 Verification of Simulation Computer Program Tech 1: Write and debug the computer program on modules or subprograms. Tech 2: More than one person review the computer program (structured walk­ through of the program). Tech 3: Run the simulation under a variety of settings of input parameters, and check to see that the output is reasonable. Tech 4: "trace", interactive debugger. Tech 5: The model should be run under simplifying assumptions for which its true characteristics are known or can easily be computed. Tech 6: Observe an animation of the simulation output. Tech 7: Compute the sample mean and variance for each simulation input probability distribution, and compare them with the desired mean and variance. Tech 8: Use a commercial simulation package to reduce the amount of programming required.

10. 5.4 Techniques for Increasing Model Validity and Credibility (1) Collect high-quality information and data on the system –Conversation with subject matter experts in MS, machine operators, engineers, maintenance personnel, schedulers, managers, vendors, … –Observation of the system Data are not representative of what one really wants to model Data are not of the appropriate type or format Data may contain measurement, recording, or rounding errors Data may be biased because of self interest Data may be inconsistent –Existing theory IID exponential random variables –Relevant results from similar simulation study –Experience and intuition of the modelers

11. 5.4 Techniques for Increasing Model Validity and Credibility (2) Interact with the manager on a regular basis –There may not be a clear idea of the problem to be solved at initiation of the study. –The manager’s interest and involvement in the study are maintained. –The manager’s knowledge of the system contributes to the actual validity of the model –The model is credible since the manager understands and accepts the model’s assumptions. Maintain an assumptions document and perform a structured walk-through Validate components of the model by using quantitative techniques. Validate the output from the overall simulation model Animation

12. 5.5. Management's Role in the Simulation Process Formulating problem objectives. Directing personnel to provide informationand data to the simulation modeler and to attend the structured walk-through. Interacting with the simulation modeler ona regular basis. Using the simulation results as an aid inthe decision-making process.

13. 5.6 Statistical Procedures for Comparing Real-world Observations and Simulation Output Data Inspection approach. Confidence-interval approach based on independent data. Time-series approach

14. 5.6.1 Inspection Approach Statistics: sample mean, sample variance,the sample correlation function, histograms. dangerous! for sample size 1. Correlated inspection approach

15. Experiment 10.900.700.20 20.700.71-0.01 31.080.350.73 Table 5.4 Results for three experiments with the inspection approach

16. Figure 5.2 The correlated inspection approach Historical system input data Historical system input data Actual systemSimulation model System output dataModel output data Compare

17. Experiment j 1 3.063.812.62-0.750.44 2 2.793.372.05-0.580.74 3 2.212.614.56-0.40-2.35 4 2.543.591.86-1.050.68 5 9.2711.022.41-1.756.86 6 3.093.751.85-0.661.24 7 2.502.841.13-0.341.37 8 0.310.713.12-0.40-2.81 9 3.173.945.09-0.77-1.92 10 0.981.181.25-0.20-0.27 Sample mean of all 500 2.102.852.70-0.75-0.60 Sample variance of all 500 2.022.282.120.084.08 Table 5.5 Results for the first 10 of 500 experiments with the correlated and basic Inspection approaches, and a summary for all 500

18. 5.6.2 Confidence-Interval Approach based on Independent Data Condition: it is possible to collect a potentially large amount of data for both the model and the system.