1. Chapter 6 Statistical Analysis of Output from Terminating Simulations

2. Simulation with Arena Chapter 5 – Detailed Modeling and Terminating Statistical Analysis 2 Statistical Analysis of Output from Terminating Simulations Random input leads to random output (RIRO) Run a simulation (once) — what does it mean? – Was this run “typical” or not? – Variability from run to run (of the same model)? Need statistical analysis of output data – From a single model configuration – Compare two or more different configurations – Search for an optimal configuration Statistical analysis of output is often ignored – This is a big mistake – no idea of precision of results – Not hard or time-consuming to do this – it just takes a little planning and thought, then some (cheap) computer time

3. Simulation with Arena Chapter 5 – Detailed Modeling and Terminating Statistical Analysis 3 Output Analysis Output analysis is concerned with Designing replications Obtain most reliable info with minimum number of replications and minimum run length. Computing statistics Point and confidence interval estimation Size and independency issues Presenting them in a textual and graphical format. Aim is to understand the system behavior and generate predictions for it!

4. Simulation with Arena Chapter 5 – Detailed Modeling and Terminating Statistical Analysis 4 Time Frame of Simulations Terminating: Specific starting, stopping conditions – Run length will be well-defined (and finite) Steady-state: Long-run (technically forever) – Theoretically, initial conditions don’t matter (but practically they usually do) – Not clear how to terminate a simulation run This is really a question of intent of the study Has major impact on how output analysis is done Sometimes it’s not clear which is appropriate

5. Model 6.1 Same as Model 5.3 Number of trunk lines=26 No additional staff during 5-8 hrs. 10 runs are made For terminating case, make IID replications Run>Setup>Replication Parameters: Number of Replications =10 Check both boxes for Initialize Between Replications Outputs are saved to.dat files Statistics Module, Type=output, Data file name= Filename.dat Asli Sencer5

6. Outputs of Model 6.1 Asli Sencer6 Category Overview report will have some statistical- analysis results of the output across the replications

7. Output Precision in Model 6.1 Asli Sencer7 This information (except standard deviation) is in Category Overview report If > 1 replication specified, Arena uses cross-replication data as above For other confidence levels or graphics – Output Analyzer

8. 8 Confidence Interval Estimation

9. Simulation with Arena Chapter 5 – Detailed Modeling and Terminating Statistical Analysis 9 Interpretation of a Confidence Interval

10. Required Number of Replications to Achieve a Certain Precision 10 Want this to be “small,” say < h where h is prespecified Chapter 5 – Detailed Modeling and Terminating Statistical Analysis Simulation with Arena

11. Chapter 5 – Detailed Modeling and Terminating Statistical Analysis 11 Half Width and Number of Replications s = sample standard deviation from “initial” number n 0 of replications h 0 = half width from “initial” number n 0 of replications n grows quadratically as h decreases.

12. Number of Replications Needed Asli Sencer12 If we require h=$250 rather than $812 for total cost, replications are needed.

13. Model 6.2 Asli Sencer13

14. Model 6.3 1000 Runs-as a trial Save the output to Total cost.dat Open Output analyzer as a separate application File>Data File>Export Export binary data in.dat file to a plain ASCII text file and save. Open Arena Input Analyzer Plot the histogram of the Total Costs Asli Sencer14

15. Histogram of 1000 Total Cost Values Since Total Cost values is a sum, law of large numbers apply. We see that the distribution approaches normal as the number of replications increase! Same is true for average statistics due to central limit theorem. It is not true for extreme value statistics like maximum or minimum. Asli Sencer15

16. Simulation with Arena Chapter 5 – Detailed Modeling and Terminating Statistical Analysis 16 Confidence Intervals (cont’d) Usual formulas assume normally-distributed data  Never true in simulation  Might be approximately true if output is an average, rather than an extreme  Central limit theorem  Issues of robustness, coverage, precision – details in book

17. Simulation with Arena Chapter 5 – Detailed Modeling and Terminating Statistical Analysis 17 Comparison of Alternatives Statistical Hypothesis Test Reject Ho if is significantly large or small, i.e., performance of system 1 is significantly different than system 2! Here: : Total Cost of Base Model (110 observations) : Total Cost of Alternative Model (110 observations) is the mean performance of system i

18. Comparing Two Scenarios Base Scenario: Model 6.4 (Same as in Model 5.3) -110 runs -26 Trunk Lines, No New Staff between 12:00-16:00 Alternative scenario: Model 6.4 (More-resources scenario) -110 runs -29 Trunk Lines, (Change the capacity from 26 to 29) -Hire three for each of Larry, Moe, Curly, Hermann and Sales Resources. (Change these variables from 0 to 3) Tradeoff is between increased salary cost but decreased excess waiting costs. Will the total costs decrease? Percent Rejected calls will decrease, but how much? Asli Sencer18

19. Comparison of Scenarios Runs both models for 110 times. Statistics Data Module Save output files –BaseCase.dat or -MoreResources.dat. 95% CI for total costs are Base model: 22,175.19 +- 369.54=[21,805, 22,544] Increased resources: 24,542.82 +- 329.11=[24,213, 24,871] Intervals do NOT overlap, hence Total Costs are significantly different at 5% significance level. 95% CI for percent rejected are Base model: 11.74 +- 0.51=[11.23, 12.25] Increased resources: 1.73 +- 0.31=[1.42, 2.04] Intervals do NOT overlap, hence Percent Rejected are significantly different at 5% significance level. Asli Sencer19

20. Arena Output Analyzer Separate application in Arena Operates in output files (.dat) generated by Arena through the Statistics data module Data in.dat file is in binary format to be opened by Arena Output Analyzer only! Provides confidence intervals on expected output statistics as also appear in Arena output reports. Provides statistical comparison of two scenarios, and others. Asli Sencer20

21. Comparison of Scenarios with Arena Output Analyzer Open Output Analyzer Select File>New to open a data group, i.e., list of.dat files Add TotalCost-BaseCase.dat TotalCost-MoreResources.dat PercentRejected-BaseCase.dat PercentRejected-MoreResources.dat Can save this data group as.dgr file to refer easily afterwards Analyze>Compare Means Add each pair of comparisons by choosing ‘lumped’ so that all 110 values are considered in the analysis Asli Sencer21

22. Hypothesis Tests Ho: Mean TC of base case = Mean TC of more resources Ha: Mean TC of base case ≠ Mean TC of more resources Ho: Mean % rejected of base case = Mean % rejected of more resources case Ha: Mean % rejected of base case ≠ Mean % rejected of more resources case Asli Sencer22

23. Output Report-Compare Means Asli Sencer23 Confidence interval on difference misses 0, so conclude that there is a (statistically) significant difference between the base model and the alternative at α=5%

24. Evaluating Many Scenarios with Process Analyzer Separate application in Arena Allows making multiple pairwise scenario comparisons at a time. PAN operates on Arena program files with.p extension, generated when.doe model is run. A PAN scenario includes a program file, a set of values for the input controls (decision variables in the form of variables and resources), a set of output responses. A PAN project is a collection of such scenarios that can be saved by.pan extension for future reference. Asli Sencer24

25. Development of a PAN Project Use Model 6.5 110 runs Output data files are deleted since they will be useless in PAN. Open a PAN project File > New, File > Open Add a new scenario. Double click on the raw Name=Base Case, Program File=Model 6.5.p Replications=110 Add contols Right click in this line OR Insert > Control Under Resources: The capacity of trunk line Under User Specified: New Tech 1, New Tech 2, New Tech 3, New Tech All, New Sales Add responses Right click on this line OR Insert > Responses Under user specified: Total Cost, Percent Rejected Asli Sencer25

26. New Scenarios Suppose you have $1360/week to spend on all additional resources. To which of the six expandable resources should you allocate the new money? Then following 6 alternative scenarios apply in addition to Base Case. 13 more trunk lines ($98 each) 4 more tech 1,2,3 people ($320 each) 3 more tech all people ($360 each) 4 more sales people ( 340 each) Run the scenarios Check the scenarios to run Run > Go OR play button OR F5 function key Asli Sencer26

27. PAN screen Asli Sencer27

28. Generating Reports for Multiple Comparison in PAN Insert > Chart OR right click on a response column. Chart type=Box whiskers Check Identify Best Scenarios box Select ‘smaller is the better’ Red boxes are significantly better than blue ones at 5% significance level. To decrease the half width of a scenario, increase the number of replications of that specific one. Error tolerance is a positive value that represents an amount small enough that you don’t care if the selected scenarios are actually inferior to the true best one by at most this amount. A positive error reduce the number of selected scenarios at the risk of being off by a little bit. Asli Sencer28

29. A PAN Report Asli Sencer29