1 OUTPUT ANALYSIS FOR SIMULATIONS

2 Introduction Analysis of One System Terminating vs. Steady-State Simulations Analysis of Terminating Simulations Obtaining a Specified Precision Analysis of Steady-State Simulations Method of Batch Means Outline

3 Introduction After understanding the under laying process, collecting data, fitting data to a distribution, coding and debugging the simulation program  selecting a performance measure to evaluate the system  evaluating your design by runs But by doing one or two runs, is it enough to evaluate your system?  Answer is No.  Because components driving your simulation include randomness, the output of simulation is also random The output is not independent and identically distributed (i.i.d), we can not use classical statistical methods

4 What Outputs to Watch? Performance measure - criteria that evaluate how god your system is ê Average, and worst (longest) time in system ê Average, and worst time in queue(s) ê Average hourly production ê Standard deviation of hourly production ê Proportion of time a machine is up, idle, or down ê Maximum queue length ê Average number of parts in system

5 Types of Simulations with Regard to Output Analysis Transient : A simulation where there is a specific starting and stopping condition that is part of the model.  transient performance measures: the performance of system finite horizon Steady-state: A simulation where there is no specific starting and ending conditions. Here, we are interested in the steady-state behavior of the system.  Steady-state performance measures: the performance for infinite horizon “The type of analysis depends on the goal of the study.”

6 Analysis for Transient Simulations Objective: Obtain a point estimate and confidence interval for some parameter Examples: = E (average time in system for n customers) = E (machine utilization) = E (work-in-process) Reminder: Can not use classical statistical methods within a simulation run because observations from one run are not independently and identically distributed (i.i.d.)

7 Analysis for Transient Simulations Make n independent replications of the model Let Y i be the performance measure from the ith replication Y i = average time in system, or Y i = work-in-process, or Y i = utilization of a critical facility Performance measures from different replications, Y 1, Y 2,..., Y n, are i.i.d. But, only one sample is obtained from each replication Apply classical statistics to Y i ’s, not to observations within a run Select confidence level 1 –  (0.90, 0.95, etc.)

8 Analysis for Transient Simulations  Approximate 100(1 – a)% confidence interval for  : estimator of  estimator of Var(Y i ) covers  with approximate probability (1 – a) is the Half-Width expression

9 Consider a single-server (M/M/1) queue. The objective is to calculate a confidence interval for the delay of customers in the queue. n = 10 replications of a single-server queue Y i = average delay in queue from ith replication Y i ’s: 2.02, 0.73, 3.20, 6.23, 1.76, 0.47, 3.89, 5.45, 1.44, 1.23 For 90% confidence interval, = 0.10 = 2.64, = 3.96, t 9, 0.95 = Approximate 90% confidence interval is 2.64 ± 1.15, or [1.49, 3.79] Example

10 Analysis for Transient Simulations Interpretation : 100(1 – a)% of the time, the confidence interval formed in this way covers  Wrong Interpretation : “I am 90% confident that   is between 1.49 and 3.79”

11 Issue 1 This confidence-interval method assumes Y i ’s are normally distributed. In real life, this is almost never true. Because of central-limit theorem, as the number of replications (n) grows, the coverage probability approaches 1 – a. In general, if Y i ’s are averages of something, their distribution tends not to be too asymmetric, and the confidence- interval method shown above has reasonably good coverage.

12 The confidence interval may be too wide In the M/M/1 queue example, the approximate 90% C.I. was: 2.64 ± 1.15, or [1.49, 3.79] The half-width is 1.15 which is 44% of the mean (1.15/2.64) That means that the C.I. is % which is not very precise. To decrease the half-width: Increase n until is small enough (this is called Sequential Sampling) There are two ways of defining the precision in the estimate Y: ê Absolute precision ê Relative precision Issue 2

13 Obtaining a Specified Precision

14 Obtaining a Specified Precision

15 Obtaining a Specified Precision Relative Precision:

16 Analysis for Steady-State Simulations Objective: Estimate the steady state mean Basic question: Should you do many short runs or one long run ?????

17 Analysis for Steady-State Simulations Advantages: ê Many short runs: l Simple analysis, similar to the analysis for terminating systems l The data from different replications are i.i.d. ê One long run: l Less initial bias l No restarts Disadvantages ê Many short runs: l Initial bias is introduced several times ê One long run: l Sample of size 1 l Difficult to get a good estimate of the variance

18 Analysis for Steady-State Simulations Make many short runs: The analysis is exactly the same as for terminating systems. The (1 – a)% C.I. is computed as before. Problem: Because of initial bias, may no longer be an unbiased estimator for the steady state mean,. Solution: Remove the initial portion of the data (warm-up period) beyond which observations are in steady-state. Specifically pick l (warm-up period) and n (number of observations in one run) such that

19 Analysis for Steady-State Simulations Make one Long run: Make just one long replication so that the initial bias is only introduced once. This way, you will not be “throwing out” a lot of data. Problem: How do you estimate the variance because there is only one run? Solution: Several methods to estimate the variance: ê Batch means (only approach to be discussed) ê Time-series models ê Spectral analysis ê Standardized time series

20 Method of Batch Means Divide a run of length m into n adjacent “batches” of length k where m = nk. Let be the sample or (batch) mean of the jth batch. The grand sample mean is computed as

21 Method of Batch Means The sample variance is computed as The approximate 100(1 – a )% confidence interval for is

22 Method of Batch Means Two important issues : Issue 1: How do we choose the batch size k? ê Choose the batch size k large enough so that the batch means, are approximately uncorrelated. Otherwise, the variance,, will be biased low and the confidence interval will be too small which means that it will cover the mean with a probability lower than the desired probability of (1 – a ).

23 Method of Batch Means Issue 2: How many batches n? ê Due to autocorrelation, splitting the run into a larger number of smaller batches, degrades the quality of each individual batch. Therefore, 20 to 30 batches are sufficient.