1 Simulation Modeling and Analysis Output Analysis

2 Outline Stochastic Nature of Output Taxonomy of Simulation Outputs Measures of Performance –Point Estimation –Interval Estimation Output Analysis in Terminating Simulations Output Analysis in Steady-state Simulations

3 Introduction Output Analysis –Analysis of data produced by simulation Goal –To predict system performance –To compare alternatives Why is it needed? –To evaluate the precision of the simulation performance parameter as an estimator

4 Introduction -contd Each simulation run is a sample point Attempts to increase the sample size by increasing run length may fail because of autocorrelation Initial conditions affect the output

5 Stochastic Nature of Output Data Model Input Variables are Random Variables The Model Transforms Input into Output Output Data are Random Variables Replications of a model run can be obtained by repeating the run using different random number streams

6 Example: M/G/1 Queue Average arrival rate Poisson with = 0.1 per minute Service times Normal with  = 9.5 minutes and  = 1.75 minutes Runs – One 5000 minute run –Five 1000 minute runs w/ 3 replications each

7 Taxonomy of Simulation Outputs Terminating (Transient) Simulations –Runs until a terminating event takes place –Uses well specified initial conditions Non-terminating (Steady-state) Simulations –Runs continually or over a very long time –Results must be independent of initial data –Termination? What determines the type of simulation?

Examples: Non-terminating Systems Many shifts of a widget manufacturing process. Expansion in workload of a computer service bureau. 8

9 Measures of Performance: Point Estimation Means Proportions Quantiles

10 Measures of Performance: Point Estimation (Discrete-time Data) Point estimator of  (of  ) based on the simulation discrete-time output (Y 1, Y 2,.., Y n )  * = (1/n)  i n Y i Unbiased point estimator E(  * ) =  Bias b = E(  * ) - 

11 Measures of Performance: Point Estimation (Continuous-time data) Point estimator of  (of  ) based on the simulation continuous-time output (Y(t), 0 < t < T e )  * = (1/ T e )  0 Te Y(t) dt Unbiased point estimator E(  * ) =  Bias b = E(  * ) - 

12 Measures of Performance: Interval Estimation (Discrete-time Data) Variance and variance estimator  2 (  ) = true variance of point estimator   2* (  ) = estimator of variance of point estimator  Bias (in variance estimation) B = E(  2* (  ) )/  2 (  )

13 Measures of Performance: Interval Estimation - contd If B ~ 1 then t = (  -  )/  2* (  ) has t  /2,f distribution (d.o.f. = f). I.e. A 100(1 -  )% confidence interval for  is  - t  /2,f  2* (  ) <  <  + t  /2,f  2* (  ) Cases –Statistically independent observations –Statistically dependent observations (time series).

14 Measures of Performance: Interval Estimation - contd Statistically independent observations –Sample variance S 2 =  i n (Y i -  ) 2 /(n-1) –Unbiased estimator of  2 (  )  2* (  ) = S 2 /n –Standard error of the point estimator   * (  ) = S /  n

15 Measures of Performance: Interval Estimation - contd Statistically dependent observations –Variance of   2 (  ) = (1/n 2 )  i n  j n cov(Y i, Y j ) –Lag k autocovariance  k = cov(Y i, Y i+k ) –Lag k autocorrelation  k =  k  0

16 Measures of Performance: Interval Estimation - contd Statistically dependent observations (contd) –Variance of   2 (  ) = (  0 /n) [  k=1 n-1 (1- k/n)  k ] = (  0 /n) c –Positively autocorrelated time series (  k > 0) –Negatively autocorrelated time series (  k < 0) –Bias (in variance estimation) B = E(S 2 /n )/  2 (  ) = (n/c - 1)/(n-1)

17 Measures of Performance: Interval Estimation - contd Statistically dependent observations (contd) Cases –Independent data  k = 0, c = 1, B = 1 –Positively correlated data  k > 0, c > 1, B < 1, S 2 /n is biased low (underestimation) –Negatively correlated data  k 1, S 2 /n is biased high (overestimation)

18 Output Analysis for Terminating Simulations Method of independent replications –n = Sample size –Number of replications r=1,2,…,R –Y ji i-th observation in replication j –Y ji, Y jk are autocorrelated –Y ri, Y sk are statistically independent –Estimator of mean (r =1,2,…,R)  r  (1/n r )  i n r Y ri

19 Output Analysis for Terminating Simulations - contd Confidence Interval (R fixed; discrete data) –Overall point estimate  * = (1/R)  1 R  r  –Variance estimate   * (  *) = [1/(R-1)R]  1 R (  r   –Standard error of the point estimator   * (  ) =    * (  *)

20 Output Analysis for Terminating Simulations - contd Estimator and Interval (R fixed; continuous data) –Estimator of mean (r =1,2,…,R)  r  (1/T e )  0 Te Y r (t) dt Overall point estimate  * = (1/R)  1 R  r  –Variance estimate   * (  *) = [1/(R-1)R]  1 R (  r  

21 Output Analysis in Terminating Simulations - contd Confidence Intervals with Specified Precision Half-length confidence interval (h.l.) h.l. = t  /2,f  2* (  ) = t  /2,f S/  R <  Required number of replications R* > ( z  /2 S o /  ) 2

22 Output Analysis for Steady State Simulations Let (Y 1, Y 2,.., Y n ) be an autocorrelated time series Estimator of the long run measure of performance  (independent of I.C.s)  = lim n =>  (1/n)  i n Y i Sample size n (or T e ) is design choice.

23 Output Analysis for Steady State Simulations -contd Considerations affecting the choice of n –Estimator bias due to initial conditions –Desired precision of point estimator –Budget/computer constraints

24 Output Analysis for Steady State Simulations -contd Initialization bias and Initialization methods –Intelligent initialization Using actual field data Using data from a simpler model –Use of phases in simulation Initialization phase (0 < t < To; for i=1,2,…,d) Data collection phase (To < t < Te; for i=d+1,d+2,…,n) Rule of thumb (n-d) > 10 d

25 Output Analysis for Steady State Simulations -contd Example M/G/1 queue –Batched data –Batched means –Averaging batch means within a replication (I.e. along the batches) –Averaging batch means within a batch (I.e. along the replications).

26 Steady State Simulations: Replication Method Cases 1.- Y rj is an individual observation from within a replication 2.- Y rj is a batch mean of discrete data from within a replication 3.- Y rj is a batch mean of continuous data over a given interval

27 Steady State Simulations: Replication Method -contd Sample average for replication r of all (nondeleted) observations Y* r (n,d) = Y* r = [1/(n-d)]  j=d+1 n Y rj Replication averages are independent and identically distributed RV’s Overall point estimator Y*(n,d) = Y* = [1/R]  r=1 R Y r (n,d)

28 Steady State Simulations: Replication Method -contd Sample Variance S 2 = [1/(R-1)]  r=1 R (Y* r - Y*) Standard error = S/  R 100(1-  )% Confidence interval Y* - t  /2,R-1 S/  R <  < Y* + t  /2,R-1 S/  R

29 Steady State Simulations: Sample Size Greater precision can be achieved by –Increasing the run length –Increasing the number of replications

30 Steady State Simulations: Batch Means for Interval Estimation Single, long replication with batches –Batch means treated as if they were independent –Batch means (continuous) Y* j = (1/m)  (j-1)m jm Y(t) dt –Batch means (discrete) Y* j = (1/m)  i=(j-1)m jm Y i

31 Steady State Simulations: Batch Size Selection Guidelines Number of batches < 30 Diagnose correlation with lag 1 autocorrelation obtained from a large number of batch means from a smaller batch size For total sample size to be selected sequentially allow batch size and number of batches grow with run length.