1 Terminating Statistical Analysis By Dr. Jason Merrick

Simulation with Arena — Intermediate Modeling and Terminating Statistical Analysis C6/2 Statistical Analysis of Output Data: Terminating Simulations Random input leads to random output (RIRO) Run a simulation (once) — what does it mean? –Was this run “typical” or not? –Variability from run to run (of the same model)? Need statistical analysis of output data Time frame of simulations –Terminating: Specific starting, stopping conditions –Steady-state: Long-run (technically forever) –Here: Terminating

Simulation with Arena — Intermediate Modeling and Terminating Statistical Analysis C6/3 Point and Interval Estimation Suppose we are trying to estimate an output measure E[Y] =  based upon a simulated sample Y 1,…,Y n We come up with an estimate –For instance How good is this estimate? –Unbiased –Low Variance (possibly minimum variance) –Consistent –Confidence Interval

Simulation with Arena — Intermediate Modeling and Terminating Statistical Analysis C6/4 T-distribution The t-statistic is given by –If the Y 1,…,Y n are normally distributed and then the t- statistic is t-distributed –If the Y 1,…,Y n are not normally distributed, but then the t-statistic is approximately t-distributed thanks to the Central Limit Theorem requires a reasonably large sample size n –We require an estimate of the variance of denoted

Simulation with Arena — Intermediate Modeling and Terminating Statistical Analysis C6/5 T-distribution Confidence Interval An approximate confidence interval for is then –The center of the confidence interval is –The half-width of the confidence interval is – is the 100(  /2)% percentile of a t-distribution with f degrees of freedom.

Simulation with Arena — Intermediate Modeling and Terminating Statistical Analysis C6/6 T-distribution Confidence Interval Case 1: Y 1,…,Y n are independent –This is the case when you are making n independent replications of the simulations Terminating simulations Try and force this with steady-state simulations –Compute your estimate and then compute the sample variance –s 2 is an unbiased estimator of the population variance, so s 2 /n is an unbiased estimator of with f = n-1 degrees of freedom

Simulation with Arena — Intermediate Modeling and Terminating Statistical Analysis C6/7 T-distribution Confidence Interval Case 2: Y 1,…,Y n are not independent –This is the case when you are using data generated within a single simulation run sequences of observations in long-run steady-state simulations –s 2 /n is a biased estimator of –Y 1,…,Y n is an auto-correlated sequence or a time-series –Suppose that our point estimator for is, a general result from mathematical statistics is

Simulation with Arena — Intermediate Modeling and Terminating Statistical Analysis C6/8 T-distribution Confidence Interval Case 2: Y 1,…,Y n are not independent –For n observations there are n 2 covariances to estimate –However, most simulations are covariance stationary, that is for all i, j and k –Recall that k is the lag, so for a given lag, the covariance remains the same throughout the sequence –If this is the case then there are n-1 lagged covariances to estimate, denoted  k and

Simulation with Arena — Intermediate Modeling and Terminating Statistical Analysis C6/9 Time-Series Examples Positively correlated sequence with lag 1 Positively correlated sequence with lags 1 & 2 Negatively correlated sequence with lag 1 Positively correlated, covariance non-stationary sequence

Simulation with Arena — Intermediate Modeling and Terminating Statistical Analysis C6/10 T-distribution Confidence Interval Case 2: Y 1,…,Y n are not independent –What is the effect of this bias term? –For primarily positively correlated sequences B < 1, so the half-width of the confidence interval will be too small Overstating the precision => make conclusions you shouldn’t –For primarily negatively correlated sequences B > 1, so the half-width of the confidence interval will be too large Underestimating the precision => don’t make conclusions you should

Simulation with Arena — Intermediate Modeling and Terminating Statistical Analysis C6/11 Strategy for Terminating Simulations For terminating case, make IID replications –Simulate module: Number of Replications field –Check both boxes for Initialization Between Reps. –Get multiple independent Summary Reports –Different random seeds for each replication How many replications? –Trial and error (now) –Approximate no. for acceptable precision –Sequential sampling Save summary statistics (e.g. average, variance) across replications –Statistics Module, Outputs Area, save to files

Simulation with Arena — Intermediate Modeling and Terminating Statistical Analysis C6/12 Half Width and Number of Replications Prefer smaller confidence intervals — precision Notation: Confidence interval: Half-width = Want this to be “small,” say < h where h is prespecified

Simulation with Arena — Intermediate Modeling and Terminating Statistical Analysis C6/13 Half Width and Number of Replications To improve the half-width, we can –Increase the length of each simulation run and so increase the m i –What does increasing the run length do? –Increase the number of replications

Simulation with Arena — Intermediate Modeling and Terminating Statistical Analysis C6/14 Half Width and Number of Replications (cont’d.) Set half-width = h, solve for Not really solved for n (t, s depend on n) Approximation: –Replace t by z, corresponding normal critical value –Pretend that current s will hold for larger samples –Get Easier but different approximation: s = sample standard deviation from “initial” number n 0 of replications h 0 = half width from “initial” number n 0 of replications n grows quadratically as h decreases.