Output Data Analysis

How to analyze simulation data? simulation –computer based statistical sampling experiment –estimates are just particular realizations of random variables that may have large variances –n independent replications –each replication terminated by same event –started with same initial conditions –replications are independent by means of using different random variables –single measure of performance one per replication || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 2

obtained random numbers Y 1, Y 2, … Y m –is an output stochastic process from a single run –generally neither independent nor identically distributed –most formulas assuming IIDs not directly applicable y 11, y 12, … y 1m –realizations for random variables Y 1, Y 2, … Y m –resulting from making a simulation run of length m observations y 21, y 22, …, y 2m –realizations for random variables Y 1, Y 2, … Y m –if simulation is run again (using different random variables) || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 3

obtained random numbers (cont) || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 4 if you make n independent replications (runs) –with different random number used –observations from particular run/row not IID –observations from form i th column are IID observations of random variable Y i (i = 1..m) ! independence across runs y 11, y 12, …y 1i, ….y 1m y 21, y 22, ….y 2i, ….y 2m …….…. y n1, y n2, …y ni, …..y nm

Transient and Steady-State Behavior stochastic output process Y 1, Y 2,.. –transient condition: F i ( y | I ) = P(Y i · y | I) for i = 1, 2… –y is a real number –I represents initial conditions density f Y i –specifies how random variable Y i can vary from one replication to another F i (y | I ) ! F(y) as i ! 1 –F(y)steady-state distribution of output process Y 1, Y 2, … –in theory only obtained at limit –in practice ! finite time index (k+1) ! distributions will be approximately the same || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 5

Transient and Steady-State Behavior (cont.) || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 6

Types of Simulations terminating simulation non-terminating simulations –steady-state parameters –steady-state cycle parameters –other parameters || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 7 we’ll focus on this type only

Example bank –5 tellers, one queue –opens at 9:00 –closes at 17:00 (stays open until all customers in the bank have been served) –terminating simulation close at/after17:00 (as soon as all customers have left) || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 8

Example (cont.) || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 9 R# servedtimeavg Delayavg Length% C’s delayed < 5 minutes

Estimating Means point estimate and confidence interval for mean ¹ = E(X) –unbiased point estimator for ¹ –approximate 100(1- ® ) percent confidence interval for ¹ || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 10

Estimating Means (example) estimate expected delay – = – S 2 (n) = – confidence interval with ® = 10% estimated proportion of customers being delayed < 5 minutes –expected proportion for a given day/run indicator function – = 0.853S 2 (n) = – CI with ® = 10% || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 11

Obtaining a desired precision so far –fixed sample size procedure (based on n replications) –disadvantage: no control over the CI’s half length (i.e. precision of ) –half length depends on population variance S 2 (n) 2 ways to measure the error in the estimate –absolute error ¯ –relative error ° –resulting number of replications may be random || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 12

Obtaining a desired precision (absolute error ¯ ) absolute error ¯ –estimator has an absolute error of at most ¯ with a probability of approximately 1 - ® approximate expression for total number of replications n a * ( ¯ ) required to obtain an absolute error of ¯ –assumes that estimate S 2 (n) will not change (appreciately) as n increases) –n a * ( ¯ ) will be determined iteratively || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 13

Obtaining a desired precision (absolute error ¯ ) example (bank) –Q: what’s the number of replications necessary in order to estimate the expected average delay with an absolute error of 0.25 minutes and a confidence level of 90%? || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 14 it i-1,0.95 t i-1,0.95 * sqrt(0.309/i) · 0.25

Obtaining a desired precision (relative error ° ) relative error ° –estimator as a relative error of at most ° /(1 - ° ) with a probability of approximately 1 - ®. approximate expression for total number of replications n a * ( ¯ ) required to obtain a relative error of ° –assumes that estimate S 2 (n) will not change (appreciately) as n increases) –n r* ( ° ) will be determined iteratively || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 15

Obtaining a desired precision (relative error ° ) example (bank) –Q: what’s the number of replications necessary in order to estimate the expected average delay with a relative error of 10% and a confidence level of 90%? || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 16 it i-1,0.95 t i-1,0.95 * sqrt(0.309/i) / mean ·

Estimating other Measures of Performance be careful! –comparing two systems by some sort of mean may result in misleading conclusions example: 2 bank policies –5 queues (one in front of every teller) –1 queue (that feeds all tellers) || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 17 Measure of performanceFive queuesOne queue Expected operating time (hours)8.14 Expected average delay (minutes)5.57 Expected average number in queue(s)5.52

Estimating other Measures of Performance Estimates of expected proportions of delays in interval || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 18 Interval (minutes)Five queuesOne queue [0,5) [5,10) [10,15) [15,20) [20,25) [25,30)0.020 [30,35) [35,40) [40,45)00 still identical?

Choosing initial conditions careful! –measures of performance depend explicitly on the state of the system at time 0 –take care when choosing appropriate initial conditions example: estimate expected average delay at bank between noon and 1pm –bank will probably be quite congested at noon starting with no customers present -> estimates will be biased low || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 19

Choosing initial conditions careful! –measures of performance depend explicitly on the state of the system at time 0 –take care when choosing appropriate initial conditions 2 heuristic approaches –use warmup period –collect data to get an idea of state of system and choose it randomly || WS 2008 || Dr. Verena Schmid || PR KFK PM/SCM/TL Praktikum Simulation I 20