1. Output Analysis for Simulation
Written by:
Marvin K. Nakayama
Presented by:
Jennifer Burke
MSIM 752

2. Outline Performance measures Output of a transient simulation
Techniques for steady-state simulations
Estimation of multiple performance measures
Other methods for analyzing simulation output

3. Example
Automatic Teller Machine (ATM)

4. Performance Measures Measure how well the simulation runs
Different types of simulations require different statistical techniques to analyze the results
Terminating (or transient)
Steady-state (or long run)

5. Terminating Performance Measures
Terminating simulation
Simulation will finish at a given event
Initial conditions have a large impact
Ex: Queue starts with no customers present

6. ATM example (Terminating)
Open 9:00am – 5:00pm
X = # of customers using ATM in a day
E(X)
P(X  500)
C = queue is empty

7. Output of a Terminating Simulation
Goal:
calculate E(X)
Approach:
n  2 i.i.d duplications
X1,X2,…,Xn
find the average of those duplications

8. Output of a Terminating Simulation
calculate the sample variance of X1,X2,…,Xn
and the sample standard deviation

9. Output of a Terminating Simulation
Central Limit Theorem
confidence interval for E(X)

10. Output of a Terminating Simulation
the confidence interval provides a form of error bound
Hn is the half-width of the confidence interval

11. ATM example (Terminating)
Expected daily withdraw
within $500
ε = 500
S(n) = sample standard
deviation

12. Steady-state Performance Measures
Steady-state simulation
Simulation that stabilizes over time
Initial condition C
Fi(y|C)
Fi(y|C) → F(y) as i → 

13. ATM example (Steady-state)
Open 24 hours a day
Yi = number of customers served on the ith day of operation
E(Y)
P(Y  400)
C = queue is empty

14. Output of a Steady-state Simulation
Case 1:
discrete-time process Y1,Y2,…,Yn
estimate v, as m → 
Case 2:
continuous-valued time index Y(s)
estimate v, as m → 

15. ATM (Continuous) Y(s) = number of customers waiting in line at time s
Assume Y(s) has a steady-state
Calculate v

16. Difficulties of Steady-state analysis
Discrete-time process
if m is large, then is a good approximation of v
Confidence interval

17. Simplifications to Steady-state Analysis
Multiple replications
Initial-data deletion
Single-replicate algorithm

18. Method of Multiple Replications
Estimate
r i.i.d replications, length k = m/r
10  r  30

19. Method of Multiple Replications
Average of jth row
Using find the sample mean

20. Method of Multiple Replications
Sample variance
Confidence interval

21. Problem with Multiple Replication Method
Simple estimation of variance
can be contaminated by initialization bias

22. Initial-Data Deletion
Partial solution
Delete first c observations
Replication mean
sample mean
sample variance
confidence interval

23. Single-Replicate Algorithm
Single simulation of length m + c
Divide the m observations into n batches
10  n  30
Batch mean

24. Single-Replicate Algorithm
Sample mean
Sample variance
Confidence interval

25. Estimating Multiple Performance Measures
Terminating simulations
Confidence interval for each performance measure
Joint confidence interval

26. ATM example (Terminating)
Open 9:00am – 5:00pm
μ1 = expected # of customers served in a day
μ2 = probability # served in a day is at least 1000
μ3 = expected amount of $ withdrawn in a day

27. Conclusions Basics of analyzing simulation output
Application potential is high
Not state of the art
Benefit
Lacked comparison