1. Modeling and Simulation: Exploring Dynamic System Behaviour
Chapter 6
Experimentation and Output Analysis

2. Generation of Output Data
PSOV
Time variables
Sample variables
DSOV
Scalar derived from the PSOV’s
Output variables are random variables, thus a simulation run provides a sample value
Objective is to find a mean (sample mean) for the random variable
In reality want to find a confidence interval for the mean (how close is it to the real mean).
SSOV
Simple Scalar Output Variables
Can defined output directly
Number of balking customers
Number of passengers that left bus stop

3. Output Data from Multiple Runs

4. Bounded Horizon Study Characteristics Point Estimate Interval Estimate
Observation interval is well defined (implicit or explicit)
Transients in the stochastic processes are common
The simulation is run n times to produce n independent values for the DSOV
Point Estimate
Mean of the DSOV output values
Interval Estimate
Confidence interval for the point estimate or
Margin of error for the point estimate.

5. Point Estimate
Select a value n that defines the number of simulation runs that creates n values for the DSOV of interest.
Collect the n observed values y1, y2, … yn
Compute the point estimate (which is an estimate of the mean of our output variable E[Y]) as:
The point estimate is itself a random variable
Central Limit Theorem states that the point estimate is Normally distributed provided that the collected values are independent and identically distributed (no matter what the distribution of Y).
This will be the case provided that the simulation runs are independent (different seeds for the random number generators set at the start of each simulation run).

6. Interval Estimate Question – how good is the point estimate?
Can compute a confidence interval:
Where
tn-1,a is obtained from the t-distribution (see Annex 1)
n correspond to the number of simulations runs
a is related to the level of confidence C, i.e. = (1-C)/2,
C has a value between 0 and 1
Level of confidence defined as 100C %: is the probability that the real mean µ falls into the interval

7. Interval estimate viewed as quality criterion
With confidence 100C%, want
Thus want interval half-length ζ to be less than ζ*
For example select ζ* as a percentage of the point estimate r where r has a value over the range (0,1)
Thus, want to find
The number of runs are increased to meet the quality criterion r

8. Applying quality criterion
Select a value for r, C, and n (no less than 30).
Collect the n observed values y1, y2, … yn
Compute
If ζ < r , accept the point estimate.
Otherwise, increase n and start again.

9. Example: Kojo’s Kitchen

10. Steady-State Studies Characteristics
Stochastic processes of interest are stationary
Right-boundary of observation interval not fixed
Length of run can be used to generate required output data and meet quality criterion
Need to reach steady-state – the warm-up period
Defining experiments
Replication-deletion method
Batch means method

11. Warm-up Period
Define a period after which output stochastic process of interest has reached steady state
Demonstrate with Welch’s Method to define such a period

12. Reaching Steady-State

13. Welch’s Moving Average Method
Use a small number of replications, say 5 to 10.
Select observation interval long enough to reach steady state.
Divide the time scale into time cells
Time cells should be large enough data points to compute an average
In each time cell for each run, compute an average , where i is the index of the time cell and j the index of the simulation run.
Compute an overall average for each time cell.

14. Welch’s Method - continued
Define a window w to compute a moving average
Smoothes out choppy values

15. Warm-up Period for Port Model
Use 10 replications
Observation interval: 15 weeks
No apparent transient for the Berth Group size output
Due to small size of the group
Required w=5 to smooth out graph for tanker waiting times.

16. Port Berth Group Size:

17. Port Berth Group Size:

18. Port Tanker Waiting Time:

19. Port Tanker Waiting Time:

20. Port Tanker Waiting Time:

21. Replication-Deletion Method
Define a right-hand boundary to collect sufficient data for providing meaningful DSOV
Apply experimentation and output analysis applied to the Bounded Horizon Study
Compute point estimate and interval estimate
Note: increasing the run length reduces the confidence interval

22. Modify applying quality criterion
Select a value for r, C, n (no less than 30) and initial value for tf (right hand boundary of observation interval).
Collect the n observed values y1, y2, … yn
Compute
If ζ < r , accept the point estimate.
Otherwise, increase n or increase the run length and start again.

23. Batch Means Method Use one single long run
Divide the long run into time cells sufficiently long so that values for output variable are IID

24. Port Model Experimentation and Output Analysis (3 berths)

25. Port Model Experimentation and Output Analysis (4 berths)

26. Comparing Two Alternatives
Basically define a difference between the output values of each alternative

27. Interpreting the Results
Let
If CI(n) lies entirely to the right of zero (all positive) then the result of case 2 exceeds the result of case 1 with a level of confidence given by 100C%.
If CI(n) lies entirely to the left of zero (all negative) then the result of case 1 exceeds the result of case 2 with a level of confidence given by 100C%.
If CI(n) includes zero then at the level of confidence, 100C%., there is no meaningful difference between the two cases.

28. Common Random Numbers
Seeds to the random number generators are varied to ensure IID of output data from simulation run to simulation run
If different seeds are used for the runs of both alternatives, output results will lack symmetry
Re-use the same seeds with the random number generators for each alternative
Use different random number generators
Assign service time random values to attributes at the arrival of consumer entities into the model
Repeat the seeds used in the simulation runs of the first alternative for the simulation runs of the second alternative
The effect is to reduce the variance in the difference between the alternatives.
See the Table 6.6 and Table 6.7 for comparing the two alternatives of the port project.

29. Comparing Multiple Alternatives
With multiple alternatives
Can compare to base alternative only
Can compare all alternatives with each other
Must keep number of comparisons small
Overall confidence level parameter C given K comparisons (each with confidence CK).
Directly related to the confidence levels selected for each comparison (based on Bonferonni inequality)
For the case where all comparison confidence levels are equal, then

30. Examples – Multiples Alternatives
Given 5 alternatives
Want 90% overall confidence (C=0.90)
Thus Ck used for each individual confidence interval becomes 1 – ( (1-C)/K ) = 1 – (.10/5) = 0.98 (98%)
Same reasoning applies to multiple performance measurements
Port problem, K=4, since two alternatives and 2 performance measures (group size and tanker waiting time)
Used Ck = 0.95 (95%), therefore
C = 1-(K*(1-CK)) = 0.80 (80%)

31. Kojo’s Kitchen Example of Multiple Alternatives
Choose 3 comparisons, alternate cases to the base case.
Select CK=0.968 which gives a value of C=.904 (i.e. an overall confidence level of 90%)
Consider

32. Results for Multiple Scheduling Alternatives