The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with SIMUL8 Corporation.

OUTLINE Introduction Methods Algorithm Test Methodology Test Results Extended Algorithm & Results Discussion Summary

Objective To provide an easy to use method, that can be incorporated into existing simulation software, that enables practitioners to obtain results of a specified accuracy from their discrete event simulation model. (Only looking at analysis of a single scenario)

Introduction Initial Setup:  Any warm-up problems already dealt with.  Run length (m) decided upon.  Modeller decided to use multiple replications to obtain better estimate of mean performance. Multiple replications performed by changing the random number streams used by the model and re-running the simulation. Output data from model Response measure of interest = summary statistic from rep1 = summary statistic from repN N replications

QUESTION IS… How many replications are needed? Limiting factors: computing time and expense. If performing N replications achieves a sufficient estimate of mean performance: > N replications: Unnecessary use of computer time and money. < N replications: Inaccurate results → incorrect decisions.

4 main methods found in the literature for choosing N: 1. Rule of Thumb Run at least 3 to 5 replications. Advantage: Very simple. Disadvantage: Does not use characteristics of model output. No measured precision level.

2. Simple Graphical Method Plot Cumulative mean -v- number of replications Visually select point where cumulative mean line becomes “flat”. Use this as N. Advantages: Simple Uses output of interest in decision. Disadvantages: Subjective No measured precision level.

3. Confidence Interval (with Specified Precision) Method User decides size of error they can tolerate. Run increasing numbers of replications, Construct Confidence Intervals around sequential cumulative mean of output variable until desired precision achieved. Advantages: Relies upon statistical inference to determine number of replications required. Allows the user to tailor accuracy of output results to their particular requirement or purpose for that model and result. Disadvantage: Many simulation users do not have the skills to apply such an approach.

4. Prediction Formula Method User decides size of error they can tolerate. Run a few replications, estimate variance & mean Use formula to predict N. Check desired precision achieved – if not amend N and repeat Advantages: Simple. Uses data from model. Provides specified precision. Disadvantage: Can be very inaccurate especially for small number of replications. If variance estimate low underestimate N If variance estimate high overestimate N

Chose to automate: Confidence Interval (with Specified Precision) Method

The replication algorithm interacts with the simulation model sequentially.

is the student t value for n-1 df and a significance of 1-α, s n is the estimate of the standard deviation, calculated using results X i (i = 1 to n) of the n current replications. Where n is the current number of replications carried out, We define the precision, d n, as the ½ width of the Confidence Interval expressed as a percentage of the cumulative mean: is the cumulative mean, ALGORITHM DEFINITIONS

Stopping Criteria Simplest method: Stop when d n 1st found to be ≤ desired precision, d required, and recommend that number of replications, Nsol, to the user. Problem: Data series could prematurely converge, by chance, to incorrect estimate of the mean, with precision d required, then diverge again. ‘Look-ahead’ procedure: When d n 1st found to be ≤ d required, algorithm performs set number of extra replications, to check that precision remains ≤ d required.

‘Look-ahead’ procedure kLimit = ‘look ahead’ value. Actual number of replications checked ahead is a function of this user defined value: Function relates ‘look ahead’ period length with current value of n.

Nsol Nsol + f(kLimit) f(kLimit) Precision ≤ 5% 95% confidence limits Cumulative mean, Replication Algorithm

Precision≤ 5% Precision> 5% Precision ≤ 5% f(kLimit) Nsol Nsol + f(kLimit) Nsol

24 artificial data sets created: Left skewed, symmetric, right skewed; Varying values of relative standard deviation (stdev/mean). Advantage: true mean and variance known. Artificial data set: 100 sequences of 2000 data values. 8 real models selected. Different lengths of ‘look ahead’ period looked at: kLimit values = 0 (i.e. no ‘look ahead’ period), 5, 10, 25. d required value kept constant at 5%. TESTING METHODOLOGY

5 performance measures 1.Coverage of the true mean 2.Bias 3.Absolute Bias 4.Average Nsol value 5.Comparison of 4. with Theoretical Nsol value For real models: ‘true’ mean and st.dev values - estimated from whole sets of output data (3000 to data points).

Results Nsol values for individual algorithm runs are very variable. Average Nsol values for 100 runs per model close to the theoretical values of Nsol. Normality assumption appears robust. Using a ‘look ahead’ period improves performance of the algorithm.

Mean bias significantly different to zero Failed in coverage of true mean Mean est. Nsol significantly different to theoretical Nsol (>3) No ‘look- ahead’ period Proportion of Artificial models 4/242/249/18 Proportion of Real models 1/8 3/5 kLimit = 5 Proportion of Artificial models 1/2401/18 Proportion of Real models 000

% decrease in absolute mean bias kLimit = 0 to kLimit = 5 kLimit = 5 to kLimit = 10 kLimit = 10 to kLimit = 25 Artificial Models 8.76%0.07%0.26% Real Models 10.45%0.14%0.33% Impact of different look ahead periods on performance of algorithm

Number of times the Nsol value changes (out of 100 runs of the algorithm per model) because of the lengthening of the ‘look ahead’ period. Model ID kLimit = 0 to kLimit = 5 kLimit = 5 to kLimit = 10 kLimit = 10 to kLimit = 25 R1000 R3200 R52401 R82441 A53013 A62663 A15100 A A A243700

Model IDkLimitNsolTheoretical Nsol (approx) Mean estimate significantly different to the true mean? A220464Yes 554No A904112Yes 5120No A Yes 5718No A210837Yes 538No R70310Yes 58No R4036Yes 57No R80345Yes 546No Eg.s of changes in Nsol & improvement in estimate of true mean

Model ID kLimitNsolTheoretical Nsol (approx) Mean estimate significantly different to the true mean? A904112Yes 5120No A Yes 5718No R70310Yes 58No R4036Yes 57No R80345Yes 546No Examples of changes in Nsol & improvement in estimate of true mean

DISCUSSION kLimit default value set to 5. Initial number of replications set to 3. Multiple response variables - Algorithm run with each response - use maximum estimated value for Nsol. Different scenarios - advisable to repeat algorithm every few scenarios to check that precision has not degraded significantly. Inclusion into simulation package: Full explanations of algorithm and results.

SUMMARY Selection and automation of Confidence Interval (with Specified Precision) Method for estimating the number of replications to be run in a simulation. Algorithm created with ‘look ahead’ period - efficient and performs well on wide selection of artificial and real model output. ‘Black box’ - fully automated and does not require user intervention.

ACKNOWLEDGMENTS This work is part of the Automating Simulation Output Analysis (AutoSimOA) project ( that is funded by the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project. Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07