1. Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies ORGS Meeting, 4th October 2007 http://www.wbs.ac.uk/go/autosimoa

2. The Problem Prevalence of simulation software: ‘easy-to-develop’ models and use by non-experts. Simulation software generally have very limited facilities for directing/advising on simulation experiments. Main exception is directing scenario selection through ‘optimisers’. With a lack of the necessary skills and support, it is highly likely that simulation users are using their models poorly.

3. The Problem Despite continued theoretical developments in simulation output analysis, little is being put into practical use. There are 3 factors that seem to inhibit the adoption of output analysis methods: Limited testing of methods Requirement for detailed statistical knowledge Methods generally not implemented in simulation software (AutoMod/AutoStat is an exception) A solution would be to provide an automated output ‘Analyser’.

4. An Automated Output Analyser Simulation model Warm-up analysis Run-length analysis Replications analysis Use replications or long-run? Recommendation possible? Recommend- ation Output data Analyser Obtain more output data For this project the Analyser looks at: Warm-up Run-length Number of replications Scenario analysis could be added.

5. A 3 year, EPSRC funded project in collaboration with SIMUL8 Corporation. The AutoSimOA Project Objectives To determine the most appropriate methods for automating simulation output analysis To determine the effectiveness of the analysis methods To revise the methods where necessary in order to improve their effectiveness and capacity for automation To propose a procedure for automated output analysis of warm-up, replications and run-length Only looking at analysis of a single scenario

6. The AutoSimOA Project WORK CARRIED OUT: 1.Literature review of warm-up, replications and run-length methods. 2.Creation of a representative and sufficient set of models / data output for testing chosen simulation output analysis methods. 3.Development of an automated algorithm for estimating the number of replications to run. 4.Selection and testing of warm-up methods from the literature.

7. Part 1. Creation of models and data sets

8. AIMS:  Provide a representative and sufficient set of models / data output for use in discrete event simulation research.  Use models / data sets to test the chosen simulation output analysis methods in the AutoSimOA Project..

9. Group A …Group N Group B Auto Correlation Normality Cycling/Seasonality Terminating Non-terminating Steady state In/out of control Transient

10. Model characteristics  Deterministic or random  Significant pre- determined model changes (by time)  Dynamic internal changes i.e. ‘feed- back’  Empty-to-empty pattern  Initial transient (warm-up)  Out of control trend ρ≥1  Cycle  Auto-correlation  Statistical distribution Output data characteristics

11. Artificial Data: Construct data which resembles real model output with known values for some specific attribute. Example: Known mean and variance. Example data: AR(1) with N(0,1) errors. Real Models: Collect range of models created in “real circumstances”. Examples: Swimming Pool complex: average number in system Production Line Manufacturing Plant: through-put / hour Fast Food Store: average queuing time

12. Part 2. Automating analysis of number of replications

13. Introduction Initial Setup:  Any warm-up problems already dealt with.  Run length (m) decided upon.  Modeller decided to use multiple replications to obtain better estimate of mean performance. Multiple replications performed by changing the random number streams used by the model and re-running the simulation. Output data from model Response measure of interest = summary statistic from rep1 = summary statistic from repN N replications

14. QUESTION IS… How many replications are needed? Limiting factors: computing time and expense. If performing N replications achieves a sufficient estimate of mean performance: > N replications: Unnecessary use of computer time and money. < N replications: Inaccurate results → incorrect decisions.

15. 4 main methods found in the literature for choosing N: 1. Rule of Thumb Run at least 3 to 5 replications. Advantage: Very simple. Disadvantage: Does not use characteristics of model output. No measured precision level.

16. 2. Simple Graphical Method Plot Cumulative mean -v- number of replications Visually select point where cumulative mean line becomes “flat”. Use this as N. Advantages: Simple Uses output of interest in decision. Disadvantages: Subjective No measured precision level.

17. 3. Confidence Interval Method User decides size of error they can tolerate. Run increasing numbers of replications, Construct Confidence Intervals around sequential cumulative mean of output variable until desired precision achieved. Advantages: Relies upon statistical inference to determine number of replications required. Allows the user to tailor accuracy of output results to their particular requirement or purpose for that model and result. Disadvantage: Many simulation users do not have the skills to apply such an approach.

18. 4. Prediction Formula Decide size of error ε that can be can tolerated. Run ≥ 2 replications - estimate variance s 2. Solve to predict N. Check desired precision achieved – if not recalculate N with new estimate of variance. Advantages: Simple. Uses output of interest in decision. Provides specified precision. Disadvantage: Can be very inaccurate especially for small number of replications. If variance estimate low underestimate N If variance estimate high overestimate N

19. AUTOMATE Confidence Interval Method: Algorithm interacts with simulation model sequentially.

20. is the student t value for n-1 df and a significance of 1-α, s n is the estimate of the standard deviation, calculated using results X i (i = 1 to n) of the n current replications. Where n is the current number of replications carried out, We define the precision, d n, as the ½ width of the Confidence Interval expressed as a percentage of the cumulative mean: is the cumulative mean, ALGORITHM DEFINITIONS

21. Stopping Criteria Simplest method: Stop when d n 1st found to be ≤ desired precision, d required, and recommend that number of replications, Nsol, to the user. Problem: Data series could prematurely converge, by chance, to incorrect estimate of the mean, with precision d required, then diverge again. ‘Look-ahead’ procedure: When d n 1st found to be ≤ d required, algorithm performs set number of extra replications, to check that precision remains ≤ d required.

22. ‘Look-ahead’ procedure kLimit = ‘look ahead’ value. Actual number of replications checked ahead is a function of this user defined value: Function relates ‘look ahead’ period length with current value of n.

23. Nsol Nsol + f(kLimit) f(kLimit) Precision ≤ 5% 95% confidence limits Cumulative mean, Replication Algorithm

24. Precision≤ 5% Precision> 5% Precision ≤ 5% f(kLimit) Nsol 2 Nsol 2 + f(kLimit) Nsol 1

25. 24 artificial data sets created: Left skewed, symmetric, right skewed; Varying values of relative standard deviation (stdev/mean). Advantage: true mean and variance known. Artificial data set: 100 sequences of 2000 data values. 8 real models selected. Different lengths of ‘look ahead’ period looked at: kLimit values = 0 (i.e. no ‘look ahead’ period), 5, 10, 25. d required value kept constant at 5%. TESTING METHODOLOGY

26. 5 performance measures 1.Coverage of the true mean 2.Bias 3.Absolute Bias 4.Average Nsol value 5.Comparison of 4. with Theoretical Nsol value For real models: ‘true’ mean & variance values - estimated from whole sets of output data (3000 to 11000 data points).

27. Results Nsol values for individual algorithm runs are very variable. Average Nsol values for 100 runs per model close to the theoretical values of Nsol. Normality assumption appears robust. Using a ‘look ahead’ period improves performance of the algorithm.

28. Mean bias significantly different to zero Failed in coverage of true mean Mean est. Nsol significantly different to theoretical Nsol (>3) No ‘look- ahead’ period Proportion of Artificial models 4/242/249/18 Proportion of Real models 1/8 3/5 kLimit = 5 Proportion of Artificial models 1/2401/18 Proportion of Real models 000

29. % decrease in absolute mean bias kLimit = 0 to kLimit = 5 kLimit = 5 to kLimit = 10 kLimit = 10 to kLimit = 25 Artificial Models 8.76%0.07%0.26% Real Models 10.45%0.14%0.33% Impact of different look ahead periods on performance of algorithm

30. Number of times the Nsol value changes (out of 100 runs of the algorithm per model) because of the lengthening of the ‘look ahead’ period. Model ID kLimit = 0 to kLimit = 5 kLimit = 5 to kLimit = 10 kLimit = 10 to kLimit = 25 R1000 R3200 R52401 R82441 A53013 A62663 A15100 A172201 A212521 A243700

31. Model ID kLimitNsolTheoretical Nsol (approx) Mean estimate significantly different to the true mean? A904112Yes 5120No A2403755Yes 5718No R70310Yes 58No R4036Yes 57No R80345Yes 546No Examples of changes in Nsol & improvement in estimate of true mean

32. INCORPORATING A FAIL SAFE INTO THE ALGORITHM Problem: If model runs ‘slowly’ the algorithm could take an unacceptable amount of time to reach the set precision. “Fail Safe” - warn the user when a model may require a ‘long time’ to reach d required. At each iteration of the algorithm estimate Nsol using : Only as accurate as current estimate of st.dev and mean. Can be very inaccurate for small n.

33. Aid to user: Report approx. time to algorithm termination User judgment: Let algorithm progress naturally or terminate prematurely. A range of typical behaviour of Nsol* values

34. Extended Algorithm Proposal: Cumulative mean line should be reasonably ‘flat’. Extra stability criteria added into algorithm: –Algorithm ‘draws’ two parallel lines - Inner Precision Limits (IPLs) - around the cumulative mean line. –IPLs: Defined as a percentage of the d required value. –Stability criteria violated: If cumulative mean crosses either IPL within ‘look ahead’ period. Tested on the real and artificial models.

35. Stability Criteria Results Causes final Nsol recommendation to be associated with a much smaller precision than user requested… …but does not significantly reduce bias. Causes algorithm to be unnecessarily complicated. Can cause confusion in the user. Equivalent results produced by setting a smaller precision (d required ) - much more easily understood by user. Hence: Extra stability criteria dropped from replication algorithm.

36. Replication Work Discussion kLimit default value set to 5. Initial number of replications set to 3. ‘Fail safe’ - Aid for user to decide to prematurely end the algorithm. Stability criteria did not significantly enhance algorithm performance – dropped. Multiple response variables - Algorithm run with each response - use maximum estimated value for Nsol. Different scenarios - advisable to repeat algorithm every few scenarios to check that precision has not degraded significantly. Inclusion into SIMUL8 package: Full explanations of algorithm and results.

37. Summary Of Replications Work Selection and automation of Confidence Interval Method for estimating the number of replications to be run in a simulation. Algorithm created with ‘look ahead’ period - efficient and performs well on wide selection of artificial and real model output. ‘Black box’ - fully automated and does not require user intervention.

38. Part 3. WORK IN PROGRESS Automating estimation of warm-up length

39. The Initial Bias Problem Model may not start in a “typical” state. This may cause initial bias in the output. Many methods proposed for dealing with initial bias: e.g. Initial steady state conditions; run model for ‘long’ time… This project uses: Deletion of the initial transient data by specifying a warm-up period.

40. Question is: How do you estimate the length of the warm-up period required?

41. 5 main types of methods: 1. Graphical Methods. 2. Heuristic Approaches. 3. Statistical Methods. 4. Initialisation Bias Tests. 5. Hybrid Methods.

42. Literature search – 42 methods Summary of methods and literature references on project web site: http://www.wbs.ac.uk/go/autosimoa

43. Creation of artificial data sets with initial bias. Aim: Controllable & comparable data for testing warm-up methods. Create initial bias Create steady state

44. 1.Artificial Initial Bias Functions i)LENGTH OF BIAS FUNCTION n = total data length Truncation point = L = initial bias proportion (%) * n / 100 Set initial bias proportion value to: 10%, 20%, 40% of total data size, n. Three Criteria:

45. ii) SEVERITY OF BIAS FUNCTION Set maximum value of bias fn, a(t), so that max |a(t)| t≤L = M×Q Q = difference between steady state mean and 1st (if bias fn +ve) or 99th (if bias fn –ve) percentile of the steady state data. M = relative maximum bias – user set: 1, 2, 5 M ≥ 1 → bias significantly separate from steady state data → easier to detect. M ≤ 1 → bias absorbed into steady state data variance → harder to detect.

46. iii)SHAPE OF BIAS FUNCTION Mean Shift: Linear: Quadratic: Exponential: Oscillating (decreasing):

47. 2. Artificial Steady State Functions i) Constant steady state variance ii) Error Terms: Normal or Exponential distribution; Using L’Ecuyer RNG iii) Auto-Correlation: No AutoCorrelation; AR(1); AR(2); AR(4); MA(2); ARMA(5,5). iv) Superpostion: Bias Fn added onto end of steady state function: E.g.

48. PROJECT OVERVIEW Created set of artificial and “real” model data including warm-up bias functions. Created replication algorithm. Currently: Testing warm-up methods.

49. ACKNOWLEDGMENTS This work is part of the Automating Simulation Output Analysis (AutoSimOA) project that is funded by the UK (EPSRC) Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project. Stewart Robinson, Katy Hoad, Ruth Davies ORGS Meeting, 4th October 2007 http://www.wbs.ac.uk/go/autosimoa