1. Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School Simulation Workshop - April 2008 The AutoSimOA Project A 3 year, EPSRC funded project in collaboration with SIMUL8 Corporation. http://www.wbs.ac.uk/go/autosimoa

2. Research Aim To create an automated system for dealing with the problem of initial bias, for implementation into simulation software. Target audience: non- (statistically) expert simulation users.

3. The Initial Bias Problem Model may not start in a “typical” state. Can cause initial bias in the output. Many methods proposed for dealing with initial bias: e.g. Initial steady state conditions; run model for ‘long’ time… This project uses: Deletion of the initial transient data by specifying a warm-up period (or truncation point).

4. Question is: How do you estimate the length of the warm-up period required?

5. Methods fall into 5 main types : 1. Graphical Methods. 2. Heuristic Approaches. 3. Statistical Methods. 4. Initialisation Bias Tests. 5. Hybrid Methods.

6. Literature search – 42 methods Summary of methods and literature references on project web site: http://www.wbs.ac.uk/go/autosimoa

7. Short-listing warm-up methods for automation using literature Short-listing Criteria Accuracy & robustness Simplicity Ease of automation Generality Number of parameters to estimate Computer running time

8. Short-listing results: reasons for rejection of methods

9. Statistical methods: –Goodness of Fit (GoF) test –Algorithm for a static data set (ASD) –Algorithm for a Dynamic data set (ADD) Heuristics: –MSER-5 –Kimbler’s Double Exponential Smoothing –Euclidean Distance Method (ED) Short-listing results: 6 Methods taken forward to testing

10. Testing Procedure Test short-listed methods using: 1.Artificial data – controllable & comparable  initial bias functions  steady state functions 2.Set of performance criteria.

11. i)Length – proportion of data length. ii)Severity – maximum bias value is a function of the difference between steady state mean and 1st (if bias fn +ve) or 99th (if bias fn –ve) percentile of the steady state data. iii)Shape and Orientation – 7 shapes: Initial bias functions - 3 Criteria: 1. Artificial Data Sets

12. Mean Shift: Linear: Quadratic: Exponential: Oscillating (decreasing):

13. i) Constant steady state variance ii) Error Terms: Normal or Exponential distribution iii) Auto-Correlation: No AutoCorrelation; AR(1); AR(2); AR(4); MA(2); ARMA(5,5). Add Initial Bias to Steady state: Superpostion: Bias Fn, a(t), added onto end of steady state function: e.g. Steady state functions - 3 Criteria:

14. Closeness of estimated truncation point (Lsol) to true truncation point (L). Coverage of true mean ½ width of 95% CI for average truncated mean. Bias and absolute bias in estimated mean. Number of failures of method. 2. Performance Criteria

15. Test Results Rejections: –ASD & ADD required a prohibitively large number of replications –GoF & Kimbler’s method consistently severely underestimated truncation point. –ED failed to give any result on majority of occasions MSER-5 most accurate and robust method.

16. MSER-5 Method MSER-5 test statistic Output data (batched)

17. MSER-5 Results MSER5 result eg.xls Does the true mean fall into the 95% CI for the estimated mean? Non-truncated data sets Truncated data sets % of cases yes 8.3% yesno0% no 19.3% noyes72.4%

18. Summary / Future Work 42 warm-up methods Short-listing and testing MSER-5 most promising method for automation Creation of heuristic framework around MSER-5 method for implementation into simulation software.

19. ACKNOWLEDGMENTS This work is part of the Automating Simulation Output Analysis (AutoSimOA) project (http://www.wbs.ac.uk/go/autosimoa) that is funded by the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project. Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School SW08

20. ii) SEVERITY OF BIAS FUNCTION Set maximum value of bias fn, a(t), so that max |a(t)| t≤L = M×Q Q = difference between steady state mean and 1st (if bias fn +ve) or 99th (if bias fn –ve) percentile of the steady state data. M = relative maximum bias – user set: 1, 2, 5 M ≥ 1 → bias significantly separate from steady state data → easier to detect. M ≤ 1 → bias absorbed into steady state data variance → harder to detect.