1. Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded project in collaboration with SIMUL8 Corporation. http://www.wbs.ac.uk/go/autosimoa

2. Research Aim To create an automated system for dealing with the problem of initial bias, for implementation into simulation software. Target audience: non- (statistically) expert simulation users.

3. The Initial Bias Problem Model may not start in a “typical” state. Can cause initial bias in the output. Method used: Deletion of the initial transient data by specifying a warm- up period (or truncation point). How do you estimate the length of the warm-up period required?

4. Literature search – 44 methods Short-listing of methods Accuracy & robustness Ease of automation Generality Computer running time Preliminary Testing – 6 methods MSER-5 most accurate and robust method.

5. MSER-5 warm-up method 0 0.002 0.004 0.006 0.008 0.01 0.012 0.014 0.016 0.018 0.02 050100150200250300350400 Truncation Point Test Statistic 0 1 2 3 4 5 6 Batch Means MSER-5 test statistic Rejection zone Estimated warm-up period Estimated truncation point, Lsol Output data (batched means values)

6. Further Testing of MSER-5 1.Artificial data – controllable & comparable  initial bias functions  steady state functions 2.Full factorial design. 3.Set of performance criteria.

7. ParametersLevels Data TypeSingle run Data averaged over 5 reps Error typeN(1,1), Exp(1) Auto- correlation None, AR(1), AR(2), MA(2), AR(4), ARMA(5,5) Bias Severity1, 2, 4 Bias Length0%, 10%, 40%, 100% (of n = 1000) Bias directionPositive, Negative Bias shape7 shapes 1. Artificial Data Parameters

8. Mean Shift: Linear: Quadratic: Exponential: Oscillating (decreasing):

9. Add Initial Bias to Steady state: Superpostion: Bias Fn, a(t), added onto end of steady state function: e.g. 2. Full factorial design 3048 types of artificial data set MSER-5 run with each type 100 times

10. i.Coverage of true mean. ii.Closeness of estimated truncation point (Lsol) to true truncation point (L). iii.Percentage bias removed by truncation. iv.Analysis of the pattern & frequency of rejections of Lsol (i.e. Lsol > n/2). 3. Performance Criteria

11. MSER-5 Results Does the true mean fall into the 95% CI for the estimated mean? Non-truncated data sets Truncated data sets % of cases yes 7.7% noyes72.5% no 19.8% yesno0% i. Coverage of true mean.

12. ii. Closeness of Lsol to L. Wide range of Lsol values. e.g. (Positive bias functions, single run data, N(1,1) errors, MA(2) auto-correlation, bias severity value of 2 and true L = 100.)

13. iii. Percentage bias removed by truncation.

14. Effect of data parameters on bias removal No significant effect: Error type Bias direction Significant effect:Data type Auto-correlation type Bias shape Bias severity Bias length

15. More bias removed by using averaged replications rather than a single run.

16. The stronger the auto-correlation, the less accurate the bias removal. Effect greatly reduced by using averaged data.

17. The more sharply the initial bias declines, the more likely MSER-5 is to underestimate the warm-up period and to remove increasingly less bias.

18. As the bias severity increases, MSER-5 removes an increasingly higher percentage of the bias.

19. Longer bias removed slightly more efficiently than shorter bias. Shorter bias - more overestimations - partly due to longer bias overestimations being more likely to be rejected.

20. Rejections caused by: high auto-correlation, bias close to n/2, smooth end to data = ‘end point’ rejection. Averaged data slightly increases probability of getting ‘end point’ rejection but increases probability of more accurate L estimates. iv. Lsol rejections

21. Giving more data to MSER-5 in an iterative fashion produces a valid Lsol value where previously the Lsol value had been rejected. e.g. ARMA(5,5)

22. Lsol valuesPercentage of cases Lsol = 071% Lsol ≤ 5093% Testing MSER-5 with data that has no initial bias. Want Lsol = 0 Lsol > 50 mainly due to highest auto-correlated data sets - AR(1) & ARMA(5,5). Rejected Lsol values: 5.6% of the 2400 Lsol values produced. 93% from the highest auto-correlated data ARMA(5,5).

23. Testing MSER-5 with data that has 100% bias. Want 100% rejection rate: Actual rate = 61%

24. Summary MSER-5 most promising method for automation –Not model or data type specific. –No estimation of parameters needed. –Can function without user intervention. –Shown to perform robustly and effectively for the majority of data sets tested. –Quick to run. –Fairly simple to understand.

25. Heuristic framework around MSER-5 Iterative procedure for procuring more data when required. ‘Failsafe’ mechanism - to deal with possibility of data not in steady state; insufficient data provided when highly auto- correlated. Being implemented in SIMUL8.

26. ACKNOWLEDGMENTS This work is part of the Automating Simulation Output Analysis (AutoSimOA) project (http://www.wbs.ac.uk/go/autosimoa) that is funded by the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project. Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08