1. Automating the Analysis of Simulation Output Data Katy Hoad (kathryn.hoad@wbs.ac.uk), Stewart Robinson, Ruth Davies, Mark Elder Funded by EPSRC and SIMUL8 Corporation

2. The Problem Prevalence of simulation software: ‘easy- to-develop’ models and use by non- experts. Simulation software generally have very limited facilities for directing/advising user how to run the model to get accurate estimates of performance. With a lack of the necessary skills and support, it is highly likely that simulation users are using their models poorly.

3. 3 Main Decisions: How long a warm-up is needed? How many replications should be run? How long a run length is needed?

4. Enter Analyser Warm-up analyser Warm-up period specified Trial Calculator Recommend number of replications Run-length calculator Recommend run-length Replication s or one long run? Replications One long run EXIT Analyser AUTOMATIC SIMULATION OUTPUT ANALYSER

5. Warm-up Analyser MSER-5 most promising method for automation –Performs robustly and effectively for the majority of data sets tested. –Not model or data type specific. –No estimation of parameters needed. –Can function without user intervention. –Quick to run. –Fairly simple to understand.

6. MSER-5 warm-up method 0 0.002 0.004 0.006 0.008 0.01 0.012 0.014 0.016 0.018 0.02 050100150200250300350400 Truncation Point Test Statistic 0 1 2 3 4 5 6 Batch Means MSER-5 test statistic Rejection zone Estimated warm-up period Estimated truncation point, Lsol Output data (batched means values)

7. Heuristic framework around MSER-5 Includes: Iterative procedure for procuring more data when required. ‘Failsafe’ mechanism - to deal with possibility of data not in steady state; insufficient data provided when highly auto-correlated. Graphical feedback to user.

8. Trial calculator

9. Precision≤ 5% Precision> 5% Precision ≤ 5% f(kLimit) Nsol 2 Nsol 2 + f(kLimit ) Nsol 1 95% confidence limits Cumulative mean, Confidence Interval Method with ‘look-ahead’

10. Run length calculator Batch Means Method. Want a robust automatable method that estimates the minimum run length needed to achieve a required precision in the output point estimator. Currently investigating literature and testing methods.

11. Enter Analyser Warm-up analyser Warm-up period specified Trial Calculator Recommend number of replications Run-length calculator Recommend run-length Replication s or one long run? Replications One long run EXIT Analyser AUTOMATIC SIMULATION OUTPUT ANALYSER IMPLEMENTED IN SIMUL8 BEING IMPLEMENTED IN SIMUL8

12. ACKNOWLEDGMENTS This work is part of the Automating Simulation Output Analysis (AutoSimOA) project (http://www.wbs.ac.uk/go/autosimoa) that is funded by the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project. Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School SW08