Classification of Discrete Event Simulation Models and Output Data: Creating a Sufficient Model Set. Katy Hoad Stewart Robinson, Ruth Davies, Mark Elder Funded by EPSRC and SIMUL8 Corporation

AIM: Provide a representative and sufficient set of models / data output for use in discrete event simulation research.

MODEL CLASSIFICATION Creating A Standard Set of Models/Outputs Outline:  Motivation  Identification of model/output characteristics  Creation of a classification system

Motivation Simulation model Warm-up analysis Run-length analysis Replications analysis Use replications or long-run? Recommendation possible? Recommend- ation Output data Analyser Obtain more output data Want to create an automated Analyser to advise user on: Warm-up length Run-length Number of replications

Motivation Needed to test output analysis methods to find the most effective methods and… …test created algorithms for effectiveness and robustness. Required a set of models / output data that sufficiently covered the different types of possible models/output. Could not find a general set in the public domain.

How do you define a sufficient and representative set of models/output? AIM To define a set of characteristics that classify/describe a model and its output.  Searched the literature.  Collected and studied ‘real’ models/output. Identification of model/output characteristics

Group A …Group N Group B Auto Correlation Normality Cycling/Seasonality Terminating Non-terminating Steady state In/out of control Transient

9 other characteristics of models / output were chosen to categorize the models / output within these two main groups. 2 main categories or groups:  Transient (including out-of-control trend)  Steady-state (including steady-state cycle)

Model characteristics  Deterministic or Stochastic (random)  Significant pre- determined model changes (by time)  Dynamic internal changes i.e. ‘feed- back’  Empty-to-empty pattern  Initial transient (warm-up)  Out of control trend ρ≥1  Cycle  Auto-correlation  Statistical distribution Output data characteristics

Looked at over 50 real models - defined as discrete event simulation models of real existing / future systems: ModelOutput / response Call Centre% of calls answered within 30 seconds Production Line in Manufacturing Plant Through-put Fast Food StoreAverage queuing time HospitalAverage number in system For example:  Justification of selection of model output: Picked most likely output result for each model, using already programmed results collection when feasible.

Further Analysis Each real model was statistically analysed as follows: 1.Steady State: Subtract mean from output data. Test residuals for Auto-correlation and Normality. 2. Steady State Cycle:Run model for many cycles. Take mean of each cycle to create a new time series. Subtract mean from this new output data. Test residuals for Auto-correlation and Normality. 3. Transient:Test for Auto-correlation on output data. Run many replications (1000) Take mean of each replication to create new (non auto-correlated) data set. Test for what type of statistical distribution best fits this new data set. 4.Out-Of-Control:Plot data

Analysis Results Steady State data: Autocorrelation: AR(1), AR(2), some AR(3+), some ARMA(n,n) & some with no auto-correlation. Distributions: Normal and non-normal. Transient data: AR(1), AR(2), some AR(3+), some ARMA(n,n) & some with no autocorrelation. Distributions found to be a ‘good’ fit to the various transient data output: Normal, Beta, Pearson5, LogNormal, Weibull, Gamma, Pearson6, Erlang, Chi squared, Bi-modal distribution

Classification Tables  MODEL SUMMARY_Steady State.xlsMODEL SUMMARY_Steady State.xls  MODEL SUMMARY_Transient.xlsMODEL SUMMARY_Transient.xls  AIM: Collect ‘real’ models to cover range of classifications of models. (On-going process) Create artificial models to cover range of classifications of output data.

Cash et al 1992: AR(1); M/M/1; Markov Chain. Robinson 2007: AR(1); M/M/1. Goldsman et al. 1994: AR(1); M/M/1. White, Cobb & Spratt 2000: AR(2). Ockerman & Goldsman: Random Walk; AR(1); MA(1) Kelton & Law 1983: M/M/1 (FIFO); M/M/1 (LIFO); M/M/1(SIRO); M/M/1 (initialized with 10 customers); E4/M/1; M/H2/1; M/M/2; M/M/4; M/M/1/M/1/M/1. Hsieh et al 2004: M/M/1/199; M/G/1/199; M/M/1/19; Number-in-stock process single item inventory management system. steady state outputs with or without a warm-up period. Sample of Artificial Models from literature:

3 main methods for creating artificial models / output data sets: 1.Create simple simulation models where theoretical value of some output / response is known.  E.g. Model: M/M/1. Output: mean waiting time. 2.Create simple simulation models where the value of some output / response is estimated but model characteristics can be controlled.  E.g. Model: Single item inventory management system. Output: Number-in-stock. 3. Create data sets from known equations, which closely resemble real model output, with known value for some specific output / response.  E.g. AR(1) with Normal(0,1) errors. Output: mean

Our Project: Replications and Warm-up Method Testing Replication Method Testing –Data sets of replicated mean values from transient output – left and right skewed, Normal and Bi-modal. –Real models Warm-up Method Testing –Steady state functions: AR(1), AR(2), AR(4), MA(2), ARMA(5,5), no auto-correlation. –Initialisation Bias functions: Severity, Length, Shape. –Real models

SUMMARY Produced a classification of model and output data types for the purpose of aiding research into simulation output analysis. Currently using artificial models that broadly cover each output type in the classification tables in our research into output analysis methods.

DISCUSSION: YOUR COMMENTS APPRECIATED  Using our chosen classification criteria, we have classified a complete set of possible models / output: But are these criteria sufficient?  Main model/output types missing from our collection: Transient with warm-up. Deterministic transient. Cycle with warm-up Are these missing model criteria feasible?

ACKNOWLEDGMENTS This work is part of the Automating Simulation Output Analysis (AutoSimOA) project that is funded by the UK (EPSRC) Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project. Stewart Robinson, Katy Hoad, Ruth Davies INFORMS November