SIMULATION MODELING AND ANALYSIS WITH ARENA T. Altiok and B. Melamed Chapter 9 Output Analysis Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Output Analysis Activities Output Analysis is the modeling stage concerned with simulation replications computing statistics from replications presenting statistics in textual or graphical format Output Analysis activities consist of the following activities: replication design estimation of performance metrics system analysis and experimentation Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Types of Simulation Models Simulation models can be classified into two main classes, based on their time horizon: terminating models steady-state models Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Terminating Simulation Models A terminating simulation model has a natural termination time for its replications the time horizon is inherent in the system and finite the modeler is interested in short-term system dynamics statistics are computed within the system’s natural time horizon The number of replications is a critical parameter of the associated Output Analysis it is the only means of controlling the sample size of any given estimator . Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Steady-State Simulation Models A steady state simulation model has no natural termination time for its replications the time horizon could be potentially infinite the modeler is interested in long-term dynamics and statistics While long-term statistics are of interest, initial system conditions tend to bias its long-term statistics it, therefore, makes sense to start statistics collection after an initial period of system warm-up, namely, after the biasing effect of the initial conditions decays to insignificance the transient-state regime is characterized by statistics that vary as function of time, while the steady-state regime prevails when statistics stabilize and do not vary over time Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Sample Steady-State Simulation Output Workstation throughput and average number of jobs as functions of time Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Steady-State Simulation Issues Since steady-state models have no natural termination time, how does one select a replication length? the replication can be stopped when statistics at the end of several successive increments are sufficiently close (e.g., within some difference, to be determined by the analyst) Since a warm-up period is needed to eliminate statistical bias, how does one select the warm-up length? in a similar vein, the length of a warm-up period is determined by observing experimentally when the time variability of the statistics of interest largely disappears Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Statistics Collection From Replications Suppose we are interested in a parameter of the system e.g., mean flow time or blocking probability The simulation will then be programmed to produce a (variate) estimator, , which evaluates to some estimate For example, let denote the random variable of flow time through the system of job in replication , and let be a realization of if the estimator of the mean flow time is the sample mean of flow times in each replication, then the corresponding replication estimates are where is the number of flow time observations recorded in replication Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Batch Means Statistics Collection The Batch Means method is a practical way of collecting multiple estimates from a single replication the idea is to group observations into batches, which are iid or approximately so, and then collect one estimate from each batch Let the observed history be a discrete sample the total of observations is divided into batches of size each, such that (the batch size is selected to be large enough so as to ensure that the corresponding estimators are iid or approximately so) this results in the following partition into sub-samples (batches) for each batch , an estimate is formed from the observations of that batch only the replication thus yields a set of estimates Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Point Estimation for Discrete Samples Let a replication collect observations from a discrete sample, where is a sequence of variates, is the corresponding sample of observations The estimator for the mean value is the sample mean The sample mean above is classified as a point estimator, because it estimates a scalar In a similar vein, the corresponding point estimate is obtained when the sample of observations (realizations) are substituted into the formula for the sample mean above Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Point Estimation for Continuous Samples Let a replication collect observations from a continuous sample over time, where is a stochastic process over some time interval is the corresponding sample of observations The estimator for the mean value is the time average point estimate so called because the variates involved are indexed by time Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Point Estimation for Continuous Samples (Cont.) Examples of time-continuous variates in queueing context the number of jobs, , at time in a queueing system (buffer and server) the server utilization process, , is defined as the indicator variate where when the server is busy, and when the server is idle the utilization statistic is the time average which is the fraction of time in that the server is busy, and as such is an estimator of the probability that the server is busy Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Point Estimation in Arena The Statistic module allows the user to obtain estimates for Tally statistics Time Persistent statistics Both Tally and Time Persistent statistics permit user access to a number of related Arena variables, such as TAVG(X) = tally average of variable X DAVG(X) = time average of variable X Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Point Estimation in Arena Consider a workstation, subject to failure, where jobs arrive with exponential inter-arrival times of mean 1 hour jobs have a fixed processing time of 0.75 hours the workstation goes through up/down cycles as follows: while busy, it fails randomly with exponentially distributed time-to-failure of mean 20 hours on failure, repair times are uniformly distributed between 1 and 5 hours Based on analytical calculations, we expect the throughput to be 1 job per hour (same as the arrival rate) the down time probability to be 0.1125 the average job delay (queue delay) to be 4.11 hours per job [see Altiok (1997), Chapter 3] Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Point Estimation in Arena (Cont.) The results of 5 replications from a simulation of a workstation subject to failures are displayed in the table below Note that the estimates vary from replication to replication the underlying estimator is a random variable! Any of the values can be used to estimate the true (unknown) parameter but how confident can the modeler be in their accuracy? Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Confidence Interval Estimation Confidence Interval (CI) estimation quantifies the confidence (probability) that an interval “covers” the true (but unknown) statistic the boundaries of the confidence interval are estimated using appropriate point estimates therefore, those boundaries are random variables, and the CI is a random interval which varies across experiments (replications)! the modeler predetermines the desired probability that the CI “covers” the true statistic (the larger the probability, the wider the interval) Confidence intervals can be computed for terminating simulations steady-state simulations Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Confidence Intervals for Terminating Simulations Assume that to estimate a parameter by simulation, independent fixed-length replications of the model were run the runs produced a sample , where is the point estimate of in replication The pooled point estimator for is the sample mean across replications this estimator is a random variable with mean and variance thus, increasing the number of replications, , would decrease the variance of , and consequently, increase our confidence in the corresponding point estimate value, Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Confidence Intervals for Terminating Simulations (Cont.) We wish to quantify the confidence in an estimate of the true parameter by computing (at least approximately) the probability of events of the form where the estimators and define a (random) confidence interval for the true parameter, is the probability that the confidence interval does not include the true parameter, is a small probability, (often 0.01 or 0.05), called the significance level Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Confidence Intervals for Terminating Simulations (Cont.) By the Central Limit Theorem, the random variable in the estimator is approximately normally distributed with mean and variance It follows that From properties of the normal distribution, the approximation improves as the sample size, , increases we have (or at least approximately so, if the bias is tolerable) Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Confidence Intervals for Terminating Simulations (Cont.) It is known that has the standard normal distribution with quantiles , where Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Confidence Intervals for Terminating Simulations (Cont.) A confidence interval for the mean parameter, , at significance level is given by In case the variance is unknown, proceed as follows: estimate the variance by use the fact that the Student t distribution satisfies the requisite confidence interval is Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Confidence Intervals for Steady-State Simulations Consider confidence interval estimation for some mean in Batch Means setting with a single replication of batches (the underlying history may be discrete or continuous) The corresponding set of estimators, , gives rise to sample mean sample variance The confidence interval for at significance level is Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Confidence Interval Estimation in Arena Standard Arena output provides 95% Batch Means confidence intervals for each replication computed for both Tally and Time Persistent statistics in terms of half widths under the Half Width column heading if, however, the estimated batch means are significantly dependent or the underlying sample history is too short to yield a sufficient number of batches, then that column will display the message (Correlated) Arena also supports the computation of confidence intervals from multiple replications in the Outputs element of the SIMAN summary report Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Output Analysis via Standard Arena Output As a working example, consider a single-machine finishing operation workstation that processes two types of parts, denoted by G_1 and G_2, where parts of type G_1 arrive according to an exponential inter-arrival time distribution with mean 2 hours, and each part has a fixed processing time of 1 hour parts of type G_2 arrive according to an exponential inter-arrival time distribution with mean 4 hours, and each part has a fixed processing time of 1.4 hours all parts are processed in FIFO order all part types have equal service priorities We wish to simulate the finishing operation for 10,000 hours in order to understand the behavior of the number of parts in the workstation buffer and the buffer delay for each part type Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Output Analysis via Standard Arena Output (Cont.) Arena model of a finishing operation with two types of parts Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Observation Collection Record module with Tally statistics for part delay in the buffer Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Observation Collection (Cont.) Dialog boxes for the Set module to tally delay times for each part type (bottom) and its members (top) Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Observation Collection (Cont.) Dialog box for the Statistic module with a Tally statistic for each part delay Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Output Summary Output summary report for Resources Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Output Summary (Cont.) Output summary report for Queues Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Output Analysis From the summary report we observe the following partial utilizations in the finishing machine: is utilization due to type G_1 parts workload is utilization due to type G_2 parts workload is utilization due to overall workload We, therefore, expect the probability of the finishing machine being in the Busy state to be around 0.85 The estimated probability in the summary report for Resources is actually 0.84 over a replication of length 10,000 hours. Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Output Analysis (Cont.) The table below shows the behavior of three performance measures as functions of replication length Observe how the estimates appear to converge to respective limiting values as the simulation length increases convergence is indicated by the fact that the values appear to stabilize and change very little for the higher range of replication lengths Since we know the true value of machine utilization, we can take advantage of this knowledge in deciding on the smallest replication length that gives rise to sufficiently accurate estimates high accuracy is indicated in the table by low variability in the estimates as function of replication length we naturally seek the smallest value, since we would like to reduce the computational effort as much as possible . Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Output Analysis (Cont.) Summary statistics based on 10 replications Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Output Analysis via the Arena Output Analyzer The Arena Output Analyzer tool provides the following capabilities: Data transfer data import from ASCII files data export to ASCII files Statistical analysis batching, correlogram, and basic statistics point estimation and confidence interval estimation for means and standard deviations statistical tests for comparing parameters of different samples Graphing data plots and charts statistics plots and charts Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Data Collection Tallied buffer delay observations for parts of type G_1 Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Graphical Statistics Graphical statistics for parts of type G_1 Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Batching Data for Independent Observations Batch/Truncate dialog box Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Batching Data for Independent Observations (Cont.) Batched observations stored in file : G_1_Delay_Batched.flt Initial Observations Truncated : 0 Number of Batches : 499 Number of Observations Per Batch : 1000 Number of Trailing Obs'ns Truncated : 486 Estimate of Covariance Between Batches : -0.002459 Batch/Truncate summary report for delay times of G_1 type parts Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Confidence Intervals for Means and Variances Confidence intervals for mean delays of parts of type G_1 and G_2 Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Comparing Means and Variances The Analyze menu in the Arena Output Analyzer tool provides the following options: Compare Means for comparing the means of two samples drawn from two populations Compare Variances for comparing the variances of two samples drawn from two populations These options perform statistical analysis for testing statistically for the equality of means and variances, respectively For example, to test the null hypothesis that the means are equal, a confidence interval is sets up for their difference since under the null hypothesis the difference is zero, one accepts equality if the confidence interval includes 0, and rejects it, otherwise Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Comparing Means Dialog boxes for comparing mean buffer delays of type G_1 and G_2 parts Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Testing for Equality of Means Test results for the equality of mean buffer delays of type G_1 and G_2 parts Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Point Estimation for Correlation The Analyze menu in the Arena Output Analyzer tool provides the Correlogram option to gauge the statistical dependence among observations within a sample performs point estimation of the correlation between two samples by estimating the coefficient of correlation between the respective random variables This activity is called correlation analysis Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Example: Correlation Analysis The left-hand correlogram was computed from simulation observations of successive delays The right-hand correlogram the sample means of batches of 1000 delays The two correlograms are strikingly different: the successive delays are strongly positively correlated, while the sample means of the batches are very nearly uncorrelated Results of correlation analysis for type G_1 parts Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

Parametric Analysis via the Arena Process Analyzer The term Parametric Analysis refers to the following set of activities: running a model multiple times with a different set of input parameters for each run then comparing the resultant performance measures Its purpose is to carry out to Sensitivity Analysis understand the impact of parameter changes on system behavior often use this understanding to find the optimal configuration (parameter set) with respect to one or more performance measures, or combination thereof In Arena Process Analyzer parlance input parameters are called controls the resultant performance measures are called responses a collection of controls and responses for a given set of runs is referred to as a scenario a collection of scenarios is termed an Arena project Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

The Process Analyzer GUI A Process Analyzer window with an initial grid Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9

The Process Analyzer GUI (Cont.) A Process Analyzer window with a populated grid Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 9