Discrete Event Simulation - 8 Design of Simulation Experiments 12/1/2018

Discrete Event Simulation - 8 How do we design a simulation experiment? This is not very different from designing any other kind of experiment: 1) determine what variables you want to examine, 2) determine what tests are most suitable for the desired variables, 3) set up the experiment: create a controlled environment that will allow for replication, 4) run the experiment, 5) collect the data, 6) analyze the data. 12/1/2018

Discrete Event Simulation - 8 You generally wish to determine some values and some "confidence intervals" for those values. For example, you may want to estimate the size of some queues under certain operating conditions: two quantities of interest are the queue means and the queue variations. The simulation experiment may introduce some experimental error itself: numerical operations tend to propagate numerical errors; subtle timing errors may be incorporated into the design, etc. 12/1/2018

Discrete Event Simulation - 8 We are going to look at several types of experiments, increasing in complexity. Terminology is - sometimes - used from some old applications. Assume we have a number of observations Yij: Where m is the common mean and eij is an error term from a population with mean 0 and variance s2. Tj is usually interpreted as the effect of a treatment - the j-th treatment (in our context it could be the j-th pair of (voice, silence) values tried in our simulation of voice traffic). One often assumes that and the null hypothesis usually tested is H0: Tj = 0, for all j - the treatments make no difference. 12/1/2018

Discrete Event Simulation - 8 Recall that we only have the observed Yij. The existence of the quantities m, Tj and eij is simply assumed, to account for the of the effects of the changes on the observables. The further hypothesis - implicit in the formula - that the relationships among the variables is linear, is also just a “working hypothesis”. It may be true, it may be almost true, or it may be quite wrong. Any further refinement would require a better understanding of the theoretical model and, probably, trial and error with several data sets. 12/1/2018

Discrete Event Simulation - 8 How can we test it? What kind of test can we employ? One thing we may observe is that we can try to separate the various (hypothesized) contributions and that most of the tests we are familiar with require variances or c2 variables (which are, most easily, sums of variances). What we need to do is extract the contribution of the Tj terms and compare it to the contribution of the error terms. If both of them can be written in the form of c2 variables with appropriate degrees of freedom, we can use them in the quotient of the F-test. 12/1/2018

Discrete Event Simulation - 8 We start - sum of squares: where The "cross product" terms vanish: we show it in one case: 12/1/2018

Discrete Event Simulation - 8 At this point we need two older theorems for guidance: Central Limit Theorem: if x is normally distributed with mean m and standard deviation s, and a random sample of size n is drawn, then the sample mean will be normally distributed with mean m and standard deviation s/√n. c2 Theorem: if x is normally distributed with variance s2 and s2 is the sample variance based on a random sample of size n, then ns2/s2 has a c2 distribution with n-1 degrees of freedom. 12/1/2018

Discrete Event Simulation - 8 The two useful terms are We start with the second one: Recall that By hypothesis, the variance depends strictly on the eij terms, which have mean 0 and variance s2. Then the Null Hypothesis (Tj = 0) implies that all these variables have the same mean (m) and (by the Central Limit Theorem) variances given by s2/nj. If all the nj are the same, then elementary standard theory applies (if they are different - text claims also … I have no reference) and we can claim: 12/1/2018

Discrete Event Simulation - 8 The expression possesses a c2 distribution with k - 1 degrees of freedom. One now needs to examine the first sum: We will not go into the details of the derivation (not even the plausibility argument given above) and conclude that possesses a c2 distribution with k(n - 1) = kn - k degrees of freedom. 12/1/2018

Discrete Event Simulation - 8 How should we interpret ? We can expand We observe that this term contains only error contributions, and is completely independent of the Tj terms. The expression is the desired F variable: contribution from the Tj terms divided by the contribution of the errors alone. 12/1/2018

Discrete Event Simulation - 8 An example: how do we use this??? 10.1-10.2 Suppose we simulate a single-channel queuing system in which arrivals occur according to a Poisson process with rate l and service times are exponentially distributed with service rate m. Suppose further that we want to test the effect of the queuing disciplines FIFO, SIRO(Serially_In_Randomly_Out??? I have NO idea), and LIFO on the number of customers in the system at the end of one hour. The response variable (i.e. the Tj) is the number of customers in the system at the end of one hour; the single factor influencing the variable is the queuing discipline. 12/1/2018

Discrete Event Simulation - 8 Setting up the experiment: choose one discipline at random, run for one hour, count. Choose second discipline at random, run, count. Choose third discipline, run count. You may want to repeat this process several times with different runs of (= seeds for) the random number generator. Let's say you have executed four runs for each of the disciplines and you have: j i FIFO SIRO LIFO 1 Y11= 50 Y12= 32 Y13= 80 2 48 18 73 3 61 64 54 4 58 72 63 12/1/2018

Discrete Event Simulation - 8 We now compute the various sums: 12/1/2018

Discrete Event Simulation - 8 The number of degrees of freedom for the numerator is 3-1=2, and for the denominator 3*4-3=9. From the tables F0.05,(2,9)=4.26, and we cannot reject the null hypothesis: we cannot show (at least at the 5% level) that the queuing discipline has any effect. 12/1/2018

Discrete Event Simulation - 8 A more complex simulation experiment would add the possibility of using different service rates at different times - say four different ones - each service rate remaining constant at least long enough for the one-hour experiment to be conducted. We now have the possibility of determining some of the interactions between service rates and queuing discipline, but let's assume for the moment that we are only interested in the effects of queuing discipline. We have each run: Where the Bj terms correspond to the "blocks" of different service rates. You need to run at least one experiment with each rate and with each discipline. 12/1/2018

Discrete Event Simulation - 8 How do we set up the appropriate c2 variables and the F variable? Since the cross product terms will cancel out. The terms of interest are: 12/1/2018

Discrete Event Simulation - 8 For the error term. For the discipline term. For the block (rate) term. Each value of i now corresponds to a different service rate. Proof that we will end up with the appropriate c2 variables is omitted - although it is possible. 12/1/2018

Discrete Event Simulation - 8 We now attempt to reconcile the formulae in the text with the formulae obtained here. The remainder of the formulae now follow in the same way. One more unexplained term is given by the formula for the number of degrees of freedom for SSERR, which differs from the one in the previous experiment. 12/1/2018

Discrete Event Simulation - 8 The general derivation (without proof) goes as follows: The left hand side of the equality has kn degrees of freedom; the terms on the right hand side have, respectively: 12/1/2018

Discrete Event Simulation - 8 The example in the text considers each of the four distinct rows as corresponding to different rates. If one wants to take into account the interactions between the disciplines and the rate changes, one must introduce appropriate "interaction terms". For example Where one carries out k = 1…m observations for each of the choices of Ai and Bj factors (or treatments), i = 1…a and j = 1…b. By analogy with the previous treatment, the contribution of the Ai terms will come through that of the Bj ones via , and that of the ABij by means of . 12/1/2018

Discrete Event Simulation - 8 Before continuing, let’s examine the idea of interactions a little more closely. Assume that we change TWO variables, but make only ONE observation for each parameter configuration. Then we would have where the ?? term must contain both interaction (ABij) and error (eij)effects. The expansion (from a previous slide) leads us to: 12/1/2018

Discrete Event Simulation - 8 Where no interaction effects - other than “error terms” - can be extracted. In order for us to have explicit access to interaction effects with just TWO parameters being changed we need to perform at least two observations for each parameter setting. This will lead us back to the discussion of Yijk - as a special case of the situation where the observable depends on THREE parameters - one of which affects only the error terms eijk. 12/1/2018

Discrete Event Simulation - 8 In general, if we have three variables, we have the expansion: In the textbook example we have two variables with a and b levels, respectively, and m observations for each variable configuration. Thus the identity above can be rewritten without the terms that correspond to segregation of the third variable: 12/1/2018

Discrete Event Simulation - 8 Which can be verified to match the figure on p.229 of our text. The computation of the degrees of freedom follows as before: a*b*m on the left, with (a - 1) + (b - 1) + (a - 1)*(b - 1) + a*b*(m - 1) + 1 on the right. The values can be obtained by the observation that an index which has not been averaged in the formula counts for the full amount, while one averaged counts for one less (there is a relation already in the formula). 12/1/2018