Experiment Design and Data Analysis When dealing with measurement or simulation, a careful experiment design and data analysis are essential for reducing costs and drawing meaningful conclusions. The two issues are coupled, since it is usually not possible to select all parameters of an experiment without doing a preliminary run and analyzing the data obtained. 3.1 Simulation Techniques The kind of simulation of interest to us is discrete event simulation, where the state of the system is updated every time an event occurs. The most common phenomenon that we shall be dealing with is the queuing of customers at a station either to receive service or to acquire some resource (buffers, channels, etc.).

-In such cases, the relevant events are the arrival of a customer at a station, service completion at a station, resource acquisition resource release, etc. -Event generation can be either trace driven or distribution driven. -In trace driven, the time of occurrence of the event (and the magnitude of auxiliary variables associated with the event, if any) are obtained from measurements on a real system and then used directly in the simulation. -In the distribution driven. They are generated to follow a given (continuous or discrete) distribution. -Distribution-driven simulations require the generation of random numbers that follow a given distribution. -This is done in two steps. -First, we generate “unif (0.1)” random numbers, i.e. random numbers that are uniformly distributed in the range 0 through 1.

-Next, we transform these into random numbers having the desired distribution. 3.2 Fundamentals of Data Analysis -The most fundamental aspect of the systems of interest to us is that they are driven by a nondeterministic workload. -This randomness in the inputs makes the outputs also random. -Thus, no single observation from the system would give us a reliable indication of the performance of the system. -One way to cope with this randomness is to use several observations in estimating how the system will behave “on the average”. -This immediately raises several questions: -1 How do we use several observations to estimate the average performance, i.e., what is a good estimator based on several observations?

2- Is an estimate based on several observations necessarily more reliable than the one based on a single observation? 3- How do we characterize the error in our estimate as a function of the number of observations? Or, put another way, given the tolerable error. How do we determine the number observations? 4- How do we perform experiments so that the error characterization is itself reliable? 5- If the number of needed observations is found to be too large, what can we do to reduce it? -Answers to these questions lie at the very core of data analysis and experiment design. -Let X denote a performance measure of interest (e.g.,the response time). -We can regard X as a random variable some unknown distribution.

-Let s and σ 2 denote its mean and variance respectively. Suppose that we obtain the observation X 1, X 2 … X n. -We can associate a random variable that represents the ith observation as well, i.e., regard X i as the value of a random variable that represents the ith observation of the system. -It is a common practice to denote this random variable also by the symbol X i, since there is rarely any confusion. -We shall also adopt this practice. -Notice that by its very definition. Each of the X i ‍ ’ s should have the same distribution as X, and hence E(X i )=s and Var (X i )= σ 2. -Now, Suppose that we compute an expected value estimate of X, denoted, Using X 1,…, X n. -We could regard also as a random variable.

- We say that is an unbiased estimator if E( )=s. -It is desirable to use an estimator that remains unbiased even when the observations are not mutually independent. -One such estimator is the sample mean, Usually denoted as X, because -irrespective of whether the X i ’s are mutually independent. -In fact, the sample mean is the only nontrivial estimator with this property. -Another disirable property of an estimator is that it should be more reliable than a single observation; i.e., it should have a smaller variance.

-Let σ 2 y be the general notation the variance of a random variable Y. -Also, let Cov(X,Y) denote the covariance of X and Y, and p xy their correlation coefficient. -Since E(x)=s,We have -Now if X 1,…. X n are independent, Cov(X i,X j )=0, and σ 2 x = σ 2 /n. -Hence, the use of n mutually independent observations reduces the variance by a factor of n. -Also, since | p xy| <1,taking the sample mean can never hurt.

-In fact, if σ is finite and [pxy] < 1, we have -That is, the sample mean will converge to the expected value as n ∞This is one form of the Law of large numbers. -Note that if the observation are positively correlated(i.e.,Pxy>0), taking the mean becomes less effective. -On the other hand, if they are negatively correlated, the variance will be less than σ 2 /n. -This last observation is the basis for the variance reduction techniques discussed in section It is clear from the above discussion that the sample mean X is a desirable estimator of E[X] (or s).

- However, We also need to see how good the estimator is by determining an interval around x that contains s with a given probability. -This requires an estimate of the variance σ 2, which is also unknown. -We estimate σ 2 by the sample variance, denoted σ 2 x, and defined as - Let us examine the expected value of σ 2 x. For this, let

-Expanding the square, taking the expectation operator inside, and noting that E[(x i – s) 2 ]= σ 2 for any i, we have -It is easy to see that E [σ 2 x ]= ø/(n-1). -Thus, we see that if x i ’s are mutually independent, σ x is an unbiased estimator of σ, but not otherwise in general. -Henceforth, we shall assume that all x i ’s are independent. -Since Var(X)= σ 2 /n in this case, we can also define an unbiased estimator of var (x), denoted σ 2 / X. as simply σ 2 x /n. -The measures X and σ 2 / X give us some idea about the real value of s.

-For a more concrete characterization, we would like to obtain an interval of width e around X, such that the real value s lies somewhere in the range X + e. -Since X is a random variable, we can specify such a finite range only with some probability p 0<1. -The parameter P0 is called the confidence level, and must be chosen a priori. -Typical values are 0.90,0.95 or 0.99 depending on -how reliable we want the result to be. -Thus, our problem is to determine e such that -The parameter 2e is called the confidence interval, and is expected to increase as P0 increases. - To determine its value, we need to know the distribution of X.

-To this end, we use the central limit theorem, and conclude that if n is large, the distribution of X can be approximated as N(s, σ √ n ),i.e., normal with mean s and variance σ2/n. -Let -Then the distribution of Y must be N(0,1). -Fig. 3-1 shows the density function of Y where e// is chosen such that Pr(|Y| ≤ é) =P0 = 1-α. -Since the normal density is symmetric about the mean, α/2 mass must be contained in each tail. -That is, FY(-é)= α/2,or é=-F -1 y (α/2).

-Thus given α, é can be obtained from the tables of standard normal distribution. -Henceforth we shall denote é as Z α/2, which means that Pr ( | Y | ≤ Z α/2 ) =1- α. -Using the definition of Y form equation(2.8), We thus get an expression for the confidem interval of s, but it contaims an unknow parameter σ. -We can substitute δ x for σ, -but that will not work because the distribution of the random variable ( X-S) √ n/δ x is unKnown and may differ substantially from the normal distributin. -TO get around this difficulty, we assume that the distribution of each X i itself is normal, i.e.,N(s,σ). -Then,Y=(X-s)√ n/δ x has the standard t-distribution with (n-1) degrees of freedom.

-We denote the latter as Φ t,n-1(.). -The density function of this distribution is also symmetric about the mean ( and FIGURE 3-1: Plot of the standard normal density function. -Indeed looks much like the normal density except for a slower decay). -Therefore, by proceeding exactly as in the above, we get

-Pr(|Y| ≤ t n-1,α/2 )= 1-α where t n-1, α/2 = -Ф -1 t, n-1 (α/2 ) (2.9) -t n-1, α/2 may be found from Table E.1.Now by using equation(2.8), we get (2.10) -We can put this equation in the follwing alternate form (2.11) -This formula can be used in two ways:

-(a) to determine confidence interval for a given number of observation -(b) to determine the number of observations needed to achieve a given confidence interval. -For the latter, suppose that the desired error(i.e., fractional half- width of the confidence interval)is q. Then -Since δ x, x, and t n-1,α/2 depend on n. we should first“guess” some value for n and determine δ x, x, and t n-1- α/2.

-Then we can check if equation (2.12) is satisfied. If it is not. More observations should be made. -In the above we assumed a two –sided confidence interval. -In some applications. We only want to find out whether the performance measure of interest exceeds(or remains below) some given threshold. -The only difference in the analysis here is that we consider the tail of t-distribution only on one side. -For example, to assert that the actual value s exceeds some threshold X-e, let Y=(X-s) √n/бx. Then

-which gives the following one-sided equivalent of equation(2.9) -Note that the left- hand side of this equation contains α(instead of α/2)because the entire mass is now on the Lower end. - Substituting for Y, We get (2.14) -We could follow the same procedure to asseert that s is less than some threshold; however, because of the symmetry of the t distribution, the result will be identical.

-That is,(2.14)will continue to hold with the inequality reversed. We shall illustrate one- sided confidence intervals by means of an example. -Example 3,1 Five independent experiments were conducted for determining the average flow rate of the coolant discharged by the cooling system. -One hundred observation were taken in each experiment,the means of which are reported below: Based on this data, could we say that the mean flow rate exceeds 3.00 at a confidence level of 99.5%? -What happens if we degarde the confidence level down to 97.5%? -Solution the sample mean and sample standard deviation can be calculated from the data as;X=3.126, δ X =

-From Table E.1, We get T4,0.005 =4.604,and the Lower bound becomes X4.604/ √ 5 = There fore, Pr(s ≥ )=0.995,and with 99.5% confidence, we can claim that the mean flow rate exceeds In other words, ae this confidence level, we can not be sure that the flow rate exceeds However, at 97.5% confidence level, the lower bound can be calculated to be 3.039, which allows us to accept the claim. -This may sound anomalous, but note that the lowering of the confidence level would allow us to raise the lower bound. 3.3Organizing simulation Runs 1. Independent Replication Method: Here we obtain each data point x i as a sample mean from a separate run(of length m).

-If these runs are started with different random number seeds, we can easily satisfy the requirement that all x i ’s be independent. -The main problem with this method is that we must discard transient data for every run. -With some packages, proper reseeding of all random number sources may also be difficult. 2.sinlge- Run Method: -Here we make one long run of size m ×n, and divide it into n subruns, each of size m. -The motivation for this method is that transient data needs to be discarded only once. -The main difficulty is that the subruns, and henc the x i ’s in the analysis, are no longer independent.

-However, the correlation between X i and X i +1 for any i is expected to decrease as m increase. -To choose an appropriate value of m, We first pick it as discussed above, -And then check if the corresponding autocorrelation coefficient is sufficiently small (<0.02). -The autocorrelation coefficient of lag K is given by R(k)/R(o) where the autocuaartiance R(K) can be defined as

3.Regenerative Method: -A detailed discussion of this method requires concepts from Chapter 5; -Therefore, We discuss it here only briefly and informally. -We start by defining a renewal (or regenerative) state. - A system state is called regenerative if the behavior of the system, Following entry to this state is independent of all past history. -Under steady state, each state (including the regenerative ones) must occur infinitely often. -Let the period between successive entry to the same regenerative state be called a cycle,

- The Cycle are probabilistic replicas of one another(i.e., in each cycle. the observation are identically distributed) and are mutually independent. -Furthermore, the system exhibits steady-state behavior in each cycle. -That is, if the system is started in a regenerative state, then there is no transient period to be discarded. -For the purposes of simulation, we can start the system in a regenerative state and treat each subsequent cycle as an independent run.