Lecture 10 Comparison and Evaluation of Alternative System Designs

2 Basic Concept of Confidence Interval  Confidence Interval provides range of values based on observations from 1 sample,.  A probability that the population parameter falls somewhere within the interval.  Confidence Interval provides range of values based on observations from 1 sample,.  A probability that the population parameter falls somewhere within the interval. Confidence Limit (Lower) Confidence Limit (Upper) Confidence Interval Sample Statistic (Point Estimate) C.I. for a mean:

3 Output Analysis for Two Systems  One of the most important uses of simulation is the comparison of alternative system designs.  A two-sided 100(1 –  )% C.I. for  1 –  2 will always be of the form:  One of the most important uses of simulation is the comparison of alternative system designs.  A two-sided 100(1 –  )% C.I. for  1 –  2 will always be of the form: ParameterEstimator System 1 11 System 2 22

4 Comparison of Alternatives  C.I. bounds the true difference  1 –  2 within the range with probability 1 – . ( x ) 00 0

5 Independent Sampling with Equal Variances  Independent sampling means that different and independent random-number streams will be used to simulate the two systems. where  Independent sampling means that different and independent random-number streams will be used to simulate the two systems. where

6 Independent Sampling with Unequal Variances  If the assumption of equal variances cannot safely be made, an approximate 100(1 –  )% C.I. for  1 –  2 can be computed as follows. where  If the assumption of equal variances cannot safely be made, an approximate 100(1 –  )% C.I. for  1 –  2 can be computed as follows. where

7 Correlated Sampling  Correlated sampling means that, for each replication, the same random numbers are used to simulate both system. Therefore, R 1 and R 2 must be equal, say R 1 = R 2 = R. where  Correlated sampling means that, for each replication, the same random numbers are used to simulate both system. Therefore, R 1 and R 2 must be equal, say R 1 = R 2 = R. where

8 Experimental Design  Experimental design provides a way of deciding before the runs are made which particular configurations to simulate so that the desired information can be obtained with the least amount of simulating.  The input parameters and structural assumptions composing a model are called factors, and the output performance measures are called responses.  Experimental design provides a way of deciding before the runs are made which particular configurations to simulate so that the desired information can be obtained with the least amount of simulating.  The input parameters and structural assumptions composing a model are called factors, and the output performance measures are called responses.

9 Experimental Design (cont’)  Each possible value of a factor is called a level of the factor.  A combination of factors all at a specified level is called a treatment.  Factors can be either quantitative or qualitative.  These factors are collectively called decision variables, or policy variables.  E.g., queue discipline (policy variable), number of physicians (decision variable).  Each possible value of a factor is called a level of the factor.  A combination of factors all at a specified level is called a treatment.  Factors can be either quantitative or qualitative.  These factors are collectively called decision variables, or policy variables.  E.g., queue discipline (policy variable), number of physicians (decision variable).

10 Factorial Design  Sometimes simulation analyses are used to determine the effects that various factors exert on selected performance criteria.  Factorial designed experiments are one means of providing this type of information.  The results produced from these experiments can be statistically analyzed to measure the 1) main effects, and 2) interactive effects that selected factors exert on performance indices (system responses).  Sometimes simulation analyses are used to determine the effects that various factors exert on selected performance criteria.  Factorial designed experiments are one means of providing this type of information.  The results produced from these experiments can be statistically analyzed to measure the 1) main effects, and 2) interactive effects that selected factors exert on performance indices (system responses).

11 Factorial Design (cont’)  A main effect (denote E i ) is the average change in a response resulting from raising the i th factor from a specified “low level” to a specified “high level”.  Suppose we perform a simulation to investigate three factors (lot size, quantity of machines, and set-up time) regarding their individual effects on a product’s makespan.  A main effect (denote E i ) is the average change in a response resulting from raising the i th factor from a specified “low level” to a specified “high level”.  Suppose we perform a simulation to investigate three factors (lot size, quantity of machines, and set-up time) regarding their individual effects on a product’s makespan. FactorLow Level (–)High Level (+) Lot Size, E Machine Quantity, E 2 12 Rework Rate, E 3 6%12%

12 Main Effect Factor  A design matrix with 2 x 2 x 2 design points is constructed. Design PointFactor 1 Level (Lot Size) Factor 2 Level (Machine Qty) Factor 3 Level (Rework Rate) Response (Makespan) 1+++R 1 = 5.7 2–++R 2 = –+R 3 = ––+R 4 = –R 5 = 5.7 6–+–R 6 = ––R 7 = –––R 8 = R 1 – R 2 + R 3 – R 4 + R 5 – R 6 + R 7 – R 8 2 k-1 Main effect factor #1 E 1 =

13 Main Effect Factor and Interactive Effect Factors + R 1 + R 2 – R 3 – R 4 + R 5 + R 6 – R 7 – R 8 2 k-1 Main effect factor #2 E 2 = + R 1 + R 2 + R 3 + R 4 – R 5 – R 6 – R 7 – R 8 2 k-1 Main effect factor #3 E 3 = Interactive effect factor #1 and #2 E 12 = Interactive effect factor #1 and #3 E 13 = Interactive effect factor #2 and #3 E 23 = + R 1 – R 2 – R 3 + R 4 + R 5 – R 6 – R 7 + R 8 2 k-1 + R 1 – R 2 + R 3 – R 4 – R 5 + R 6 – R 7 + R 8 2 k-1 + R 1 + R 2 – R 3 – R 4 – R 5 – R 6 + R 7 + R 8 2 k-1

14 Analyzing Factorial Designed Experiments  An interactive effect tells us if the effect of a given factor is influenced by the level of another factor.  If there is a significant interactive effect, then we cannot be certain that a main effect is due solely to the raising or lowering of a factor level.  An interactive effect tells us if the effect of a given factor is influenced by the level of another factor.  If there is a significant interactive effect, then we cannot be certain that a main effect is due solely to the raising or lowering of a factor level. Main Effects Lot Size, E Machine Quantity, E Rework Rate, E Interactive Effects Lot Size & Machine Qty, E Lot Size & Rework Rate, E Machine Qty & Rework Rate, E Raising a lot size from 5 to 10 is to increase product makespan an average of 0.8 days per part. Adding an additional machine decreased the makespan by an average of 6.3 days.

15 Documentation and Conclusions  Documentation can be divided into five areas:  Objectives and Assumptions  Model Input Parameters  Experimental Design  Results  Conclusions  Objectives and Assumptions  All objectives and assumptions should be recorded at the onset of any simulation project. Any changes or modifications made during the course of building a model need to be included in the final report.  Documentation can be divided into five areas:  Objectives and Assumptions  Model Input Parameters  Experimental Design  Results  Conclusions  Objectives and Assumptions  All objectives and assumptions should be recorded at the onset of any simulation project. Any changes or modifications made during the course of building a model need to be included in the final report.

16 Documentation and Conclusions (cont’)  Model Input Parameters  This section contains a recap of the data used with a simulation. System flow charts, mathematical calculations, performance criteria, solution constraints, solution restrictions, and any cost related information should be included.  Experimental Design  The information summarized in this category is comprised of descriptions regarding the alternatives investigated, the experiments designed for comparing alternatives, starting conditions, stopping conditions, a history of the random number streams employed with each experiment, and an account for the number of model replications performed for each alternative.  Model Input Parameters  This section contains a recap of the data used with a simulation. System flow charts, mathematical calculations, performance criteria, solution constraints, solution restrictions, and any cost related information should be included.  Experimental Design  The information summarized in this category is comprised of descriptions regarding the alternatives investigated, the experiments designed for comparing alternatives, starting conditions, stopping conditions, a history of the random number streams employed with each experiment, and an account for the number of model replications performed for each alternative.

17 Documentation and Conclusions (cont’)  Results  This section is composed of the output data produced by a simulation. It also provides an overview of the statistical analyses performed on the data. Tables and graphical charts which illustrate the findings are very beneficial.  Conclusions  One of the final steps in any decision-making process is to make conclusions and recommendations. This demands that benefit-to- cost ratios be investigated for each alternative. What are the total costs (tangible and intangible) needed to implement an alternative, and what are the total benefits anticipated from doing it?  Results  This section is composed of the output data produced by a simulation. It also provides an overview of the statistical analyses performed on the data. Tables and graphical charts which illustrate the findings are very beneficial.  Conclusions  One of the final steps in any decision-making process is to make conclusions and recommendations. This demands that benefit-to- cost ratios be investigated for each alternative. What are the total costs (tangible and intangible) needed to implement an alternative, and what are the total benefits anticipated from doing it?

18 Risk and Uncertainties  Since decision-making is based on the precept of prediction, risk and uncertainties are almost always involved. The potential labor requirements forecasted with a given alternative may fall within a range. This can be classified as an uncertainty. The potential outcome for an alternative may also vary. This can be designated as a risk. Any uncertainties and risks associated with an alternative should be discussed in the final documentation.