1. Experimental Design, Response Surface Analysis, and Optimization

2. Outline Motivation and Terminology
Difficulties in Solving the Basic Problem
Examples of Factors and Responses
Types/Examples of Experimental Design
Full Factorial Designs
Randomness of Effects
Example: Full Factorial Design
Situations with Many Factors
Response Surfaces and Metamodels
Regression Analysis
Response Surface Methodology

3. Motivation and Terminology
Useful when there are many alternatives to consider (e.g., numerous capacity levels of various types, numerous parameters for a proposed inventory system)
Two basic types of variables: factors and responses
Factors: input parameters:
-controllable or uncontrollable
-quantitative or qualitative
Responses: outputs from the simulation model:
-uncertain in nature
Basic problem: find the best levels (or values of the parameters) in terms of the responses
Experimental Design can tell you which alternatives to simulate so that you obtain the desired information with the least amount of simulation

4. Difficulties in Solving the Basic Problem
Multiple Responses
Uncertain Responses

5. Examples of Factors
1. Mean interarrival time (uncontrollable, quantitative)
2. Mean service time (controllable or uncontrollable, quantitative)
3. Number of servers (controllable, quantitative)
4. Queuing discipline (controllable, qualitative)
5. Reorder point (controllable, quantitative)
6. Mean interdemand time (uncontrollable, quantitative)
7. Distribution of interdemand time (uncontrollable, qualitative)

6. Examples of Responses 1. Mean daily production rate
2. Mean time in the system for patients
3. Mean inventory level
4. Number of customers who wait for more than 5 minutes

7. Types/Examples of Experimental Designs
Completely Randomized Designs
Randomized Complete Block Designs
Nested Factorial Designs
Split Plot Type Designs
Latin Square Type Designs
Full Factorial Designs
Fractional Factorial Designs

8. 2k Factorial Designs
Suppose that we have k (k > 2) factors. A 2k factorial design would require that two levels be chosen for each factor, and that n simulation runs (replications) be made at each of the 2k possible factor-level combinations (design points). For 3 factors, this yields a Design Matrix:

9. Design Matrix for 3 Factors
2k Factorial Designs
Design Matrix for 3 Factors
Factor 1 Factor 2 Factor 3
Design Point Level Level Level Response O1 O2 O3 O4 O5 O6 O7 O8
“+” refers to one level of a factor and “-” refers to the other level. Normally, for quantitative factors, the smallest and largest levels for each factor are chosen.

10. 2k Factorial Designs: Estimating Main Effects
The main effect of factor 1 is the change in the response variable as a result of the change in the level of the factor, averaged over all levels of all of the other factors
If the effect of some factor depends on the level of another factor, these factors are said to interact.
The degree of interaction (two-factor interaction effect) between two factors i and j is defined as half the difference between the average effect of factor i when factor j is at its “+” level ( and all factors other than i and j are held constant) and the average effect of i when j is at its “-” level; for example,

11. Example: Full Factorial (2k) Design
Consider a simulation model of reorder point, reorder quantity inventory system. The two decision variables, or factors, to consider are the reorder point (P) and the reorder quantity (Q) for the inventory system. The maximum and minimum allowable values for each are given below:
Suppose that the response variable output by the model is the long-run average monthly cost (composed of three components: holding cost, shortage costs, and ordering costs) in thousands of dollars.

12. Example: Full Factorial (2k) Design
A 22 factorial design matrix with simulation results (for 10 replications at each design point) might be given by:
where a factor level of “-” indicates the minimum possible value for that factor, and a factor level of “+” indicates the maximum possible level; for example, design point 2 has P=40 and Q=15.

13. Example: Full Factorial (2k) Design
The response given is the average cost over the 10 replications. Now, the main effects are given by:
The interaction effect (ePQ) is given by:
Therefore, the average effect of increasing P from 20 to 40 is to increase monthly cost by 1.2, and the average effect of increasing Q from 15 to 50 is to decrease monthly cost by Hence, it would be advisable to set P as low as possible and set Q as high as possible.

14. Example: Full Factorial (2k) Design
Also since the interaction effect is positive, it would seem advisable to set P and Q at opposite levels. (Of course, all of the above could be inferred from a cursory analysis of the responses for the various design points.) Note also that the literal interpretation of main effects assumes no interaction effects (pages 669 and 670 of Law and Kelton, 1991).

15. Randomness of Effects
Note that the main and interaction effects computed in the previous examples are just random variables. To determine if the effects are “significant” or real, and not due to random fluctuations, one could compute values for the main and interaction effects 10 times (once for each replication) and form confidence intervals for each of the main effects, and the interactions effect. If the confidence interval contains 0, then the effect is not statistically significant. (Note that statistical significance does not necessarily imply practical significance).

16. Situations with Many Factors
When there are many factors to consider, full factorial, or even fractional factorial designs may not be feasible from a computational standpoint.
Other types of design (e.g., Plackett-Burman designs or “supersaturated designs”) may be appropriate (Mauro, 1986).
Another approach is to reduce the number of factors to consider via “factor screening” techniques, involving, for example, group screening in which a whole group is treated as a factor.

17. Response Surfaces and Metamodels
A response surface is a graph of a response variable as a function of the various factors.
A metamodel (literally, model of the simulation model), is an algebraic representation of the simulation model, with the factors as independent variables and a response as the dependent variable. It represents an approximation of the response surface.
The typical metamodel used in a simulation application is a regression model.

18. Response Surfaces and Metamodels
A metamodel through the use of response surface methodology can be used to find optimal values for a set of factors. It can also be used to answer “what if” questions. (Experimentation with a metamodel is typically much less expensive than using a simulation model directly).
An experimental design process assumes a particular metamodel, e.g.,

19. Basic Concepts of Regression Analysis
Regression is used to determine the “best” functional relation among variables.
Suppose that the functional relationship is represented as:
E(Y) = f (X1, ..., Xp / B1, ..., BE)
where E(Y) is the expected value of the response variable Y; the X1, ..., Xp are factors; and the B1, ..., BE are function parameters; e.g.,
E(Y) = B1 + B2 X1 + B3 X2 + B4 X1 X2

20. Basic Concepts of Regression Analysis
The observed value for Y, for a given set of X ’s, is assumed to be a random variable, given by:
Y = f (X1, ..., Xp/B1, ..., BE) +
Where is a random variable with mean equal to 0 and variance The values for B1,...,BE are obtained by minimizing the sum of squares of the deviations.

21. Response Surface Methodology
Source: (Fu, 1994)
Response surface methodology (RSM) involves a combination of metamodeling (i.e., regression) and sequential procedures (iterative optimization).

22. Response Surface Methodology
RSM involves two phases:
Fit a linear regression model to some initial data points in the search space (through replications of the simulation model). Estimate a steepest descent direction from the linear regressions model, and a step size to find a new (and better) solution in the search space. Repeat this process until the linear regression model becomes inadequate (indicated by when the slope of the linear response surface is approximately 0; i.e., when the interaction effects become larger than than the main effects).
Fit a nonlinear quadratic regression equation to this new area of the search space. Then find the optimum of this equation.

23. Terminology in Experimental Design
Source:(Ostle, 1963)
Replication - the repetition of the basic experiment
Treatment - a specific combination of several factor levels
Experimental Unit - the unit to which a single treatment is applied to one replication of the basic experiment
Experimental Error - the failure of two identically treated experimental units to yield identical results
Confounding - the “mixing up” of two or more factors so that it’s impossible to separate the effects

24. Terminology in Experimental Design
Randomization - randomly assigning treatments to experimental units (assures independent distribution of errors)
Main Effect (of a factor) - a measure of the change in a response variable to changes in the level of the factor averaged over all levels of all the other factors
Interaction is an additional effect (on the response) due to the combined influence of two or more factors