AZIZ KUSTIYO DEPARTEMEN ILMU KOMPUTER FMIPA IPB

INTRODUCTION TO EXPERIMENTAL DESIGN  The goal of a proper experimental design is to obtain the maximum information with the minimum number of experiments.  This saves considerable labor that would have been spent gathering data.  A proper analysis of experiments also helps in separating out the effects of various factors that might affect the performance.  Also, it allows determining if a factor has a significant effect or if the observed difference is simply due to random variations caused by measurement errors and parameters that were not controlled.

1. TERMINOLOGY The problem is to design a personal workstation, where several choices have to be made.  First, a microprocessor has to be chosen for the CPU. The alternatives are intel core 2 duo, intel core i3, or intel core i5 microprocessor.  Second, a memory size of 1 GB, 2 GB, or 4 GB has to be chosen.  Third, the workload on the workstations could be one of three types—secretarial, managerial, or scientific.  Performance also depends on user characteristics, such as whether users are at a high school, college, or postgraduate level

1. TERMINOLOGY  RESPONSE VARIABLE  FACTORS  LEVELS  PRIMARY FACTORS  SECONDARY FACTORS  REPLICATION  DESIGN  EXPERIMENTAL UNIT  INTERACTION

RESPONSE VARIABLE  Response Variable: The outcome of an experiment is called the response variable.  Generally the response variable is the measured performance of the system.  For example, in the workstation design study the response variable could be the throughput expressed in  tasks completed per unit time, or  response time for tasks, or  any other metric.

FACTORS  Factors: Each variable that affects the response variable and has several alternatives is called a factor.  For example, there are f0ur factors in the workstation design study.  The factors are CPU type, memory size, workload used, and user’s educational level.  The factors are also called predictor variables or predictors.

LEVELS  Levels: The values that a factor can assume are called its levels.  In other words, each factor level constitutes one alternative for that factor. For example, in the workstation design study  the CPU type has three levels: core 2 duo, core i3, core i5  Memory size has three levels: 1 GB, 2 GB, 4 GB.  The workload has three levels: secretarial, managerial, or scientific.  Finally, users could be placed in one of three educational levels—high school graduates, college graduates, and postgraduates.  An alternative term treatment is also used in experimental design literature in place of levels.

PRIMARY FACTORS  Primary Factors: The factors whose effects need to be quantified are called primary factors.  For example, in the workstation design study, one may be primarily interested in quantifying the effect of CPU type and memory size only  Thus, there are two factors in this case.

SECONDARY FACTORS  Secondary Factors: Factors that impact the performance but whose impact we are not interested in quantifying are called secondary factors.  For example, in the workstation study we may not be interested in determining whether performance with postgraduates is better than that with college graduates.  Similarly, we do not want to quantify the difference between the three workloads.  These are the secondary factors.

REPLICATION  Replication: Repetition of all or some experiments is called replication.  For example, if all experiments in a study are repeated three times, the study is said to have three replications.

DESIGN  Design: An, experimental design consists of specifying the number of experiments, the factor level combinations for each experiment, and the number of replications of each experiment.  For example, in the workstation design study, we could perform experiments corresponding to all possible combinations of levels of five factors. This would require 3 × 3 × 3 × 3, or 81, experiments. We could repeat each experiment five times, leading to a total of 405 observations.  This is one possible experimental design.

EXPERIMENTAL UNIT  Experimental Unit: Any entity that is used for the experiment is called an experimental unit.  For example, in the workstation design study, the users hired to use the workstation while measurements are being performed could be considered the experimental unit.  Other examples of experimental units are patients in medical experiments or land used in agricultural experiments.  In all such cases, we are really not interested in comparing the experimental units, although they affect the response.  Therefore, one goal of the experimental design is to minimize the impact of variation among the experimental units.

INTERACTION  Interaction: Two factors A and B are said to interact if the effect of one depends upon the level of the other.  For example, the performance of a system with two factors. As the factor A is changed from level A 1 to level A 2, the performance increases by 2 regardless of the level of factor B. In this case there is no interaction.  Another possibility, as the factor A is changed from level A 1 to level A 2, the performance increases either by 2 or by 3 depending upon whether B is at level B 1 or level B 2, respectively. The two factors interact in this case.  A graphical presentation of this example is given in Figure below. In case (a), the lines are parallel, indicating no interaction. In the second case, the lines are not parallel, indicating interaction.

INTERACTION

REFERENCE  RAJ Jain. The Art of Computer System Performance Analysis: Techniques for Experimental Design, Measurement, Simulation and Modelling. John Wiley and Sons, Inc.