Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 1 Factorial Designs Week 9 Lecture 2

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 2 Agenda Basic factorial design concepts Main and interaction effect Factorial design in computer system performance analysis

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 3 What are factorial designs Two or more independent variables are manipulated in a single experiment They are referred to as factors The major purpose of the research is to explore their effects jointly Factorial design produce efficient experiments, each observation supplies information about all of the factors

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 4 A simple example Investigate an education program with a variety of variations to find out the best combination –Amount of time receiving instruction 1 hour per week vs. 4 hour per week –Settings In-class vs. pull out 2 X 2 factorial design –Number of numbers tells how many factors –Number values tell how many levels –The result of multiplying tells how many treatment groups that we have in a factorial design

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 5 Null outcome None of the treatment has any effect Main effect –is an outcome that is a consistent difference between levels of a factor. Interaction effect –An interaction effect exists when differences on one factor depend on the level you are on another factor.

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 6 Main effects Main effect of time Main effect of setting Main effects on both

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 7 Interaction effect An interaction effect exists when differences on one factor depend on the level of another factor How do we know if there is an interaction in a factorial design? –Statistical analysis will report all main effects and interactions. –If you can not talk about effect on one factor without mentioning the other factor –Spot an interaction in the graphs – whenever there are lines that are not parallel there is an interaction present!

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 8 Interaction effect Interaction as a difference in magnitude of response Interaction as a difference in direction of response

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 9 Factorial design variations A 2 X 3 example study the effect of different treatment combinations for cocaine abuse. –Factor 1: treatment psychotherapy behavior modification –Factor 2: inpatient day treatment outpatient –Dependent variable severity of illness rating

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 10 Factorial designs in computer system performance analysis Personal workstation design –Processor: 68000, Z80, 8086 –Memory size: 512K 2M or 8M bytes –Number of disks: one, two or three –Workload: Secretarial, managerial or scientific –User education: high school, college, post- graduate level Dependent variable –Throughput, response time

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney factorial design Two factors, each at two levels Example: workstation design –Factor 1: memory size –Factor 2: cache size –DV: performance in MIPS Cache size Memory size 4M byte8M byte 1K1545 2K2575

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 12 Quantify effects We want to learn which factor contribute more to the performance. –Define two variable –The regression model

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 13 Quantifying results (cont) Resolving those coefficients We get How do you read this?

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 14 Quantify effects by sign table Sign table method IABABy Total Total/4

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 15 2 K factorial design K factors, each at two level 2 K experiments 2 3 design example –In designing a personal workstation, the three factors needed to be studied are: cache size, memory size and number of processors FactorLevel -1Level 1 Memory size4Mbytes16Mbytes Catch size1Kbytes2Kbytes Number of processors 12 Cache size (Kbytes) 4 Mbytes16 Mbytes

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 16 2 k factorial design with replication r replications of 2 k experiments –2 K r observations –Allows estimation of experimental errors –2 2 3 design example The memory-cache experiments were repeated three times each. The result is shown below Cache sizeMemory size 4M8M 1 K15, 18, 1245, 48,51 2K25, 28, 1975,75,81

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 17 Full and fractional factorial design Full factorial design –Study all combinations –Can find effect of all factors –May try 2 K factorial design first Fractional (incomplete) factorial design –Leave some treatment groups empty –Less information –May not get all interactions –No problem if interaction is negligible

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 18 2 k-p Fractional factorial design Large number of factors –Large number of experiments –Full factorial design too expensive –Use a fractional factorial design 2 k-p design allows analyzing k factors with only 2 k-p experiments. –2 k-1 design requires only half as many experiments –2 k-2 design requires only one quarter of the experiments

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 19 Example: Design Exp No.ABCCEFG Study 7 factors with only 8 experiments When quantify the effects, just calculate the main effects Will be able to eliminate some factors in further study.

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 20 Quantify model IABCCEFG Total Total/8

Friday, May 14, 2004 ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney 21 Preparing the sign table Choose k - p factors and prepare a complete sign table for a full factorial design with k-p factors Of the 2 k-p –k +p -1 column on the right, choose p columns and mark them with the p factors that were not chosen in step 1.