A Really Bad Graph

For Discussion Today Project Proposal 1.Statement of hypothesis 2.Workload decisions 3.Metrics to be used 4.Method

© 1998, Geoff Kuenning Designing Experiments Introduction 2 k factorial designs 2 k r factorial designs 2 k-p fractional factorial designs One-factor experiments Two-factor full factorial design without replications Two-factor full factorial design with replications General full factorial designs with k factors © 1998, Geoff Kuenning

Introduction To Experiment Design You know your metrics You know your factors You know your levels You’ve got your instrumentation and test loads Now what? © 1998, Geoff Kuenning

Goals in Experiment Design Obtain maximum information With minimum work –Typically meaning minimum number of experiments More experiments aren’t better if you’re the one who has to perform them Well-designed experiments are also easier to analyze © 1998, Geoff Kuenning

Experimental Replications The system under study will be run with varying levels of different factors, potentially with differing workloads A run with a particular set of levels and other inputs is a replication Often, you need to do multiple replications with a single set of levels and other inputs –For statistical validation © 1998, Geoff Kuenning

Interacting Factors Some factors have effects completely independent of each other –Double the factor’s level, halve the response, regardless of other factors But the effects of some factors depends on the values of other factors –Interacting factors Presence of interacting factors complicates experimental design © 1998, Geoff Kuenning

Basic Problem in Designing Experiments You have chosen some number of factors They may or may not interact How can you design an experiment that captures the full range of the levels? –With minimum amount of work Which combination or combinations of the levels of the factors do you measure? © 1998, Geoff Kuenning

Common Mistakes in Experimentation Ignoring experimental error Uncontrolled parameters Not isolating effects of different factors One-factor-at-a-time experiment designs Interactions ignored Designs require too many experiments © 1998, Geoff Kuenning

Types of Experimental Designs Simple designs Full factorial design Fractional factorial design

Experimental Design ( l 1,0, l 1,1, …, l 1,n1-1 ) x ( l 2,0, l 2,1, …, l 2,n2-1 ) x … x ( l k,0, l k,1, …, l k,nk-1 ) k different factors, each factor with n i levels r replications Factor 1 Factor k Factor 2

© 1998, Geoff Kuenning Simple Designs Vary one factor at a time For k factors with i th factor having n i levels - Assumes factors don’t interact Usually more effort than required Don’t use it, usually

Simple Designs ( l 1,0, l 1,1, …, l 1,n-1 ) x ( l 2,0, l 2,1, …, l 2,n-1 ) x … x ( l k,0, l k,1, …, l k,n-1 ) Factor 1 Factor k Factor 2 fix vary

Simple Designs ( l 1,0, l 1,1, …, l 1,n-1 ) x ( l 2,0, l 2,1, …, l 2,n-1 ) x … x ( l k,0, l k,1, …, l k,n-1 ) Factor 1 Factor k Factor 2

© 1998, Geoff Kuenning Full Factorial Designs For k factors with i th factor having n i levels - Test every possible combination of factors’ levels Captures full information about interaction A hell of a lot of work, though

Full Factorial Designs ( l 1,0, l 1,1, …, l 1,n-1 ) x ( l 2,0, l 2,1, …, l 2,n-1 ) x … x ( l k,0, l k,1, …, l k,n-1 ) Factor 1 Factor k Factor 2

© 1998, Geoff Kuenning Reducing the Work in Full Factorial Designs Reduce number of levels per factor –Generally a good choice –Especially if you know which factors are most important - use more levels for them Reduce the number of factors –But don’t drop important ones Use fractional factorial designs

© 1998, Geoff Kuenning Fractional Factorial Designs Only measure some combination of the levels of the factors Must design carefully to best capture any possible interactions Less work, but more chance of inaccuracy Especially useful if some factors are known not to interact

( l 1,0, l 1,1, …, l 1,n-1 ) x ( l 2,0, l 2,1, …, l 2,n-1 ) x … x ( l k,0, l k,1, …, l k,n-1 ) Fractional Factorial Designs Factor 1 Factor k Factor 2

© 1998, Geoff Kuenning 2 k Factorial Designs Used to determine the effect of k factors –Each with two alternatives or levels Often used as a preliminary to a larger performance study –Each factor measured at its maximum and minimum level –Perhaps offering insight on importance and interaction of various factors

© 1998, Geoff Kuenning Unidirectional Effects Effects that only increase as the level of a factor increases –Or visa versa If this characteristic is known to apply, a 2 k factorial design at minimum and maximum levels is useful Shows whether the factor has a significant effect

© 1998, Geoff Kuenning 2 2 Factorial Designs Two factors with two levels each Simplest kind of factorial experiment design Concepts developed here generalize A form of regression can be easily used here Simplest to show with an example

© 1998, Geoff Kuenning 2 2 Factorial Design Example The Time Warp Operating System Designed to run discrete event simulations in parallel Using an optimistic method Goal is fastest possible completion of a given simulation Usually quality is expressed in terms of speedup Here, the simpler metric of runtime is used

© 1998, Geoff Kuenning Factors and Levels for Time Warp Example First factor - number of nodes used to run the simulation –Vary between 8 and 64 Second factor - whether or not dynamic load management is used –To migrate work from node to node as load in the simulation changes Other factors exists, but ignore them for now

© 1998, Geoff Kuenning Defining Variables for the 2 2 Factorial TW Example if 8 nodes if 64 nodes if no dynamic load management if dynamic load management used

© 1998, Geoff Kuenning Sample Data For Example Single runs of one benchmark simulation DLM (+1) NO DLM (-1) 8 Nodes (-1)64 Nodes (+1)

© 1998, Geoff Kuenning Regression Model for Example y = q 0 + q A x A + q B x B + q AB x A x B Note this is a nonlinear model 820 = q 0 -q A - q B + q AB 217 = q 0 +q A - q B - q AB 776 = q 0 -q A + q B - q AB 197 = q 0 +q A + q B + q AB

© 1998, Geoff Kuenning Regression Model, Con’t 4 equations in 4 unknowns Another way to look at it shown in this table - ExperimentABy 1-1-1y y y y 4

© 1998, Geoff Kuenning Solving the Equations q 0 = 1/4( ) = q A = 1/4( ) = q B = 1/4( ) = -16 q AB = 1/4( ) = 6 So, y = x A - 16x B + 6x A x B

© 1998, Geoff Kuenning The Sign Table Method Another way of looking at the problem in a tabular form IABAB y Total Total/4

© 1998, Geoff Kuenning Allocation of Variation for 2 2 Model Calculate the sample variance of y Numerator is the SST - total variation SST = 2 2 q A q B q AB 2 We can use this to explain what causes the variation in y

© 1998, Geoff Kuenning Terms in the SST 2 2 q A 2 is part of variation explained by the effect of A - SSA 2 2 q B 2 is part of variation explained by the effect of B - SSB 2 2 qA B 2 is part of variation explained by the effect of the interaction of A and B - SSAB SST = SSA + SSB + SSAB

© 1998, Geoff Kuenning Variations in Our Example SST = SSA = SSB = 1024 SSAB = 144 We can now calculate the fraction of the total variation caused by each effect

© 1998, Geoff Kuenning Fractions of Variation in Our Example Fraction explained by A is 99.67% Fraction explained by B is 0.29% Fraction explained by the interaction of A and B is 0.04% So almost all the variation comes from the number of nodes So if you want to run faster, apply more nodes, don’t turn on dynamic load management

© 1998, Geoff Kuenning General 2 k Factorial Designs Used to explain the effects of k factors, each with two alternatives or levels 2 2 factorial designs are a special case Methods developed there extend to the more general case But many more possible interactions between pairs (and trios, etc.) of factors

For Discussion Tuesday March 25 Survey your proceedings for just one paper in which factorial design has been used or, if none, one in which it could have been used effectively. © 2003, Carla Ellis