1. For Discussion Today (when the alarm goes off) Survey your proceedings for just one paper in which factorial design has been used or, if none, one in which it could have been used effectively. © 2003, Carla Ellis

2. For Discussion Today? Project Proposal 1.Statement of hypothesis 2.Workload decisions 3.Metrics to be used 4.Method

3. Experimental Design ( l 1,0, l 1,1, …, l 1,n1-1 ) x ( l 2,0, l 2,1, …, l 2,n2-1 ) x … x ( l k,0, l k,1, …, l k,nk-1 ) k different factors, each factor with n i levels r replications Factor 1 Factor k Factor 2

4. ( l 1,0, l 1,1, …, l 1,n-1 ) x ( l 2,0, l 2,1, …, l 2,n-1 ) x … x ( l k,0, l k,1, …, l k,n-1 ) 2 k-p Fractional Factorial Designs Factor 1 Factor k Factor 2 Only considering 2 levels of k factors with 2 k-p experiments for suitable p

5. ( l 1,0, l 1,1, …, l 1,n-1 ) x ( l 2,0, l 2,1, …, l 2,n-1 ) x … x ( l k,0, l k,1, …, l k,n-1 ) 2 k-p Fractional Factorial Designs Factor 1 Factor k Factor 2 Only considering 2 levels of k factors with 2 k-p experiments for suitable p

6. © 1998, Geoff Kuenning 2 k-p Fractional Factorial Designs Introductory example of a 2 k-p design Preparing the sign table for a 2 k-p design Confounding Algebra of confounding Design resolution

7. Why and When to Use Getting by on fewer experiments than 2 k When you know certain interactions are negligible and you can “overload” them (confound). When you want to focus in on what further experiments to do (where it matters) without wasting time and effort.

8. © 1998, Geoff Kuenning Introductory Example of a 2 k-p Design Exploring 7 factors in only 8 experiments:

9. © 1998, Geoff Kuenning Analysis of 2 7-4 Design Column sums are zero: Sum of 2-column product is zero: Sum of column squares is 2 7-4 = 8 Orthogonality allows easy calculation of effects:

10. © 1998, Geoff Kuenning Effects and Confidence Intervals for 2 k-p Designs Effects are as in 2 k designs: % variation proportional to squared effects For standard deviations, confidence intervals: –Use formulas from full factorial designs –Replace 2 k with 2 k-p

11. © 1998, Geoff Kuenning Preparing the Sign Table for a 2 k-p Design Prepare sign table for k-p factors Assign remaining factors

12. © 1998, Geoff Kuenning Sign Table for k-p Factors Same as table for experiment with k-p factors –I.e., 2 (k-p) table –2 k-p rows and 2 k-p columns –First column is I, contains all 1’s –Next k-p columns get k-p selected factors –Rest are products of factors

13. © 1998, Geoff Kuenning Assigning Remaining Factors 2 k-p -(k-p)-1 product columns remain Choose any p columns –Assign remaining p factors to them –Any others stay as-is, measuring interactions

14. © 1998, Geoff Kuenning Confounding The confounding problem An example of confounding Confounding notation Choices in fractional factorial design

15. © 1998, Geoff Kuenning The Confounding Problem Fundamental to fractional factorial designs Some effects produce combined influences –Limited experiments mean only combination can be counted Problem of combined influence is confounding –Inseparable effects called confounded effects

16. © 1998, Geoff Kuenning An Example of Confounding Consider this 2 3-1 table: Extend it with an AB column:

17. © 1998, Geoff Kuenning Analyzing the Confounding Example Effect of C is same as that of AB: q C = (y 1 -y 2 -y 3 +y 4 )/4 q AB = (y 1 -y 2 -y 3 +y 4 )/4 Formula for q C really gives combined effect: q C +q AB = (y 1 -y 2 -y 3 +y 4 )/4 No way to separate q C from q AB –Not a problem if q AB is known to be small

18. © 1998, Geoff Kuenning Confounding Notation Previous confounding is denoted by equating confounded effects: C = AB Other effects are also confounded in this design: A = BC, B = AC, C = AB, I = ABC –Last entry indicates ABC is confounded with overall mean, or q 0

19. © 1998, Geoff Kuenning Choices in Fractional Factorial Design Many fractional factorial designs possible –Chosen when assigning remaining p signs –2 p different designs exist for 2 k-p experiments Some designs better than others –Desirable to confound significant effects with insignificant ones –Which usually means low-order with high-order

20. © 1998, Geoff Kuenning Algebra of Confounding Rules of the algebra Generator polynomials

21. © 1998, Geoff Kuenning Rules of Confounding Algebra Particular design can be characterized by single confounding –Traditionally, the I = wxyz... confounding Others can be found by multiplying by various terms –I acts as unity (e.g., I times A is A) –Squared terms disappear (AB 2 C becomes AC)

22. © 1998, Geoff Kuenning Example: 2 3-1 Confoundings Design is characterized by I = ABC Multiplying by A gives A = A 2 BC = BC Multiplying by B, C, AB, AC, BC, and ABC: B = AB 2 C = AC, C = ABC 2 = AB, AB = A 2 B 2 C = C, AC = A 2 BC 2 = B, BC = AB 2 C 2 = A, ABC = A 2 B 2 C 2 = I Note that only first line is unique in this case

23. © 1998, Geoff Kuenning Generator Polynomials Polynomial I = wxyz... is called generator polynomial for the confounding A 2 k-p design confounds 2 p effects together –So generator polynomial has 2 p terms –Can be found by considering interactions replaced in sign table

24. © 1998, Geoff Kuenning Example of Finding a Generator Polynomial Consider a 2 7-4 design Sign table has 2 3 = 8 rows and columns First 3 columns represent A, B, and C Columns for D, E, F, and G replace AB, AC, BC, and ABC columns respectively –So confoundings are necessarily: D = AB, E = AC, F = BC, and G = ABC

25. © 1998, Geoff Kuenning Turning Basic Terms into Generator Polynomial Basic confoundings are D = AB, E = AC, F = BC, and G = ABC Multiply each equation by left side: I = ABD, I = ACE, I = BCF, and I = ABCG or I = ABD = ACE = BCF = ABCG

26. © 1998, Geoff Kuenning Finishing the Generator Polynomial Any subset of above terms also multiplies out to I –E.g., ABD times ACE = A 2 BCDE = BCDE Expanding all possible combinations gives the 16-term generator (book is wrong): I = ABD = ACE = BCF = ABCG = BCDE = ACDF = CDG = ABEF = BEG = AFG = DEF = ADEG = BDFG = CEFG = ABCDEFG

27. © 1998, Geoff Kuenning Design Resolution Definitions leading to resolution Definition of resolution Finding resolution Choosing a resolution

28. © 1998, Geoff Kuenning Definitions Leading to Resolution Design is characterized by its resolution Resolution measured by order of confounded effects Order of effect is number of factors in it –E.g., I is order 0, and ABCD is order 4 Order of confounding is sum of effect orders –E.g., AB = CDE would be of order 5

29. © 1998, Geoff Kuenning Definition of Resolution Resolution is minimum order of any confounding in design Denoted by uppercase Roman numerals –E.g, 2 5-1 with resolution of 3 is called R III –Or more compactly,

30. © 1998, Geoff Kuenning Finding Resolution Find minimum order of effects confounded with mean –I.e., search generator polynomial Consider earlier example: I = ABD = ACE = BCF = ABCG = BCDE = ACDF = CDG = ABEF = BEG = AFG = DEF = ADEG = BDFG = ABDG = CEFG = ABCDEFG So it’s R III design

31. © 1998, Geoff Kuenning Choosing a Resolution Generally, higher resolution is better Because usually higher-order interactions are smaller Exception: when low-order interactions are known to be small –Then choose design that confounds those with important interactions –Even if resolution is lower

32. © 1998, Geoff Kuenning One Factor Experiments If there’s only one important categorical factor But it has more than two interesting alternatives –Methods work for two alternatives, but they reduce to 2 1 factorial designs If the single variable isn’t categorical, examine regression, instead Method allows multiple replications

33. One Factor Design ( l 0, l 1, …, l n-1 ) Categorical Single Factor

34. © 1998, Geoff Kuenning What is This Good For? Comparing truly comparable options –Evaluating a single workload on multiple machines –Or with different options for a single component –Or single suite of programs applied to different compilers

35. © 1998, Geoff Kuenning What Isn’t This Good For? Incomparable “factors” –Such as measuring vastly different workloads on a single system Numerical factors –Because it won’t predict any untested levels

36. © 1998, Geoff Kuenning An Example One Factor Experiment You are buying an authentication server to authenticate single-sized messages Four different servers are available Performance is measured in response time –Lower is better This choice could be assisted by a one- factor experiment

37. © 1998, Geoff Kuenning The Single Factor Model y ij =  j + e ij y ij is the i th response with factor at alternative j  is the mean response  j is the effect of alternative j e ij is the error term

38. © 1998, Geoff Kuenning One Factor Experiments With Replications Initially, assume r replications at each alternative of the factor Assuming a alternatives of the factor, a total of ar observations The model is thus

39. © 1998, Geoff Kuenning Sample Data for Our Example Four alternatives, with four replications each (measured in seconds) A B C D 0.960.751.010.93 1.051.220.891.02 0.821.130.941.06 0.940.981.381.21

40. © 1998, Geoff Kuenning Computing Effects We need to figure out  and  j We have the various y ij ’s So how to solve the equation? Well, the errors should add to zero And the effects should add to zero

41. © 1998, Geoff Kuenning Calculating  Since sum of errors and sum of effects are zero, And thus,  is equal to the grand mean of all responses

42. © 1998, Geoff Kuenning Calculating  for Our Example

43. © 1998, Geoff Kuenning Calculating  j  j is a vector of responses –One for each alternative of the factor To find the vector, find the column means For each j, of course We can calculate these directly from observations

44. © 1998, Geoff Kuenning Calculating a Column Mean But we also know that y ij is defined to be So,

45. © 1998, Geoff Kuenning Calculating the Parameters Remember, the sum of the errors for any given row is zero, so So we can solve for  j -

46. © 1998, Geoff Kuenning Parameters for Our Example Server A BC D Column.9425 1.02 1.055 1.055 Mean Subtract  from column means to get parameters Parameters -.076.002.037.037

47. © 1998, Geoff Kuenning Estimating Experimental Errors Estimated response is But we measured actual responses –Multiple ones per alternative So we can estimate the amount of error in the estimated response Using methods similar to those used in other types of experiment designs yjyj N

48. © 1998, Geoff Kuenning Finding Sum of Squared Errors SSE estimates the variance of the errors We can calculate SSE directly from the model and observations Or indirectly from its relationship to other error terms

49. © 1998, Geoff Kuenning SSE for Our Example Calculated directly - SSE = (.96-(1.018-.076)) 2 + (1.05 - (1.018-.076)) 2 +... + (.75- (1.018+.002)) 2 + (1.22 - (1.018 +.002)) 2 +... + (.93 -(1.018+.037)) 2 =.3425

50. © 1998, Geoff Kuenning Allocating Variation To allocate variation for this model, start by squaring both sides of the model equation Cross product terms add up to zero

51. © 1998, Geoff Kuenning Variation In Sum of Squares Terms SSY=SS0+SSA+SSE Giving us another way to calculate SSE

52. © 1998, Geoff Kuenning Sum of Squares Terms for Our Example SSY = 16.9615 SS0 = 16.58256 SSA =.03377 So SSE must equal 16.9615-16.58256-.03377 –Which is.3425 –Matching our earlier SSE calculation

53. © 1998, Geoff Kuenning Assigning Variation SST is the total variation SST = SSY - SS0 = SSA + SSE Part of the total variation comes from our model Part of the total variation comes from experimental errors A good model explains a lot of variation

54. © 1998, Geoff Kuenning Assigning Variation in Our Example SST = SSY - SS0 = 0.376244 SSA =.03377 SSE =.3425 Percentage of variation explained by server choice

55. © 1998, Geoff Kuenning Analysis of Variance The percentage of variation explained can be large or small Regardless of which, it may be statistically significant or insignificant To determine significance, use an ANOVA procedure –Assumes normally distributed errors...

56. © 1998, Geoff Kuenning Running an ANOVA Procedure Easiest to set up a tabular method Like method used in regression models –With slight differences Basically, determine ratio of the Mean Squared of A (the parameters) to the Mean Squared Errors Then check against F-table value for number of degrees of freedom

57. © 1998, Geoff Kuenning ANOVA Table for One- Factor Experiments Compo- Sum of Percentage of Degrees of Mean F- F- nent Squares Variation Freedom Square Comp Table y ar 1 SST=SSY-SS0 100 ar-1 A a-1 F[1-  ; a-,a(r-1)] e SSE=SST-SSA a(r-1)

58. © 1998, Geoff Kuenning ANOVA Procedure for Our Example Compo- Sum of Percentage of Degrees of Mean F- F- nent Squares Variation Freedom Square Comp Table y 16.96 16 16.58 1.376 100 15 A.034 8.97 3.011.394 2.61 e.342 91.0 12.028

59. © 1998, Geoff Kuenning Analysis of Our Example ANOVA Done at 90% level Since F-computed is.394, and the table entry at the 90% level with n=3 and m=12 is 2.61, –The servers are not significantly different

60. © 1998, Geoff Kuenning One-Factor Experiment Assumptions Analysis of one-factor experiments makes the usual assumptions –Effects of factor are additive –Errors are additive –Errors are independent of factor alternative –Errors are normally distributed –Errors have same variance at all alternatives How do we tell if these are correct?

61. © 1998, Geoff Kuenning Visual Diagnostic Tests Similar to the ones done before –Residuals vs. predicted response –Normal quantile-quantile plot –Perhaps residuals vs. experiment number

62. © 1998, Geoff Kuenning Residuals vs. Predicted For Our Example

63. © 1998, Geoff Kuenning Residuals vs. Predicted, Slightly Revised

64. © 1998, Geoff Kuenning What Does This Plot Tell Us? This analysis assumed the size of the errors was unrelated to the factor alternatives The plot tells us something entirely different –Vastly different spread of residuals for different factors For this reason, one factor analysis is not appropriate for this data –Compare individual alternatives, instead –Using methods discussed earlier

65. © 1998, Geoff Kuenning Could We Have Figured This Out Sooner? Yes! Look at the original data Look at the calculated parameters The model says C & D are identical Even cursory examination of the data suggests otherwise

66. © 1998, Geoff Kuenning Looking Back at the Data A B C D 0.960.751.010.93 1.051.220.891.02 0.821.130.941.06 0.940.981.381.21 Parameters -.076.002.037.037

67. © 1998, Geoff Kuenning Quantile-Quantile Plot for Example

68. © 1998, Geoff Kuenning What Does This Plot Tell Us? Overall, the errors are normally distributed If we only did the quantile-quantile plot, we’d think everything was fine The lesson - test all the assumptions, not just one or two

69. © 1998, Geoff Kuenning One-Factor Experiment Effects Confidence Intervals Estimated parameters are random variables –So we can compute their confidence intervals Basic method is the same as for confidence intervals on 2 k r design effects Find standard deviation of the parameters –Use that to calculate the confidence intervals –Typo in book, pg 366, example 20.6, in formula for calculating this –Also typo on pg. 365 - degrees of freedom is a(r-1), not r(a- 1)

70. © 1998, Geoff Kuenning Confidence Intervals For Example Parameters s e =.158 Standard deviation of  =.040 Standard deviation of  j =.069 95% confidence interval for  = (.932, 1.10) 95% CI for    = (-.225,.074) 95% CI for    = (-.148,.151) 95% CI for    = (-.113,.186) 95% CI for    = (-.113,.186)

71. © 1998, Geoff Kuenning Unequal Sample Sizes in One-Factor Experiments Can you evaluate a one-factor experiment in which you have different numbers of replications for alternatives? Yes, with little extra difficulty See book example for full details

72. © 1998, Geoff Kuenning Changes To Handle Unequal Sample Sizes The model is the same The effects are weighted by the number of replications for that alternative: And things related to the degrees of freedom often weighted by N (total number of experiments)