1. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 1 Introduction to Experiment Design Shiv Kalyanaraman Rensselaer Polytechnic Institute shivkuma@ecse.rpi.edu http://www.ecse.rpi.edu/Homepages/shivkuma Refs: Chap 16-17 of Raj Jain’s book Adapted from Prof. Raj Jain’s slides

2. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 2 Why Experiment design? q Problem: validation and results only as good as your test cases! q How to design a critical set of test cases ? q Idea: Parameterize and use black-box strategy System Parameters or Factors Metrics

3. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 3 Performance, Metrics and Parameters q Performance questions: q Absolute: How fast does computer A run MY program ? q Relative: Is machine A faster than machine B, and if so, how much faster ? q Parameters: factors or inputs q Eg: clock rate, poisson inter-arrivals q Metrics: functions of factors or parameters q Eg: throughput, response time, queue length... q Metric should characterize the design tradeoffs adequately q Metrics are usually functions of many factors. Use of one factor alone may be misleading.

4. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 4 Choose metrics to reflect appropriate tradeoffs q Network users: services and performance that their applications need, e.g., guarantee that each message it sends will be delivered without error within a certain amount of time q Network designers: cost-effective design e.g., that network resources are efficiently utilized and fairly allocated to different users q Users + designer perspectives enough to meet 3 factors of interface/architecture design. But... q Network providers: system that is easy to administer and manage e.g., that faults can be easily isolated and it is easy to account for usage q Require management tools and interfaces. q Often considered to the basic protocol interface design

5. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 5 Goals of Experiment Design q Design a proper set of (simulation or measurement) experiments: maximize information gained with minimum experiments! q Develop a (regression) model that best describes the data obtained q Estimate the contribution of each factor and its values (I.e alternatives) to the performance variation q Isolate measurement errors q Estimate confidence intervals for model parameters q Check if the alternatives are significantly different q Check if the model is adequate

6. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 6 Example: TCP/AQM Design Problem q TCP versions: Reno or SACK q Max Segment sizes: 100 B vs 1000 B q Buffer Size: 10 pkts vs 100 pkts q AQM: Drop tail vs RED vs REM q AQM parameters … q Workload: FTP q Configuration parameters: 3 flows vs 10 flows q Metrics: total throughput (efficiency), C.o.V of per-flow throughputs (fairness)

7. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 7 Terminology q Response Variable: Outcome. Eg: response time q Factors: Variables that affect the response variable q Eg: buffer size, RED parameters q Levels: The values that a factor can assume q Eg: TCP type has 2 levels: SACK or Reno q Primary Factors: The “interesting” factors whose effects need to be quantified. q Secondary Factors: Factors whose impact need not be quantified q Eg: We have excluded minor factors like receiver window size, or delay ack (whose effects we roughly know and/or consider minor). Some workload parameters also are secondary factors q Replication: Repetition of all or some experiments q Design: The number of experiments, the factor level, and number of replications for each experiment q Eg: Full Factorial Design with replications q Interaction: Effect of one factor depends upon other factors!

8. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 8 Experiment Design Problem q A system with k parameters and n i level for parameter x i and a performance metric f q f=s(x 1, x 2,…,x k ) can be evaluated by an experiment, I.e. simulation or measurement or analytical formula q Find the effect of each parameter or parameter interaction on the system performance, the best parameter combination, etc. System Parameters or Factors Metrics

9. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 9 Simple Experiment Design q Start with a typical configuration q Vary one parameter at a time q Number of experiments: q Disadvantage: only good for the system with no interaction between parameters q Wrong conclusions if the factors have interaction q Not statistically efficient

10. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 10 Full factorial Design q Try every possible combination of all the parameters. q Number of experiments q Disadvantage: too many experiments

11. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 11 Reduce experiment number?  Reduce the number of levels for each parameter, e.g., reduce n i to 2 levels for each parameter  Reduce the number of parameters  Use fractional factorial design: q Try a subset of the all possible parameter combinations  Less information  May not get all interactions  Not a problem if negligible interactions

12. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 12 2 k Full Factorial Design  Using a nonlinear regression model (assuming 2 parameters, x 1, x 2 ) y = q 0 + q 1 x 1 + q 2 x 2 + q 3 x 3 + q 12 x 1 x 2 + q 23 x 2 x 3 + q 13 x 1 x 3 + q 123 x 1 x 2 x 3 q Goal: Run 2 k experiment (e.g., take the two extreme values of each parameter) and solve the above equation for the regression parameters: q 0, q 1, q 2, q 12, q 23, q 13, q 123. q Disadvantage: only good for systems where effects of parameters or interactions are monotonous q i.e., the system has to be consistent with the model

13. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 13 2 k-p Fractional Factorial Design q Use a simplified regression model, ignore some interactions (especially high-order interactions since their effect are usually small) q For example, 2 3-1 design: y=q 0 + q 1 x 1 + q 2 x 2 + q 3 x 3 All interactions are ignored, only 4 unknowns, only 2 3-1 experiments are need to solve them

14. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 14 Simple Full Factorial Problem Qn: What is the effect of memory & cache on workstation performance ?

15. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 15 Underlying Regression Model q Interpretation: Mean performance = 40 MIPS q Effect of memory = 20 MIPS q Effect of cache = 10 MIPS q Interaction between memory and cache = 5 MIPS

16. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 16 Computation of Effects (I.e. coeffs)

17. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 17 Computation of Effects (cont’d) q Note: effects (q j ) are linear combinations of responses (y i ) q Sum of the coefficients is zero: aka “contrasts” q Note that the above equations can be expressed in terms of column A, B etc ! => Sign table method!

18. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 18 Sign Table Method q Each column multiplies the responses (y i ) ; q Sum the multiples, and q Divide the sums by 2 k

19. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 19 Allocation of Variation q A.k.a. which factors and alternatives matter! q Importance of a factor: proportion of variation explained by that factor q Note: variation is not variance!

20. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 20 Allocation of Variation: 2 2 design q Fractions explained by B and interaction between A & B can be obtained similarly

21. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 21 Allocation of Variation: Example 2 2 design Note: A is most impt factor!

22. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 22 General 2 k Factorial Designs EXAMPLE

23. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 23 2 k Factorial Design Example

24. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 24 Allocation of Variation q Factor C (I.e. number of processors) is the most important (accounts for 71% of performance variation!)

25. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 25 Exercise

26. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 26 Experiment Designer Package q Run 2 k full factorial or 2 (k-p) fractional factorial design q Executable: edesign q Experiment results are in designer.res

27. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 27 Example Configuration File q ExpCountLimit 100 q Experiment ns my.script q LogOn y q Designer Factorial

28. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 28 Example Configuration File q [parameters] q #namemin max default step is_int? to be tuned? q x1-2 20 0.5ny q x2-2 2-1 0.5ny q [/parameters]

29. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 29 Extra Slides

30. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 30 Example y=(q 0 + q 123 x 1 x 2 x 3 ) + (q 1 x 1 + q 23 x 2 x 3 ) + (q 2 x 2 + q 13 x 1 x 3 ) + (q 3 x 3 + q 12 x 1 x 2 ) Let : Then is called confounding of effects

31. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 31 Design with Sign Table q First create a 2 3 sign table, replace x 12 with x 3, x1x1 +1+1 x2x2 +1 x 3 (x 1 x 2 )+1 +1