1. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 1 Introduction to Experiment Design Shiv Kalyanaraman Rensselaer Polytechnic Institute shivkuma@ecse.rpi.edu http://www.ecse.rpi.edu/Homepages/shivkuma Refs: Chap 16-17 of Raj Jain’s book Adapted from Prof. Raj Jain’s slides

2. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 2 Why Experiment design? q Problem: validation and results only as good as your test cases! q How to design a critical set of test cases ? q Idea: Parameterize and use black-box strategy System Parameters or Factors Metrics

3. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 3 Performance, Metrics and Parameters q Performance questions: q Absolute: How fast does computer A run MY program ? q Relative: Is machine A faster than machine B, and if so, how much faster ? q Parameters: factors or inputs q Eg: clock rate, poisson inter-arrivals q Metrics: functions of factors or parameters q Eg: throughput, response time, queue length... q Metric should characterize the design tradeoffs adequately q Metrics are usually functions of many factors. Use of one factor alone may be misleading.

4. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 4 Choose metrics to reflect appropriate tradeoffs q Network users: services and performance that their applications need, e.g., guarantee that each message it sends will be delivered without error within a certain amount of time q Network designers: cost-effective design e.g., that network resources are efficiently utilized and fairly allocated to different users q Users + designer perspectives enough to meet 3 factors of interface/architecture design. But... q Network providers: system that is easy to administer and manage e.g., that faults can be easily isolated and it is easy to account for usage q Require management tools and interfaces. q Often considered to the basic protocol interface design

5. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 5 Goals of Experiment Design q Design a proper set of (simulation or measurement) experiments: maximize information gained with minimum experiments! q Develop a (regression) model that best describes the data obtained q Estimate the contribution of each factor and its values (I.e alternatives) to the performance variation q Isolate measurement errors q Estimate confidence intervals for model parameters q Check if the alternatives are significantly different q Check if the model is adequate

6. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 6 Example: TCP/AQM Design Problem q TCP versions: Reno or SACK q Max Segment sizes: 100 B vs 1000 B q Buffer Size: 10 pkts vs 100 pkts q AQM: Drop tail vs RED vs REM q AQM parameters … q Workload: FTP q Configuration parameters: 3 flows vs 10 flows q Metrics: total throughput (efficiency), C.o.V of per-flow throughputs (fairness)

7. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 7 Terminology q Response Variable: Outcome. Eg: response time q Factors: Variables that affect the response variable q Eg: buffer size, RED parameters q Levels: The values that a factor can assume q Eg: TCP type has 2 levels: SACK or Reno q Primary Factors: The “interesting” factors whose effects need to be quantified. q Secondary Factors: Factors whose impact need not be quantified q Eg: We have excluded minor factors like receiver window size, or delay ack (whose effects we roughly know and/or consider minor). Some workload parameters also are secondary factors q Replication: Repetition of all or some experiments q Design: The number of experiments, the factor level, and number of replications for each experiment q Eg: Full Factorial Design with replications q Interaction: Effect of one factor depends upon other factors!

8. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 8 Experiment Design Problem q A system with k parameters and n i level for parameter x i and a performance metric f q f=s(x 1, x 2,…,x k ) can be evaluated by an experiment, I.e. simulation or measurement or analytical formula q Find the effect of each parameter or parameter interaction on the system performance, the best parameter combination, etc. System Parameters or Factors Metrics

9. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 9 Simple Experiment Design q Start with a typical configuration q Vary one parameter at a time q Number of experiments: q Disadvantage: only good for the system with no interaction between parameters q Wrong conclusions if the factors have interaction q Not statistically efficient

10. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 10 Full factorial Design q Try every possible combination of all the parameters. q Number of experiments q Disadvantage: too many experiments

11. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 11 Reduce experiment number?  Reduce the number of levels for each parameter, e.g., reduce n i to 2 levels for each parameter  Reduce the number of parameters  Use fractional factorial design: q Try a subset of the all possible parameter combinations  Less information  May not get all interactions  Not a problem if negligible interactions

12. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 12 2 k Full Factorial Design  Using a nonlinear regression model (assuming 2 parameters, x 1, x 2 ) y = q 0 + q 1 x 1 + q 2 x 2 + q 3 x 3 + q 12 x 1 x 2 + q 23 x 2 x 3 + q 13 x 1 x 3 + q 123 x 1 x 2 x 3 q Goal: Run 2 k experiment (e.g., take the two extreme values of each parameter) and solve the above equation for the regression parameters: q 0, q 1, q 2, q 12, q 23, q 13, q 123. q Disadvantage: only good for systems where effects of parameters or interactions are monotonous q i.e., the system has to be consistent with the model

13. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 13 2 k-p Fractional Factorial Design q Use a simplified regression model, ignore some interactions (especially high-order interactions since their effect are usually small) q For example, 2 3-1 design: y=q 0 + q 1 x 1 + q 2 x 2 + q 3 x 3 All interactions are ignored, only 4 unknowns, only 2 3-1 experiments are need to solve them

14. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 14 Simple Full Factorial Problem Qn: What is the effect of memory & cache on workstation performance ?

15. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 15 Underlying Regression Model q Interpretation: Mean performance = 40 MIPS q Effect of memory = 20 MIPS q Effect of cache = 10 MIPS q Interaction between memory and cache = 5 MIPS

16. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 16 Computation of Effects (I.e. coeffs)

17. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 17 Computation of Effects (cont’d) q Note: effects (q j ) are linear combinations of responses (y i ) q Sum of the coefficients is zero: aka “contrasts” q Note that the above equations can be expressed in terms of column A, B etc ! => Sign table method!

18. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 18 Sign Table Method q Each column multiplies the responses (y i ) ; q Sum the multiples, and q Divide the sums by 2 k

19. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 19 Allocation of Variation q A.k.a. which factors and alternatives matter! q Importance of a factor: proportion of variation explained by that factor q Note: variation is not variance!

20. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 20 Allocation of Variation: 2 2 design q Fractions explained by B and interaction between A & B can be obtained similarly

21. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 21 Allocation of Variation: Example 2 2 design Note: A is most impt factor!

22. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 22 General 2 k Factorial Designs EXAMPLE

23. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 23 2 k Factorial Design Example

24. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 24 Allocation of Variation q Factor C (I.e. number of processors) is the most important (accounts for 71% of performance variation!)

25. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 25 Exercise

26. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 26 Experiment Designer Package q Run 2 k full factorial or 2 (k-p) fractional factorial design q Executable: edesign –s q Experiment results are in designer.res edesign Configuration File Experiment Script Designer.res

27. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 27 Example Configuration File ScriptFile my_script.tcl LogOn y Designer Factorial P0 [parameters] #namemin max x1-2 2 x2-2 2 [/parameters]

28. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 28 Example Configuration File Farmer localhost:6666:farmer:any Worker localhost:worker [deployer]./get_effects.pl designer.res [/deployer] edesignfarmer worker … E E E R R R

29. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 29 Example Experiment Script #SETPARAMETERS return_result [expr $x1*$x1+$x2*$x2] edesign Original Experiment Script Generate a set of parameter values Experiment Script Farmer-worker System Return experiment result

30. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 30 Example Output(design.res) q Format of designer.res 1 st parameter, 2 nd parameter,…, metric q Example -1,-1,-1,-1,-0.999 1,-1,-1,-1,1.001 -1,1,-1,-1,1.001 q get-effects.pl will process this file and output the effect and variation percentage of each parameter (interaction).

31. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 31 Example get-effects.pl Output The parameter index represents: 1 x1 2 x2 Parameter(s) Effect Variation Percentage ---------------------------------------------------------------- none 40 1 20 76.19% 2 10 19.05% 1 2 5 4.76% --------------------------------------------------------

32. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 32 What you need to do? q Finish the tcl script q Open a terminal window, run start_system.sh q Open another terminal windows, run: edesign –s ex_ff.conf q Sit and wait to see the output of get-effects.pl q Any problem, ask Yong or me.