1. Introduction to Experiment Design
Refs: Chap of Raj Jain’s book
Shiv Kalyanaraman
Rensselaer Polytechnic Institute
Adapted from Prof. Raj Jain’s slides

2. Recall: Iterative Design Process
Measure,
simulate,
experiment
Model,
Hypothesis,
Predictions
How to make
this empirical
design process
EFFICIENT??
How to avoid
pitfalls in
inference!

3. Recall: System-Under-Test (SUT) Model
Parameters
System-
Under-
Test
Metrics
Factors
Subject system to a set of tests (workloads/conditions)

4. Recall: Performance tradeoffs ?
A situation where you cannot get something
for nothing!
Also known as a zero-sum game.
R=link bandwidth (bps)
L=packet length (bits)
a=average packet arrival rate
Traffic intensity = La/R

5. What’s a performance tradeoff ?
La/R ~ 0: average queuing delay small
La/R -> 1: delays become large
La/R > 1: average delay infinite (service degrades unboundedly => instability)!
Summary: Multiplexing using bus topologies has both direct resource costs and intangible costs like potential instability, buffer/queuing delay.

6. Recall: Amdahl’s Law http://en.wikipedia.org/wiki/Amdahl's_law
Guides the iterative design process.
If design implies a set of tradeoffs, the question is how to redesign components so that the system cost-performance (in terms of expensive resources) is improved.
Amdahl’s law talks about the maximum expected improvement to an overall system when only a part of the system is improved.
Statement of “diminishing returns” when the focus is on improving only a single component
System Speedup =

7. Amdahl’s Law (contd)
If a part of a system accounts for 12% of performance (P = 0.12) and
You speed it up 100-fold (S = 100)
The actual system speedup is only: 13.6% !!!!
Lesson #1: Optimize the common cases (accounting for a large fraction of system performance)
Lesson #2: Bottlenecks shift! If you optimize one component, another will become the new bottleneck!

8. Why Experiment design? Parameters Metrics or System Factors
Problem: How to know what’s the “common” case to apply Amdahl’s law?
Problem: validation and results only as good as your test cases!
How to design a critical set of test cases ?
Idea: Parameterize and use black-box strategy
Parameters or
Factors
Metrics
System

9. Goal of Experiment Design
How to choose experiments such that we get the maximum information with the minimum number of experiments?

10. Expt Design

11. Performance, Metrics and Parameters
Performance questions:
Absolute: How fast does computer A run MY program ?
Relative: Is machine A faster than machine B, and if so, how much faster ?
Parameters: factors or inputs
Eg: clock rate, poisson inter-arrivals
Metrics: functions of factors or parameters
Eg: throughput, response time, queue length ...
Metric should characterize the design tradeoffs adequately
Metrics are usually functions of many factors. Use of one factor alone may be misleading.

12. Choose metrics to reflect appropriate tradeoffs
Network users: services and performance that their applications need, e.g., guarantee that each message it sends will be delivered without error within a certain amount of time
Network designers: cost-effective design e.g., that network resources are efficiently utilized and fairly allocated to different users
Users + designer perspectives enough to meet 3 factors of interface/architecture design. But ...
Network providers: system that is easy to administer and manage e.g., that faults can be easily isolated and it is easy to account for usage
Require management tools and interfaces.
Often considered to the basic protocol interface design

13. Goals of Experiment Design
Design a proper set of (simulation or measurement) experiments: maximize information gained with minimum experiments!
Develop a (regression) model that best describes the data obtained
Estimate the contribution of each factor and its values (I.e alternatives) to the performance variation
Isolate measurement errors
Estimate confidence intervals for model parameters
Check if the alternatives are significantly different
Check if the model is adequate

14. Example: TCP/AQM Design Problem
TCP versions: Reno or SACK
Max Segment sizes: 100 B vs 1000 B
Buffer Size: 10 pkts vs 100 pkts
AQM: Drop tail vs RED vs REM
AQM parameters …
Workload: FTP
Configuration parameters: 3 flows vs 10 flows
Metrics: total throughput (efficiency), C.o.V of per-flow throughputs (fairness)

15. Research Example: Online Network Parameter Tuning

16. Example (contd) Experiment Design Network Abstraction
Link, simulated at packet level
Experiment Design
Performance
Parameter 1
P1min
P1max
Parameter 2
Current operating point
Trial operating point triggering
simulation
All three trial points can be
Evaluated concurrently
Network Abstraction
And Decomposition
Parallel Discrete
Event Simulation
Domain 1
Domain 2
Domain 3
Models of Inter-domain flows
Processor 1
Processor 2
Processor 3
Processor 4
router
Links crossing processor
Boundary may cause rollback

17. Example (contd) Note: linear models don’t fit this response surface!
The goal of experiment design is more modest here: to
find better local minima with fewer experiments
(optimization, not modeling/curve-fitting)

18. Terminology Response Variable: Outcome. Eg: response time
Factors: Variables that affect the response variable
Eg: buffer size, RED parameters
Levels: The values that a factor can assume
Eg: TCP type has 2 levels: SACK or Reno
Primary Factors: The “interesting” factors whose effects need to be quantified.
Secondary Factors: Factors whose impact need not be quantified
Eg: We have excluded minor factors like receiver window size, or delay ack (whose effects we roughly know and/or consider minor). Some workload parameters also are secondary factors
Replication: Repetition of all or some experiments
Design: The number of experiments, the factor level, and number of replications for each experiment
Eg: Full Factorial Design with replications
Interaction: Effect of one factor depends upon other factors!

19. Experiment Design Problem
A system with:
k parameters
ni levels for parameter xi and
a performance metric f
f=s(x1, x2,…,xk) can be evaluated by an experiment, I.e. simulation or measurement or analytical formula
Find the effect of each parameter or parameter interactions on the system performance, the best parameter combination, etc.
Parameters or
Factors
Metrics
System

20. Expt Design Problem (Contd)
The analysis tells us what proportion of the variation is explained by each regression factor
Note: even parameter interactions are separate “factors” in the regression.
I.e.: How sensitive is the system performance to changes in the factor?
This information, combined with Amdahl’s law, tells us where to focus our design improvements
Parameter interactions, if significant, are harder to fix (assuming the design provides direct hooks only to each parameter).

21. Simple Experiment Design
Start with a typical configuration
Vary one parameter at a time
Number of experiments:
Disadvantage: only good for the system with no interaction between parameters
Wrong conclusions if the factors have interaction
Not statistically efficient

22. Full factorial Design
Try every possible combination of all the parameters.
Number of experiments
Disadvantage: too many experiments

23. Reduce experiment number?
Reduce the number of levels for each parameter, e.g., reduce ni to 2 levels for each parameter
Reduce the number of parameters
Use fractional factorial design:
Try a subset of the all possible parameter combinations
Less information
May not get all interactions
Not a problem if negligible interactions

24. 2k Full Factorial Design
Using a nonlinear regression model (assuming 3 parameters, x1, x2, x3).
Note: linear model if interactions are also considered separate “factors”
y = q0 + q1x1+ q2x2 + q3x3 +
q12x1x2 + q23x2x3 + q13x1x3 +
q123x1x2x3
Goal: Run 2k experiment (e.g., take the two extreme values of each parameter) and solve the above equation for the regression parameters: q0, q1, q2, q12, q23, q13, q123.
Disadvantage: only good for systems where effects of parameters or interactions are monotonous
i.e., the system has to be consistent with the model

25. 2k-p Fractional Factorial Design
Use a simplified regression model, ignore some interactions (especially high-order interactions since their effect are usually small)
For example, 23-1 design:
y=q0 + q1x1+ q2x2 + q3x3
All interactions are ignored, only 4 unknowns, only 23-1 experiments are need to solve them

26. Simple Full Factorial Problem
Metric
Factors
Qn: What is the effect of memory & cache
on workstation performance (in MIPS)?

27. Underlying Regression Model
Plug in
xa, xb
values
Original model
after solving for
the qs…
Interpretation: Mean performance = 40 MIPS
Effect of memory = 20 MIPS
Effect of cache = 10 MIPS
Interaction between memory and cache = 5 MIPS

28. Computation of Effects (I.e. coeffs)
Note: yi

29. Computation of Effects (cont’d)
Note: effects (qj) are linear combinations of responses (yi)
Sum of the coefficients is zero: aka “contrasts”
Note that the above equations can be expressed in terms of column A, B etc !
=> Sign table method!
Note: qi

30. Sign Table Method Each column multiplies the responses (yi) ;
Sum the multiples, and
Divide the sums by 2k

31. Allocation of Variation
A.k.a. which factors and alternatives matter!
Importance of a factor: proportion of variation explained by that factor
Note: variation is not variance!

32. Allocation of Variation: 22 design
Fractions explained by B and interaction between A & B can be obtained similarly

33. Allocation of Variation: Example 22 design
Note: A is
most impt
factor!

34. General 2k Factorial Designs
EXAMPLE

35. 2k Factorial Design Example

36. Allocation of Variation
Factor C (I.e. number of processors) is the most important (accounts for 71% of performance variation!)

37. Exercise

38. Common Mistakes

39. Remember: Assumptions!

40. Visual Tests

41. Visual Tests (contd)

42. Non-linear regression: Make linear through transformation of samples!

44. Finally Remember! Even Regression has Confidence Intervals
Note:
1. As we move farther from the mean, the bands get wider.
2. The prediction interval bands are wider. Why? (extra Syx)
124

45. Experiment Designer Package
Run 2k full factorial or 2(k-p) fractional factorial design
Executable: edesign –s <configuration_file>
Experiment results are in designer.res
Configuration
File
edesign
Designer.res
Experiment
Script

46. Example Configuration File
ScriptFile my_script.tcl
LogOn y
Designer Factorial
P 0
[parameters]
#name min max x x
[/parameters]

47. Example Configuration File
Farmer localhost:6666:farmer:any
Worker localhost:worker
[deployer]
./get_effects.pl designer.res
[/deployer]
worker E E R …
edesign
farmer R
worker R E

48. Example Experiment Script
#SETPARAMETERS
return_result [expr $x1*$x1+$x2*$x2]
edesign
Return experiment result
Generate a set of parameter values
Original
Experiment
Script
Farmer-worker
System
Experiment Script

49. Example Output(design.res)
Format of designer.res
1st parameter, 2nd parameter,…, metric
Example
-1,-1,-1,-1,-0.999
1,-1,-1,-1,1.001
-1,1,-1,-1,1.001
get-effects.pl will process this file and output the effect and variation percentage of each parameter (interaction).

50. Example get-effects.pl Output
The parameter index represents: x1 x2
Parameter(s) Effect Variation Percentage
none % % %

51. What you need to do? Finish the tcl script
Open a terminal window, run start_system.sh (don’t repeat even you need to do multiple edesign )
Open another terminal windows, run:
edesign –s ex_ff.conf
Sit and wait to see the output of get-effects.pl
Any problem, ask the in-class TA.

52. Extra Slides

53. Experiment Designer Package
Run 2k full factorial or 2(k-p) fractional factorial design
Executable: edesign <configuration_file>
Experiment results are in designer.res

54. Example Configuration File
ExpCountLimit 100
Experiment ns my.script
LogOn y
Designer Factorial

55. Example Configuration File
[parameters]
#name min max default step is_int? to be tuned?
x n y
x n y
[/parameters]

56. Example
y=(q0 + q123x1x2x3) + (q1x1 + q23x2x3) + (q2x2 + q13x1x3) + (q3x3 + q12x1x2)
Let :
Then
is called confounding of effects

57. Design with Sign Table
First create a 23 sign table, replace x12 with x3, x1 -1 +1 x2
x3(x1x2)