Design and Analysis of Experiments for Research Projects Rong Zheng

Types of Experiments Simulation – Numerical, event-driven, flow-level, etc. Emulation (Testbed) experiments Which one to pick?

Deciding factors Nature of the project: – E.g., Energy-efficient activity tracking using mobile phones – E.g, Influence maximization on online social networks – E.g., structure health monitoring

An example of hybrid approaches Physical Environments Sensing Actuation Signal Processing Communication Sensor node RealAbstracted Our solution

Deciding factor Scale, controllability, repeatability and fidelity – Simulation wins on the first 3 – Test experiments win on fidelity (but not always) – Emulation comes in between However, if you are doing systems research, the expectation is to have testbed experiments complimented by simulations

When do you design the experiments From the onset of the project! – What datasets? – What hardware/software? – Which user group? – What metrics? – What to compare against?

Analyzing the results Know what to expect – Don’t bring completely out-of-the-line results to your advisor Understand what you didn’t expect – Verify the implementation is correct Unit test (isolation), sanity check – Find an explanation In CS, there is always an explanation to everything When you are done with the experiments, you are still far from being done Unexpected results can be quite useful in improving your solutions

An Example GNU Radio Signal Generator GNU Radio Signal Generator USRP2 Pre - Amp Trans Amp Concrete Channel Oscilloscope USRP Daughterboard USRP Daughterboard File Sink 440mVpp4.161Vpp238.05Vpp 1.081Vpp 2.5mVpp Gain = dB

Presenting your results Statistics – Please give me error bars or confidence intervals – Is the observation really statistically significant? Obama vs Romeney Gullop poll 47% vs 49% – Both sides of the story: false positive and false negative Experimental settings please! 3D graphs should be banned Up and down curves a bad sign Colors on black-white printout? Fonts? – Embedded font issues Tell what you observe and provide insight gained – Don’t just run out of stream when you reach the evaluation section of your paper