1. Workshop: Experimental research in practice
Roland Geraerts
16 November 2016
Picture:
Please send comments to

2. Bad repeatability
Why can it be hard to reproduce papers’ claims?

3. Bad repeatability (1) Problem Cause
The results cannot be reproduced easily
Cause
Details of the method are lacking
Parts of the method are not described
Degenerate cases are missing
References to other papers (without mentioning details)
Parameters don’t get assigned values (usually weights)
Source code is not available
The experimental setup is not clear
Tested hardware (e.g. which PC/GPU, the number of cores used)
Statistical setup (e.g. number of runs, seed)
Details of the scenario(s) are missing
Picture:

4. Bad repeatability (2) Problem Cause Solution
The results cannot be reproduced easily
Cause
Low significance caused by a low number of runs
Hard problems can be hard to implement
Solution
Let someone else implement the method/paper
Provide the source code
Picture:

5. Data collection errors
What kind of errors occur during the collection of (raw) data?
Picture:

6. Data collection errors (1)
Problem
Errors occur during collection of raw data
E.g., copy/paste values from GUIs into excel sheets or text files
Cause
The data collection process was not automated
There is a GUI but not a command line (console) version
Variables aren’t assigned the right values (how to verify?)
The precision of the stored numbers is too low
Statistics are computed wrongly (e.g. how to compute the SD)
Only the execution of a part of the algorithm is recorded
The visualization part is not strictly separated from the execution part of the algorithm
E.g. While the method performs its computations, the results are being written to a log file and sent to the GPU for visualization purposes
Standard deviation: Must we divide by n or n-1?
Partial results:

7. Data collection errors (2)
Solution
Automate the process using a console called from a batch file
For small experiments, call the arguments in the batch file
Otherwise, build a load/save mechanism
Create an API that supports setting up experiments
Standard deviation: Must we divide by n or n-1?
Partial results:

8. Data collection errors (3): Time measurement errors
Problem
Time is measured wrongly
Cause
Lack of timer’s accuracy
C++: Don’t use time.h
Don’t start/stop the timer inside the method, especially not if the parts take less than 1 ms to compute
Intervening network/CPU/GPU processes

9. Data collection errors (4): Time measurement errors
Solution
Use accurate timers
C++: Use QueryPerformanceCounter(…) instead; be careful of 0.3s jumps, or C++ 11: std::chrono::high_resolution_clock
Run fast methods many times and take the average; watch out for non-deterministic behavior
Take the average of some runs, also in case of deterministic algorithms
Only measure the running time of the algorithm
Switch off the network
Kill the virus killer
Stop the program
Disable update functionality
Use only 1 core
Don’t work on your thesis while running the experiments on the same machine; and yes, this happens

10. Bad figures When do figures convey information badly?
Figure 7. Motion planning result considering the effect of currents.

11. Bad figures Problem Cause Solution
The figures convey information badly
Cause
The figures are hard to read (e.g. too small or bitmapped)
Axes haven’t been labeled
The y-axis doesn’t start at 0 which amplifies (random) differences
Use the right number precision/format
Don’t display 100,
Don’t display s, or …
The meaning is not conveyed clearly
Some colors/patterns don’t do well on black & white printers
Solution
Use e.g. GNUplot (set all labels and export to vector: EPS or PDF)
Use vector images as much as possible (e.g. use IPE)
Explain all phenomena
Use vector graphics
Picture:

12. Conclusions are too general
When are drawn conclusions too general?

13. Conclusions are too general (1)
Problem
The conclusions drawn are often too general
Cause
Only one instance is tested, e.g.
environment / moving entity
Only one problem setting is tested
A favorable setup is used, e.g.
a few axis-aligned rectangular obstacles
polygonal convex obstacles
1 fixed query
Deterministic experiments do suffer from the ‘variance problem’
Variance problem: Deterministic versus randomized techniques
Deterministic techniques can respond sensitively to small changes in the problem setting
Even worse, there might be a statistical significance while a better implementation might halve the running times

14. Conclusions are too general (2)
Solution
Try to sample the problem space as good as possible
Don’t try to bias any method
Use a favorable setup (to show certain properties) and a ‘normal’ one
Also choose worst-case scenarios
Tune all methods equally
Compare against the state-of-the-art instead of old methods only
Dare to show the weakness(es) of your method
Variance problem: Deterministic versus randomized techniques
Deterministic techniques can respond sensitively to small changes in the problem setting
Even worse, there might be a statistical significance while a better implementation might halve the running times

15. Statistical weaknesses
When are the statistics less reliable?

16. Statistical weaknesses
Problem
Statistics are done badly
Cause
Results have been collected on different sets of hardware
Too few runs
Not all running times are mentioned (e.g. initialization)
Only averages are mentioned
Solution
Use the same machine (and don’t change the setup)
Use e.g. GNUplot and set all (relevant) labels
Use other measures, e.g. SD
Boxplot
Student’s t-test: statistical hypothesis test
ANOVA: Analysis of variance
Student's t-test, source:
“A t-test is any statistical hypothesis test in which the test statistic follows a Student's t distribution if the null hypothesisis supported. It can be used to determine if two sets of data are significantly different from each other, and is most commonly applied when the test statistic would follow a normal distribution if the value of a scaling term in the test statistic were known. When the scaling term is unknown and is replaced by an estimate based on the data, the test statistic (under certain conditions) follows a Student's t distribution.”
Anova, source:
“Analysis of variance (ANOVA) is a collection of statistical models used to analyze the differences between group means and their associated procedures (such as "variation" among and between groups), developed by R.A. Fisher. In the ANOVA setting, the observed variance in a particular variable is partitioned into components attributable to different sources of variation. In its simplest form, ANOVA provides a statistical test of whether or not the means of several groups are equal, and therefore generalizes the t-test to more than two groups. Doing multiple two-sample t-tests would result in an increased chance of committing a type I error. For this reason, ANOVAs are useful in comparing (testing) three or more means (groups or variables) for statistical significance.”

17. Statistically significant?
Picture:
Please send comments to

18. So your method is statistically significant
While a method was granted being statistically significant, this does not have to mean anything in practice…
…due to the programmer’s bias.
Suppose different methods run in 10.2, 10.0, 10.3, and 9.6 seconds (with appropriate SDs etc). While the latter one might be better, in reality it does not have to be…
…since the third one might be the only one that wasn’t optimized.
Picture:

19. Ways to bias your results (1)
Run the code with choices in of
Hardware (CPU, GPU, memory, cache, #cores, #threads)
Language (C++/C#, 32/64bit, different optimizations)
Software libraries (own code/boost/STL)
Implementation is done by different people

20. Ways to bias your results (2): Some code optimizations
Enable optimizations in your compiler
Run in release mode!
Visual studio
full optimization
inline function expansion
Enable intrinsic functions
Etc.
Compile the code with a 64-bit compiler
2-15% improvement of running times due to
usage of a larger instruction set
Not having to simulate 32-bit code
However, watch code that deals with memory and loops
use memsize-types in address arithmetic
See

21. Ways to bias your results (3): Some code optimizations
Unroll loops
Improves usage of parallel execution (e.g. SSE2)
Create small code
E.g. by improving the implementation; properly align data
Improves cache behavior
Avoid mixed arithmetic
Use STL
Is heavily optimized
Avoid disk usage and writing to a console etc.
Follow the course: Optimization and vectorization

22. Ethics versus mistakes
Let’s have a discussion here!