1. Network of Excellence Benchmark Carlos Oscar Sorzano Techn. Director I 2 PC Natl. Center Biotechnology (CSIC)

2. NoE Benchmark

3. There was a defined task. The community accepted the challenge. The result of the challenge could have served as a starting point to improve 3D angular orientation.

4. NoE Benchmark The challenge did not arise from the community: although the community accepted it.

5. NoE Benchmark Evaluation was a black box: the evaluation Figure of Merit was unknown a priori.

6. NoE Benchmark There was not a clear interchange standard: there was no way of testing that the submitted results where actually in the correct format, coordinate system, Euler signs, …

7. NoE Benchmark There was a single dataset: there was no exploration of different packages performing better under different circumstances.

8. NoE Benchmark Egos were too high

9. NoE Benchmark It was a single shot. 1891:10.8s 1906:10.6s 1911:10.5s 1924:10.4s 1930:10.3s 1936:10.2s … 2009:9.58s

10. For future benchmarks Well defined task. Well defined interchange format. Task proposed by the community. Different datasets. Transparent evaluation with pre-test datasets. Acceptance of competition as a healthy way to improve. Possibility to withdraw from competition. Competitions should be repeated over time.