Network of Excellence Benchmark Carlos Oscar Sorzano Techn. Director I 2 PC Natl. Center Biotechnology (CSIC)

NoE Benchmark

There was a defined task. The community accepted the challenge. The result of the challenge could have served as a starting point to improve 3D angular orientation.

NoE Benchmark The challenge did not arise from the community: although the community accepted it.

NoE Benchmark Evaluation was a black box: the evaluation Figure of Merit was unknown a priori.

NoE Benchmark There was not a clear interchange standard: there was no way of testing that the submitted results where actually in the correct format, coordinate system, Euler signs, …

NoE Benchmark There was a single dataset: there was no exploration of different packages performing better under different circumstances.

NoE Benchmark Egos were too high

NoE Benchmark It was a single shot. 1891:10.8s 1906:10.6s 1911:10.5s 1924:10.4s 1930:10.3s 1936:10.2s … 2009:9.58s

For future benchmarks Well defined task. Well defined interchange format. Task proposed by the community. Different datasets. Transparent evaluation with pre-test datasets. Acceptance of competition as a healthy way to improve. Possibility to withdraw from competition. Competitions should be repeated over time.