Introduction to Benchmarking (II): The current situation in other imaging communities Introduction to Benchmarking (II): The current situation in other imaging communities

The term benchmarking was first used by cobblers to measure people's feet for shoes. They would place someone's foot on a "bench" and mark it out to make the pattern for the shoes. Benchmarking is used to measure performance using a specific indicator (cost per unit of measure, productivity per unit of measure, cycle time of x per unit of measure or defects per unit of measure) resulting in a metric of performance that is then compared to others. cobblersindicatorcobblersindicator

General View Of Benchmarking Computing Performance Bioimaging Bioinformatics Data Mining Business 3DEM

What to Meassure? Performance Quality & Time & Cost Results

CHALLENGE PUBLIC DATASET

BIOIMAGING

PATTERN RECOGNITION & DATA MINING & COMPUTATIONAL LEARNING

TBD: 1. neural information processing systems (NIPS) 2. BCI competitions

NEUROSCIENCE

COMPUTATIONAL/MOLECULAR BIOLOGY

PUBLIC DATASET

In the nut shell

It aims to establish the state-of-the-art in microarray data analysis, as well as identify progress and highlight promising directions for future efforts. CAMDA

Every year, thousands of papers are published that describe new algorithms to be applied to medical images. But very few provide a fair and direct comparison of the newly proposed algorithm with the state of the art. We believe that such comparisons can help to substantially improve the performance of medical image analysis algorithms. Grand Challenge in Medical Imaging Analysis

Its purposes are: To encourage research on algorithms that scale to commercial sizes To provide a reference dataset for evaluating research […] Million Song Dataset

To exploit the competitive nature of many researchers and drive progress forwards at a faster pace. PASCAL

KEY POINTS FOR A CHALLENGE To engage the community To generate or collect some datasets To identify the tasks under evaluation To define the figures of merit To plan a schedule

Precedents in 3DEM