1. NA-MIC National Alliance for Medical Image Computing http://na-mic.org Evaluating Brain Tissue Classifiers S. Bouix, M. Martin-Fernandez, L. Ungar, M. Nakamura, M.-S. Koo, R.W. McCarley, M.E. Shenton.

2. National Alliance for Medical Image Computing http://na-mic.org Motivation Classification of brain tissue into Cortical Spinal Fluid (CSF), White Matter (WM) and Gray Matter (GM) is an essential step for most brain imaging study. There are a multitude of automated methods to do this task, but how can we assess their performance? If no reference segmentation is given, can we still evaluate these techniques?

3. National Alliance for Medical Image Computing http://na-mic.org Comparing two binary images Let L be a grid of n spatial sites x and I 1 and I 2 be two binary images over this grid. Similarity measures: –Jaccard: JC= –Simple Matching: SC= –Volume Similarity: VS=

4. National Alliance for Medical Image Computing http://na-mic.org STAPLE What if there is no ground truth? Estimate a ground truth (STAPLE) –For each rater i, label l, let P(X i =l|T=l). –Use an EM framework in which the hidden data is the ground truth and the parameters are the set of all P(X i =l|T=l). –Gives an estimation of the quality of the P(X i =l|T=l) and an estimated ground truth (EGT). Use the EGT as the ground truth and perform the “Automatic vs. Ground Truth” analysis.

5. National Alliance for Medical Image Computing http://na-mic.org Williams’ Index Rate by level of agreement (Williams’ Index) For a total of r raters, Williams’ Index of rater j is defined as: If I j >1, then rater j agrees with all other raters as much as they agree with each other.

6. National Alliance for Medical Image Computing http://na-mic.org Segmentation Pipeline, Oversimplified Filtering Bias Field Correction Brain Stripping Tissue Classification

7. National Alliance for Medical Image Computing http://na-mic.org Methods Tested kNN: k Nearest Neighbor classifier trained by non linear atlas registration MINC: artificial neural network classifier, trained through affine atlas registration FSL: EM algorithm with Hidden Markov Model SPM EM: original EM segmenter EMAtlas: EM, Markov Model and non linear atlas Watershed: watershed with non linear atlas

8. National Alliance for Medical Image Computing http://na-mic.org Pre/Post Processing Filtering: Krissian’s anisotropic filter. Bias correction: EM’s output. Brain Stripping: Brain Extraction Tool.

9. National Alliance for Medical Image Computing http://na-mic.org Data Set 40 female subjects 2 MR pulse sequences on a GE 1.5T scanner: –SPGR: 0.9375x0.9375x1.5mm, Coronal PA. –Double-Echo, Spin-Echo: two volumes (PD and T2W) 0.9375x0.9375x3mm, Axial IS. T2W was automatically aligned to the SPGR using a Mutual Information rigid registration technique. SPGR/T2W aligned pair was used as input. PD was not used.

10. National Alliance for Medical Image Computing http://na-mic.org Visualization

11. National Alliance for Medical Image Computing http://na-mic.org Multidimensional Scaling Multidimensional scaling (MDS) is a data exploration technique that represents measurements of similarity among pairs of objects as distances between points of a low-dimensional space.

12. National Alliance for Medical Image Computing http://na-mic.org Visualization MDS

13. National Alliance for Medical Image Computing http://na-mic.org Conclusions If no ground truth is available, it is possible to compare automatic segmentation technique. If the methods are believed to be different enough then their level of agreement is a good measure of performance. Williams index is a nice alternative to STAPLE if no estimated ground truth is needed.

14. National Alliance for Medical Image Computing http://na-mic.org Concerns If most classifiers make the same mistake then the mistake is believed to be a correct classification. You cannot evaluate something that you believe is very significantly better! Presenting the results in a compact yet meaningful way is becoming a research problem in itself.

15. National Alliance for Medical Image Computing http://na-mic.org Future work for NAMIC Create a public benchmark data set Create a public evaluation pipeline Create a web-based dashboard for segmentation.

16. National Alliance for Medical Image Computing http://na-mic.org Automatic vs. partial ground truth. For 20 subjects, 3 raters manually segmented subsets of the volume (50x50 squares in 4 different locations, excluding cerebellum and basal ganglia). STAPLE was used to get the EGT based on the manual segmentations. Each automatic segmenter was compared to EGT using JC, SC and VD.

17. National Alliance for Medical Image Computing http://na-mic.org Automatic vs. Ground Truth For each each multilabel segmentation: –Create a binary image per label I l. –Compute JC( I l,GT l ); TN(I l,GT l ); VS(I l,GT l ). –Analyze the results with your favorite plots.