1. Computational Radiology Laboratory Harvard Medical School www.crl.med.harvard.edu Children’s Hospital Department of Radiology Boston Massachusetts A Survey of Validation Techniques for Image Segmentation and Registration, with a focus on the STAPLE algorithm Simon K. Warfield, Ph.D. Associate Professor of Radiology Harvard Medical School

2. Computational Radiology Laboratory. Slide 2 Outline Validation of image segmentation –Overview of approaches –STAPLE Validation of image registration STAPLE algorithm available as open source software from: –http://www.nitrc.org/projects/staple –http://crl.med.harvard.edu/

3. Computational Radiology Laboratory. Slide 3 Segmentation Goal: identify or label structures present in the image. Many methods: –Interactive or manual delineation, –Supervised approaches with user initialization, –Alignment with a template, –Statistical pattern recognition. Applications: –Quantitative measurement of volume, shape or location of structures, –Provides boundary for visualization by surface rendering. Newborn MRI Segmentation.

4. Computational Radiology Laboratory. Slide 4 Validation of Image Segmentation Spectrum of accuracy versus realism in reference standard. Digital phantoms. –Ground truth known accurately. –Not so realistic. Acquisitions and careful segmentation. –Some uncertainty in ground truth. –More realistic. Autopsy/histopathology. –Addresses pathology directly; resolution. Clinical data ? –Hard to know ground truth. –Most realistic model.

5. Computational Radiology Laboratory. Slide 5 Validation of Image Segmentation Comparison to digital and physical phantoms: –Excellent for testing the anatomy, noise and artifact which is modeled. –Typically lacks range of normal or pathological variability encountered in practice. MRI of brain phantom from Styner et al. IEEE TMI 2000

6. Computational Radiology Laboratory. Slide 6 Comparison To Higher Resolution MRIPhotographMRI Provided by Peter Ratiu and Florin Talos.

7. Computational Radiology Laboratory. Slide 7 Comparison To Higher Resolution PhotographMRI PhotographMicroscopy Provided by Peter Ratiu and Florin Talos.

8. Computational Radiology Laboratory. Slide 8 Comparison to Autopsy Data Neonate gyrification index –Ratio of length of cortical boundary to length of smooth contour enclosing brain surface

9. Computational Radiology Laboratory. Slide 9 Staging Stage 3 Stage 5 Stage 4 Stage 6 Stage 3: at 28 w GA shallow indentations of inf. frontal and sup. Temp. gyrus (1 infant at 30.6 w GA, normal range: 28.6 ± 0.5 w GA) Stage 4: at 30 w GA 2 indentations divide front. lobe into 3 areas, sup. temp.gyrus clearly detectable (3 infants, 30.6 w GA ± 0.4 w, normal range: 29.9 ± 0.3 w GA) Stage 5: at 32 w GA frontal lobe clearly divided into three parts: sup., middle and inf. Frontal gyrus (4 infants, 32.1 w GA ± 0.7 w, normal range: 31.6 ± 0.6 w GA) Stage 6: at 34 w GA temporal lobe clearly divided into 3 parts: sup., middle and inf. temporal gyrus (8 infants, 33.5 w GA ± 0.5 w normal range: 33.8 ± 0.7 w GA) “Assessment of cortical gyrus and sulcus formation using MR images in normal fetuses”, Abe S. et al., Prenatal Diagn 2003

10. Computational Radiology Laboratory. Slide 10 Neonate GI: MRI Vs Autopsy

11. Computational Radiology Laboratory. Slide 11 GI Increase Is Proportional to Change in Age.

12. Computational Radiology Laboratory. Slide 12 GI Versus Qualitative Staging

13. Computational Radiology Laboratory. Slide 13 Neonate Gyrification

14. Computational Radiology Laboratory. Slide 14 Validation of Image Segmentation Comparison to expert performance; to other algorithms. Why compare to experts ? –Experts are currently doing the segmentation tasks that we seek algorithms for. –Surgical planning. –Neuroscience research. What is the appropriate measure for such comparisons ?

15. Computational Radiology Laboratory. Slide 15 Measures of Expert Performance Repeated measures of volume –Intra-class correlation coefficient Spatial overlap –Jaccard: Area of intersection over union. –Dice: increased weight of intersection. –Vote counting: majority rule, etc. Boundary measures –Hausdorff, 95% Hausdorff. Bland-Altman methodology: –Requires a reference standard. Measures of correct classification rate: –Sensitivity, specificity ( Pr(D=1|T=1), Pr(D=0|T=0) ) –Positive predictive value and negative predictive value (posterior probabilities Pr(T=1|D=1), Pr(T=0|D=0) )

16. Computational Radiology Laboratory. Slide 16 Validation of Image Segmentation STAPLE (Simultaneous Truth and Performance Level Estimation): –An algorithm for estimating performance and ground truth from a collection of independent segmentations.

17. Computational Radiology Laboratory. Slide 17 STAPLE papers –Image segmentation with labels: Warfield, Zou, Wells ISBI 2002 Warfield, Zou, Wells MICCAI 2002. Warfield, Zou, Wells, IEEE TMI 2004. Commowick and Warfield IPMI 2009 –Image segmentation with boundaries: Warfield, Zou, Wells MICCAI 2006. Warfield, Zou, Wells PTRSA 2008. –Diffusion data and vector fields: Commowick and Warfield IEEE TMI 2009

18. Computational Radiology Laboratory. Slide 18 STAPLE: Estimation Problem Complete data density: Binary ground truth T i for each voxel i. Expert j makes segmentation decisions D ij. Expert performance characterized by sensitivity p and specificity q. –We observe expert decisions D. If we knew ground truth T, we could construct maximum likelihood estimates for each expert’s sensitivity (true positive fraction) and specificity (true negative fraction):

19. Computational Radiology Laboratory. Slide 19 Expectation-Maximization Since we don’t know ground truth T, treat T as a random variable, and solve for the expert performance parameters that maximize: Parameter values θ j =[p j q j ] T that maximize the conditional expectation of the log-likelihood function are found by iterating two steps: –E-step: Estimate probability of hidden ground truth T given a previous estimate of the expert quality parameters, and take expectation. –M-step: Estimate expert performance parameters by comparing D to the current estimate of T.

20. Computational Radiology Laboratory. Slide 20 Probability Estimate of True Labels Estimate probability of tissue class in reference standard:

21. Computational Radiology Laboratory. Slide 21 Binary Input: True Segmentation

22. Computational Radiology Laboratory. Slide 22 Expert Performance Estimate p (sensitivity, true positive fraction) : ratio of expert identified class 1 to total class 1 in the image. q (specificity, true negative fraction) : ratio of expert identified class 0 to total class 0 in the image.

23. Computational Radiology Laboratory. Slide 23 Newborn MRI Segmentation

24. Computational Radiology Laboratory. Slide 24 Newborn MRI Segmentation Summary of segmentation quality (posterior probability Pr(T=t|D=t) ) for each tissue type for repeated manual segmentations. Indicates limits of accuracy of interactive segmentation.

25. Computational Radiology Laboratory. Slide 25 Expert and Student Segmentations Test imageExpert consensusStudent 1 Student 2Student 3

26. Computational Radiology Laboratory. Slide 26 Phantom Segmentation ImageExpertStudentsVotingSTAPLE ImageExpert segmentation Student segmentations

27. Computational Radiology Laboratory. Slide 27 STAPLE Summary Key advantages of STAPLE: –Estimates ``true’’ segmentation. –Assesses expert performance. Principled mechanism which enables: –Comparison of different experts. –Comparison of algorithm and experts. Extensions for the future: –Prior distribution or extended models for expert performance characteristics. –Estimate bounds on parameters.

28. Computational Radiology Laboratory. Slide 28 Image registration A metric: measures similarity of images given an estimate of the transformation. Best metric depends on nature of the images. Alignment quality ultimately possible depends on model of transformation. The transformation is identified by solving an optimization problem. –Seek the transform parameters that maximize the metric of image similarity

29. Computational Radiology Laboratory. Slide 29 Validation of Registration Compare transformations –Take some images, apply a transformation to them. –Estimate the transform using registration –How well does the estimated transformation match the applied transform? Check alignment of key image features –Fiducial alignment –Spatial overlap Segment structures, assess overlap after alignment.

30. Computational Radiology Laboratory. Slide 30 Intraoperative Nonrigid Registration Fast: it should not take more than 1 min to make the registration. Robust: the registration should work with poor quality image, artifacts, tumor... Physics based: we are not only concerned in the intensity matching, but also interested in recovering the physical (mechanical) deformation of the brain. Accurate: neuro-surgery needs a precise knowledge of the position of the structures. Archip et al. NeuroImage 2007

31. Computational Radiology Laboratory. Slide 31 Block Matching Algorithm Divide a global optimization problem in many simple local ones Highly parallelizable, as blocks can be matched independently. Similarity measure: coefficient of correlation

32. Computational Radiology Laboratory. Slide 32 Block Matching Algorithm Displacement estimates are noisy.

33. Computational Radiology Laboratory. Slide 33 Patient-specific Biomechanical Model Pre-operative image Automatic brain segmentation Brain finite element model (linear elastic)

34. Computational Radiology Laboratory. Slide 34 Registration Validation Landmark matching assessment in six cases Parallel version runs in 35 seconds on a 10 dual 2GHz PC cluster –7x7x7 block size –11x11x25 window –1x1x1 step –50 000 blocks –10 000 tetrahedra 60 landmarks: –Average error = 0.75mm –Maximum error = 2.5mm –Data voxel size 0.8x0.8x2.5 mm 3

35. Computational Radiology Laboratory. Slide 35 Registration Validation 11 prospective consecutive cases, Alignment computed during the surgery. Estimate of the registration accuracy – 95% Hausdorff distance of the edges of the registered preoperative MRI and the intraoperative MRI.

36. Computational Radiology Laboratory. Slide 36 Automatic selection of fiducials (1)Non-rigid alignment of preoperative MPRAGE. Contours extracted from (1) with the Canny edge detector (2) Intraoperative whole brain SPGR at 0.5T Contours extracted from (2) with the Canny edge detector 95% Hausdorff metric computed

37. Computational Radiology Laboratory. Slide 37 Alignment improvement Tumor positionTumor pathology Non-rigid registration – preop to intraop scans (95% Hausdorff distance) Max Displacement measured (mm) Rigid registration accuracy – preop to intraop (mm) Non-Rigid registration accuracy – preop to intraop (mm) Ratio Rigid/Non- Rigid Case 1right posterior frontaloligoastrocytoma Grade II10.685.951.903.13 Case 2left posterior temporalglioblastoma Grade IV21.0310.712.903.69 Case 3left medial temporalglioblastoma Grade IV15.277.651.704.50 Case 4left temporalanaplastic oligoastrocytoma Grade III 10.006.800.858.00 Case 5right frontaloligoastrocytoma Grade II9.875.101.274.01 Case 6left frontalanaplastic astrocytoma Grade III 17.4810.203.572.85 Case 7right medial temporalanaplastic astrocytoma Grade III 19.969.352.553.66 Case 8right frontaloligoastrocytoma Grade II17.448.331.197.00 Case 9right frontotemporaloligoastrocytoma Grade II15.087.141.873.81 Case 10right occipitalanaplastic oligodendroglioma Grade III 9.485.951.444.13 Case 11left frontotemporal oligodendroglioma Grade II 10.744.760.855.60 AVG14.277.441.824.58

38. Computational Radiology Laboratory. Slide 38 Visualization of aligned data Matched preoperative fMRI and DT-MRI aligned with intraoperative MRI. Tensor alignment: Ruiz et al. 2000

39. Computational Radiology Laboratory. Slide 39 Conclusion Validation strategies for registration: –Comparison of transformations. –Fiducials Manual, automatic. –Overlap statistics – as for segmentation. Validation strategies for segmentation: –Digital and physical phantoms. –Comparison to domain experts. –STAPLE.

40. Computational Radiology Laboratory. Slide 40 Acknowledgements Neil Weisenfeld. Andrea Mewes. Richard Robertson. Joseph Madsen. Karol Miller. Michael Scott. This study was supported by: R01 RR021885, R01 EB008015, R01 GM074068 Collaborators William Wells. Kelly H. Zou. Frank Duffy. Arne Hans. Olivier Commowick. Alexandra Golby. Vicente Grau.