Automatic segmentation of brain structures Alireza Akhondi-Asl Boston Children’s Hospital, and Harvard Medical School 300 Longwood Ave. Boston MA 02115 USA

Table of Contents Introduction Methods Results Some Applications Definition Why we need segmentation Problems of manual segmentation Challenges of automatic segmentations Literature review Methods Results Some Applications Conclusion

Introduction Segmentation Visual assessment of images Identification of structure in images. Visual assessment of images Head position 2D Longitudinal changes Quantitative manual efforts Time Consuming Sometimes we do not have an expert for our application Inter/Intra rater variability Automated segmentation algorithms are thus of special interest.

Introduction Problems of Automatic Segmentation: Tissue Inhomogenity Partial Volume Effect Noise and Artifact Various Imaging Modalities Changes in the Structures Due to Diseases Modeling Limitations Number of parameters Complexity of the algorithm

Introduction Applications: Quantitative image analysis Abnormality Detection Cortical gray matter thickness Gray-matter/white-matter blurring Lesion detection MS, Tumor Volumetry Longitudinal Analysis Image guided therapy Visualization

Segmentation Parts Many different algorithms and a wide range of principles upon which they are based. However, they all have these three key parts: Energy Function Shape Representation Optimization

Segmentation Methods Shape Based Parametric Snake, Balloon,… Non-Parametric Level Set Clustering Probabilistic Template Based How they work: Prior Knowledge Intensity Statistics Shape Statistics Location Relationship Shape Relative Position Intensity Intersection Symmetry

Evaluation (Spatial overlap)

Evaluation

Multiple-Template-Based Segmentation

Fusion Why we need multiple templates? Aligned label map of a template is not perfectly representing the anatomy of the target image: Registration errors Registration: Maximization of the local intensity similarity of the two images in the space of the topology preserving maps. Topological differences between the anatomies of the template and the target image. Sub-optimal selection of the registration parameters. Template error Template segmentation is the output of a manual or an automatic segmentation procedure: Systematic and random errors To get the perfect fusion result: We need to have an appropriate template population. We need to know the performance of the templates. Since the performances are not a priori, we need to estimate the performance of the templates.

Registration Errors

Template Errors

Multiple Template Segmentation Registration Non-rigid registration Template Selection It reduces the computation time and also may increase the quality Intensity Based Transform Based Fusion Majority voting Weighted majority voting STAPLE Post-Processing Active Contours Graph Cut

STAPLE STAPLE and its extensions are Bayesian algorithms. STAPLE (Simultaneous Truth and Performance Level Estimation): An algorithm for estimating performance and ground truth from a collection of independent segmentations. Simultaneous estimation of hidden ``ground truth’’ and local or global expert performance using intensity and/or label information. Enables comparison between and to experts. Expectation-Maximization General procedure for estimation problems that would be simplified if some missing data was available. Other extensions STAPLE MAP Probabilistic STAPLE LOP STAPLE Empirical Bayesian Based STAPLE LOCAL STAPLE MAP STAPLE and its extensions are Bayesian algorithms.

Algorithm Description True segmentation Ti for each voxel i May be binary May be categorical Expert j makes segmentation decisions Dij Expert performance s’s characterizes probability of deciding label s’ when true label is s.

STAPLE STAPLE Posterior probability of reference segmentation Prior Probability distribution of the reference segmentation Performance parameters of the raters

Intensity information When there is an anatomical difference between the target image and a template, it is more likely to have: A mismatch between the segmentation labels of the two images. Some intensity dissimilarity between the two images. It is sensible to use the intensity information of the templates and the target image as a source of information in the fusion process. Intensity information can be used in a variety of ways as a source of information in the STAPLE formulation: Probabilistic STAPLE: The segmentations can be modified based on intensity information. LOP-STAPLE: The intensity similarity can be used as a reliability weight in the decision process. MAP STAPLE: The intensity information can be used as the prior information.

Probabilistic STAPLE (PSTAPLE) We propose modifying each one of the aligned template segmentations to correct errors due to the uncaptured anatomical differences between the target image and the templates. Each one of the trained classifiers is used to segment the target image. The output of each one of the classifiers is a probabilistic segmentation of the target image. Train Segment Probabilistic STAPLE Train Segment

Probabilistic STAPLE

LOP-STAPLE In this model, we assume that the rater decision at each voxel is associated with a reliability weight where the higher weights indicate higher credibility of the decisions. We use logarithmic opinion pool (LOP) to combine the weighted decisions. LOP is a model where the linear weighted average of the logarithm of the rater probabilities is used as the aggregating method. Local Intensity Similarity LOP STAPLE Local Intensity Similarity

LOP-STAPLE

MAP STAPLE A MAP formulation of the STAPLE algorithm which uses Dirichlet distribution to model the prior probability of the performance parameters of the templates. This MAP formulation has a closed form solution. This significantly increases the speed of the algorithm compared to the previous approach. We estimate the parameters of the prior distributions based on the intensity and label information of the templates and the target image. Estimate MAP Parameters MAP STAPLE Estimate MAP Parameters

Dirichlet Distribution Dirichlet distribution is the multivariate generalization of the Beta distribution which can be used to model probabilities. It has all benefits of the Beta distribution such as capability to model any prior on the parameters and easy computation of the logarithm and the derivatives.

MAP STAPLE

Empirical Bayesian Based STAPLE E-STEP: M-STEP: The estimation of the prior probability of reference standard at each step is the estimated posterior probability of reference standard at the previous step.

Prior in Bayesian Analysis In the standard Bayesian analysis the prior distribution is assumed to be known and the observations do not re-scales the prior distribution. To improve the fusion performance, we use a novel framework to estimate the prior using a parametric empirical Bayesian procedure. We assume that prior probability of reference standard is another set of unknown parameters and to solve the problem, we use the EM algorithm to iteratively estimate the ground truth, the performance parameters, and the prior distribution of the hidden ground truth.

Local STAPLE In this way expert’s performance is computed locally If we apply STAPLE algorithm for each voxel independently, then the problem reduces to the majority voting. We have two extreme approaches for performance evaluation: - STAPLE - Majority Voting Is there any other way? - LOCAL STAPLE For each voxel in the image: Use its neighbor and define an ROI Consider the same ROI for all of experts segmentations Apply STAPLE based on this ROI. Combine the results. In this way expert’s performance is computed locally

Results We have evaluated our method and some of the state-of-the-art fusion algorithms for the automatic segmentation Two MRI datasets. IBSR: 18 Subjects, 34 gray and white matter structures and 96 structures in Cerebral Cortex. NMM: 15 Subjects, 53 gray and white matter structures and cortical gray matter lobes. We have used the leave-one-out strategy for the evaluation. ANTS was chosen for the alignment of the templates to the target images. We have used the fusion algorithm for each structure independently and compared the fusion results.

RESULTS (PSTAPLE) Quantitative Comparison of the IBSR Multi-Atlas Segmentation Results. Comparison of generalized Dice coefficient of M12 Local MAP STAPLE and M13 : Local MAP PSTAPLE for structures in 18 IBSR datasets. It indicates that Local MAP PSTAPLE is superior to the Local STAPLE. The horizontal axis represents each subject.

RESULTS (PSTAPLE) Comparison of Dice coefficients for the proposed methods. Results of the segmentation of 16x2 +1=33 sample structures and average of the 128 brain structures of the IBSR dataset for M_0:Majority Voting, M_1: Artaechevarria et al. method, M_2: Sabuncu et al. algorithm, M_3: SIMPLE, M_4: STAPLER, M_5: COLLATE, M_6: STAPLE , M_7: Multi-STEPS, M_8: NLS, M_9: PICSL-MALF, M_10: Probabilistic Voting, M_11:PSTAPLE.

RESULTS (PSTAPLE) Quantitative Comparison of the IBSR Multi-Atlas Segmentation Results. Comparison of generalized Dice coefficient of M13: Local MAP PSTAPLE, M1 : method of Artaechevarria et al., M2 : method of Sabuncu et al., M7 : Multi-STEPS, M8 : NLS, and M9 : PICSL-MALF for structures in 18 IBSR datasets. It indicates that Local MAP PSTAPLE is superior to the other methods. The horizontal axis represents each subject.

RESULTS (PSTAPLE) on IBSR Data

RESULTS (PSTAPLE) Quantitative Comparison of the NMM Multi-Atlas Segmentation Results. Comparison of generalized Dice coefficient of M13: Local MAP PSTAPLE, M1 : method of Artaechevarria et al., M2 : method of Sabuncu et al., M7 : Multi-STEPS, M8 : NLS, and M9 : PICSL-MALF for structures in 15 NMM datasets. It indicates that Local MAP PSTAPLE is superior to the other methods. The horizontal axis represents each subject.

RESULTS (PSTAPLE) On NMM Data

Applications (Epilepsy Surgery) Brain parcellation helps neurophysiologists recognize the functional/structural areas where the electrodes are placed

Applications (Fetal MRI) Volumetric fetal brain MRI reconstruction and automatic segmentation allows 3D visualization of the ventricles which is quite useful in the analysis of ventriculomegaly. Marching cubes surface model rendering of fetal ventricles over a transparent surface model rendering of the intracranial volume (ICV) based on volumetric fetal brain MRIeas where the electrodes are placed.

Applications (Functional Connectivity) Brain Segmentation Multiple-Atlas-Based Segmentation Pre-Processing of rsfMRI Building The Connectivity Matrix Graph Analysis rsfMRI Preprocessing Connectivity Matrix Graph Analysis

Conclusions Intensity information can help us to improve the segmentation accuracy. Intensity information in the STAPLE framework: Probabilistic STAPLE: The segmentations can be modified based on intensity information. Weighted STAPLE: The intensity similarity can be used as a reliability weight in the decision process. MAP STAPLE: The intensity information can be used as the prior information. The prior has an important role in determining the local optimum to which the EM algorithm converges. Our new algorithm is robust and has superior performance as compared to the other methods.

References Akhondi-Asl A, Hoyte L, Lockhart ME, Warfield SK. A Logarithmic Opinion Pool Based STAPLE Algorithm for The Fusion of Segmentations With Associated Reliability Weights, IEEE Transactions on Medical Imaging. 2014, 33(10),1997-2009. Siversson C, Akhondi-Asl A, Bixby S, Kim YJ, Warfield SK. Three-dimensional hip cartilage quality assessment of morphology and dGEMRIC by planar maps and automated segmentation, Osteoarthritis and Cartilage. 22(10), 1511-1515, 2014. Akhondi-Asl A, Warfield SK. Simultaneous Truth and Performance Level Estimation through Fusion of Probabilistic Segmentations. IEEE Transactions on Medical Imaging, 2013. Commowick O, Akhondi-Asl A, Warfield SK. Estimating A Reference Standard Segmentation with Spatially Varying Performance Parameters: Local MAP STAPLE, IEEE Transactions on Medical Imaging, 2012. Gholipour A, Akhondi-Asl A, Estroff J A, Warfield SK. Multi-atlas multi-shape segmentation of fetal brain MRI for volumetric and morphometric analysis of ventriculomegaly. Neuroimage, 2012. Akhondi-Asl A, Warfield SK. Estimation of the Prior Distribution of Ground Truth in the STAPLE Algorithm: An Empirical Bayesian Approach. MICCAI 2012. Akhondi-Asl A, Jafari-Khouzani K, Elisevich K, Soltanian-Zadeh H. Hippocampal volumetry for lateralization of temporal lobe epilepsy: Automated versus manual methods. Neuroimage, 2011.

Thank you!