A. Criminisi, T. Sharp and K. Siddiqui

Properties of our algorithm efficient on high-res./nD images (~milliseconds) easy to edit and fix accurate (e.g. handling of thin structures) robust to noise edge sensitive captures uncertainty (probabilistic output) State of the art segmentation algorithms QPBO Tree-reweighted mess. passing Random Walker Min-cut/max-flow Belief propagation Level Sets Geodesic active contours Region growing K-means Most based on complex energy minimization -> slow.

Input imageUser-entered brush strokesLikelihood of Fg v Bg This is noisy! In order to obtain segmentation we need to encourage spatial smoothness Notation Point position Image intensities Output segmentation Appearance likelihood Appearance likelihood is computed from histograms of intensities accumulated under the two brush strokes Legend Green indicates high probability of Fg Red indicates high probability of Bg Grey for uncertain

A very efficient way of propagating image information around

ImageBinary mask unit tangent vector Geodesic distance (Euclidean for =0) Coronal CT view of r. kidney User-drawn brush stroke Binary mask of brush strokeGradient and tangent vectors Output geodesic distance D(p)

Output geodesic distance Forward pass: (top-left to bottom-right) Backward pass: (bottom-right to top-left) Properties of algorithm Contiguous memory access Parallelizable GPU-friendly no region growing no FMM no level set Input Image (8bpp) after W/L mapping! GDT raster scan algorithm with

GPU algorithm implemented on NVidia processors using the CUDA language. Typical CT image resolution = 512 X 512 Timings Downward pass: with …similarly for the other three passes. The downward pass. The red column shows pixels processed by the current thread. The distance values in the top (green) row Have already been computed. Distances along the arrow directions Are computed from texture reads as in the equation on the left panel. In 2D four passes are necessary: top-bottom, bottom-top, left-right, right-left Downward pass: 123 y x

direction of raster scan GPU algorithm implemented on NVidia processors using the CUDA language. Pass 1 of 6: In 3D six passes are necessary. with Pass 1 of 6: y x z …similarly for the other five passes.

How do we impose smoothness ?

Input binary mask M with Signed distance from boundary Signed distance D s from boundary (in green) Signed distance D s (zoomed) For ease of explanation we focus on a toy 2D example here.

Geodesic morphology Signed distance D s from boundary (green) Input binary mask M Geodesic dilation Geodesic erosion Eroded mask Dilated mask In each of the eroded and dilated masks part of the noise has been removed. Now we need to combine the two so as to remove all of the noise.

Geodesic morphology, real example This is all extremely fast to compute. Distance D s Geodesic erosion Geo. dilation, small d Geo. dilation, large d

Eroded maskDilated mask Final, symmetric signed distance Symmetric signed distance The new distance is much smoother than the original one because the effect of noise has been reduced. Original signed distance

The GSF operator produces the final, noise-free mask M s as: Input binary mask M Symmetric signed distanceThresholding at 0Final noise-free mask More examples with different noise patterns Input Distance Filtered imageInput Distance Filtered image

Binary case was Real-valued case inputGSF GGDT:Generalized Geodesic Distance Ms: output segmentation GSF( )

Ms: Output of GSF for small Ms: Output of GSF for large smoothlocked Larger values of produce smoother segmentations

The actual algorithm

Output of GSF for fixed Appearance lik. Soft input mask Distance term with Data likelihood I. Compute pixel-wise likelihoods from user hints combine User brushes Appearance likelihood L a Distance term L d

II. Segmentation from pixel-wise likelihoods Output of GSF for fixed Geodesic symmetric distance Segmentation via GSF GSF( ) Appearance likelihood L a Output segmentation M s Distance signal L d

Aortic aneurism The aorta, bottom of heart and pelvis have been segmented in 3D by our technique. The whole process (including user interaction) takes only a few seconds. Three views of the segmented aorta. Bony structures (Bg) are shown faded to provide spatial context. Input is CT.

Carotid arteries The carotids have been segmented interactively by our technique. Segmenting such long and thin structures is usually a problem for other existing algorithms. Three views of segmented carotids. Bony structures (Bg) are shown faded to provide spatial context. Input is CT.

Studying the interaction between aorta and spine The aorta and its thin bifurcations have been segmented by our technique in only a few seconds. Three views of segmented aorta. The spine and heart (Bg) are shown faded to provide spatial context. Input is CT.

A liver tumor The tumor has been segmented with a single user click. The whole operation has taken only a few milliseconds. Now that the tumor has been isolated statistics about its density, shape and texture are easily computed (right panel). Axial CT slice. The user segments a tumor in only a few millisecondsAutomatic measurements of tumor

Segmenting different structures in MR images

Segmentation of noisy images Very noisy input image. Input is CT. Our segmentation Encouraging spatial smoothness is especially useful when dealing with very noisy images.

Very noisy input image. Input is MROur segmentation Segmentation of noisy images Encouraging spatial smoothness is especially useful when dealing with very noisy images.

Testing accuracy in low-contrast images Very low-contrast input image Our segmentation The geodesic term in the definition of the geodesic distance enables segmentation in low contrast images.

Capturing uncertainty in low-contrast regions In low gradient regions the algorithm correctly returns low confidence segmentation In low gradient regions the algorithm correctly returns low confidence segmentation Sharp segmentation (high confidence) because of strong gradients Sharp segmentation (high confidence) because of strong gradients

Images and ground-truth from the standard GrabCut test dataset Not much of a difference in terms of accuracy. [ Rother, C. Kolmogorov, V and Blake, A. GrabCut: Interactive foreground extraction using iterated graph cut. SIGGRAPH ] [Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., Tappen, M., Rother, C.: A comparative study of energy minimization methods for markov random fields. In: ECCV. (2006)]

up to 60X speed up compared to min-cut. with similar accuracy [ Rother, C. Kolmogorov, V and Blake, A. GrabCut: Interactive foreground extraction using iterated graph cut. SIGGRAPH ]

[ Bai, X. and Sapiro, G. A geodesic framework for fast interactive image and video segmentation and matting. ICCV 2007, Rio, Brasil. ] up to 30X speed up compared to Bai et al., while avoiding topology issues. also, Bai et al. do not impose spatial smoothness