1. Course
Introduction to Medical Imaging Segmentation 1 – Mean Shift and Graph-Cuts
Guy Gilboa

2. Topics
The segmentation problem
Mean Shift
Graph Cuts

3. The Segmentation Problem
Separate the image domain Ω into several disjoint subdomains Ω 𝑖 , (Ω= 𝑖 Ω 𝑖 ), according to input image 𝑓, where each subdomain is homogeneous in some sense.
Taken from

4. Segmentation Goal
The goal of segmentation is generally to generate a higher level representation and understanding of the image by partitioning it into homogeneous parts (or combining similar pixels to a set).
The idea behind it is that objects are often combined of just a few segments. A good segmentation of the image helps various computer vision algorithms (e.g. detection, recognition, labeling).

5. Segmentation solutions
The segmentation problem is very hard to solve. It is also not completely well defined, as homogeneous regions can be homogeneous not only in color but also in textures, patterns, orientations and other features.
For specific applications, often some previous segmentation examples or direction from the user is needed to obtain meaningful results.

6. Problems of simple thresholding
Not robust to noise.
Many holes and very small segments.
No spatial structure and smoothness.
[Cannot handle texture or illumination changes.]

7. Mean Shift Segmentation
A simple statistical way for non-parametric clustering of data.
Can be used for filtering, segmentation and more.
Very popular, easy to implement. Has less spatial regularity (thus somewhat less adequate for some medical imaging applications).
D. Comaniciu and P. Meer. "Mean shift: A robust approach toward feature space analysis." Pattern Analysis and Machine Intelligence, IEEE Trans on 24.5 (2002):
K. Fukunaga and L. Hostetler. "The estimation of the gradient of a density function, with applications in pattern recognition." Information Theory, IEEE Trans on 21.1 (1975):

8. Example 2: Liver CT Scans
Metastatic liver lesion segmentation of follow-up CT scans using mean-shift.
Ben Cohen, A., Diamant, I., Klang, E., Amitai, M., & Greenspan, H. (2014, March). Automatic detection and segmentation of liver metastatic lesions on serial CT examinations. In SPIE Medical Imaging (pp ). International Society for Optics and Photonics.‏

9. What is Mean Shift ? Taken from Ukrainitz & Sarel slides
A tool for:
Finding modes in a set of data samples, manifesting an
underlying probability density function (PDF) in RN
PDF in feature space
Color space
Scale space
Actually any feature space you can conceive …
Non-parametric
Density Estimation
Data
Discrete PDF Representation
Non-parametric
Density GRADIENT Estimation
(Mean Shift)
PDF Analysis

10. Non-Parametric Density Estimation
Assumption : The data points are sampled from an underlying PDF
Data point density
implies PDF value !
Assumed Underlying PDF
Real Data Samples

11. We would like to get to the peak of the hills (modes)

12. Mean Shift procedure Simple Mean Shift procedure:
Compute mean shift vector
Translate the Kernel window by m(x)

13. Discontinuity Preserving Smoothing
Feature space : Joint domain = spatial coordinates + color space
Meaning : treat the image as data points in the spatial and gray level domain
Image Data
(slice)
Mean Shift
vectors
Smoothing
result
Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer

14. Segmentation Example
…when feature space is only
gray levels…

15. Segmentation Example

16. Medical Application – Brain MRI
An Adaptive Mean-Shift Framework for MRI Brain Segmentation, Arnaldo Mayer and Hayit Greenspan (Tel Aviv Univ.), IEEE TMI 2009.
Partitioning the data to
Gray matter
White matter
Cerebro-spinal fluid

17. Segmentation process

18. Algorithm diagram

19. Graph cuts We will concentrate on Shi & Malik Normalized Cuts.
We will follow the lecture of CS Illinois
J. Shi, and J. Malik. "Normalized cuts and image segmentation."Pattern Analysis and Machine Intelligence, IEEE Transactions on 22.8 (2000):

20. Images as graphs Fully-connected graph node for every pixel
wij c j
Fully-connected graph
node for every pixel
link between every pair of pixels, i, j
similarity wij for each link
Source: Seitz

21. Similarity matrix
Increasing sigma

22. Segmentation by Graph Cutsw A B C
Break Graph into Segments
Delete links that cross between segments
Easiest to break links that have low cost (low similarity)
similar pixels should be in the same segments
dissimilar pixels should be in different segments
Source: Seitz

23. Cuts in a graph cut 𝐴,𝐵 = 𝑖𝜖𝐴,𝑗∈𝐵 𝑤 𝑖𝑗 Link CutB A
Link Cut
set of links whose removal makes a graph disconnected
cost of a cut:
cut 𝐴,𝐵 = 𝑖𝜖𝐴,𝑗∈𝐵 𝑤 𝑖𝑗
One idea: Find minimum cut
gives you a segmentation
fast algorithms exist for doing this
Source: Seitz

24. But min cut is not always the best cut...

25. Cuts in a graph a cut penalizes large segmentsB A
Normalized Cut
a cut penalizes large segments
fix by normalizing for size of segments
volume(A) = sum of costs of all edges that touch A
Source: Seitz

26. Some graph notations
Given an image or image sequence, set up a weighted graph: G=(V, E)
Vertex for each pixel
Edge weight for nearby pairs of pixels
Degree matrix ( 𝐷) 𝑖𝑖 = 𝑗 𝑤 𝑖𝑗 and 0 for 𝑖 ≠𝑗.
Adjacency matrix (𝐴 ) 𝑖𝑗 = 𝑤 𝑖𝑗 .
Graph Laplacian: L = D-W

27. Recursive normalized cuts
Solve for eigenvectors with the smallest eigenvalues: (D − A)y = λDy
Use the eigenvector with the second smallest eigenvalue to bipartition the graph
Note: this is an approximation
Recursively repartition the segmented parts if necessary
Details:

28. Normalized cuts results

29. Normalized cuts: Pros and cons
Generic framework, can be used with many different features and affinity formulations
Provides regular segments
Cons
Need to chose number of segments
High storage requirement and time complexity
Bias towards partitioning into equal segments

30. Medical example
“Learning-Based Vertebra Detection and Iterative Normalized-Cut Segmentation for Spinal MRI”, S-H Huang et al. IEEE TMI 2009.

31. Block diagram of the various stages

32. Comparsion to snakes