Segmentation

Terminology Segmentation, grouping, perceptual organization: gathering features that belong together Fitting: associating a model with observed features Top-down segmentation: pixels belong together because they come from the same object Bottom-up segmentation: pixels belong together because they look similar

The goals of segmentation Separate image into coherent “objects” Berkeley segmentation database: image human segmentation

The goals of segmentation Separate image into coherent “objects” Top-down or bottom-up process? Supervised or unsupervised? Group together similar-looking pixels for efficiency of further processing X. Ren and J. Malik. Learning a classification model for segmentation. ICCV 2003.Learning a classification model for segmentation. “superpixels”

The goals of segmentation Separate image into coherent “objects” Top-down or bottom-up process? Supervised or unsupervised? Group together similar-looking pixels for efficiency of further processing Related to image compression Measure of success is often application-dependent

Segmentation: Outline Inspiration from psychology Segmentation as clustering K-means Mean shift Segmentation as partitioning Graph-based segmentation, normalized cuts Integrating top-down and bottom-up segmentation for recognition

The Gestalt school Grouping is key to visual perception The Muller-Lyer illusion

The Gestalt school Grouping is key to visual perception Elements in a collection can have properties that result from relationships “The whole is greater than the sum of its parts”

The Gestalt school Grouping is key to visual perception Elements in a collection can have properties that result from relationships “The whole is greater than the sum of its parts” subjective contours occlusion familiar configuration

Figure-ground discrimination

The ultimate Gestalt?

Gestalt factors These factors make intuitive sense, but are very difficult to translate into algorithms

Gestalt factors They may be hard to put into algorithms, but understanding them can come in useful for interface design

Gestalt factors They may be hard to put into algorithms, but understanding them can come in useful for interface design

Segmentation as clustering Source: K. Grauman

Segmentation as clustering Source: K. Grauman

Different clustering strategies Agglomerative clustering Start with each point in a separate cluster At each iteration, merge two of the “closest” clusters Divisive clustering Start with all points grouped into a single cluster At each iteration, split the “largest” cluster K-means clustering Iterate: assign points to clusters, compute means K-medoids Same as k-means, only cluster center cannot be computed by averaging The “medoid” of each cluster is the most centrally located point in that cluster (i.e., point with lowest average distance to the other points)

Image Intensity-based clustersColor-based clusters K-Means clustering K-means clustering based on intensity or color is essentially vector quantization of the image attributes Clusters don’t have to be spatially coherent

K-Means clustering K-means clustering based on intensity or color is essentially vector quantization of the image attributes Clusters don’t have to be spatially coherent Clustering based on (r,g,b,x,y) values enforces more spatial coherence

K-Means pros and cons Pros Simple and fast Converges to a local minimum of the error function Cons Need to pick K Sensitive to initialization Sensitive to outliers Only finds “spherical” clusters

Mean shift segmentation An advanced and versatile technique for clustering-based segmentation D. Comaniciu and P. Meer, Mean Shift: A Robust Approach toward Feature Space Analysis, PAMI 2002.Mean Shift: A Robust Approach toward Feature Space Analysis

The mean shift algorithm seeks a mode or local maximum of density of a given distribution Choose a search window (width and location) Compute the mean of the data in the search window Center the search window at the new mean location Repeat until convergence Mean shift algorithm

Region of interest Center of mass Mean Shift vector Mean shift Slide by Y. Ukrainitz & B. Sarel

Region of interest Center of mass Mean Shift vector Mean shift Slide by Y. Ukrainitz & B. Sarel

Region of interest Center of mass Mean Shift vector Mean shift Slide by Y. Ukrainitz & B. Sarel

Region of interest Center of mass Mean Shift vector Mean shift Slide by Y. Ukrainitz & B. Sarel

Region of interest Center of mass Mean Shift vector Mean shift Slide by Y. Ukrainitz & B. Sarel

Region of interest Center of mass Mean Shift vector Mean shift Slide by Y. Ukrainitz & B. Sarel

Region of interest Center of mass Mean shift Slide by Y. Ukrainitz & B. Sarel

Cluster: all data points in the attraction basin of a mode Attraction basin: the region for which all trajectories lead to the same mode Mean shift clustering Slide by Y. Ukrainitz & B. Sarel

Find features (color, gradients, texture, etc) Initialize windows at individual pixel locations Perform mean shift for each window until convergence Merge windows that end up near the same “peak” or mode Mean shift clustering/segmentation

Mean shift segmentation results

More results

Mean shift pros and cons Pros Does not assume spherical clusters Just a single parameter (window size) Finds variable number of modes Robust to outliers Cons Output depends on window size Computationally expensive Does not scale well with dimension of feature space

Graph-based segmentation Represent features and their relationships using a graph Cut the graph to get subgraphs with strong interior links and weaker exterior links

Images as graphs Node for every pixel Edge between every pair of pixels (or every pair of “sufficiently close” pixels) Each edge is weighted by the affinity or similarity of the two nodes w ij i j Source: S. Seitz

Segmentation by graph partitioning Break Graph into Segments Delete links that cross between segments Easiest to break links that have low affinity –similar pixels should be in the same segments –dissimilar pixels should be in different segments ABC Source: S. Seitz w ij i j

Measuring Affinity Intensity Color Distance

Scale affects affinity Small σ: group only nearby points Large σ: group far-away points

Graph cut Set of edges whose removal makes a graph disconnected Cost of a cut: sum of weights of cut edges A graph cut gives us a segmentation What is a “good” graph cut and how do we find one? A B Source: S. Seitz

Graph cut

Affinity matrixBlock detection * Slides from Dan Klein, Sep Kamvar, Chris Manning, Natural Language Group Stanford University Multi-way graph cut

Minimum cut We can do segmentation by finding the minimum cut in a graph Efficient algorithms exist for doing this Drawback: minimum cut tends to cut off very small, isolated components * Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003

Minimum cut We can do segmentation by finding the minimum cut in a graph Efficient algorithms exist for doing this Drawback: minimum cut tends to cut off very small, isolated components Ideal Cut Cuts with lesser weight than the ideal cut * Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003

Normalized cut A minimum cut penalizes large segments This can be fixed by normalizing the cut by component size The normalized cut cost is: The exact solution is NP-hard but an approximation can be computed by solving a generalized eigenvalue problem assoc(A, V) = sum of weights of all edges in V that touch A J. Shi and J. Malik. Normalized cuts and image segmentation. PAMI 2000Normalized cuts and image segmentation.

Example results

Using texture features for segmentation Texture descriptor is vector of filter bank outputs J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001."Contour and Texture Analysis for Image Segmentation"

Using texture features for segmentation Texture descriptor is vector of filter bank outputs Textons are found by clustering J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001."Contour and Texture Analysis for Image Segmentation"

Using texture features for segmentation Texture descriptor is vector of filter bank outputs Textons are found by clustering Affinities are given by similarities of texton histograms over windows given by the “local scale” of the texture J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001."Contour and Texture Analysis for Image Segmentation"

The importance of scale J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001."Contour and Texture Analysis for Image Segmentation"

Example results J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001."Contour and Texture Analysis for Image Segmentation"

Pros Generic framework, can be used with many different features and affinity formulations Cons High storage requirement and time complexity Bias towards partitioning into equal segments * Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003 Normalized cuts: Pro and con

Integrating top-down and bottom-up segmentation Z.W. Tu, X.R. Chen, A.L. Yuille, and S.C. Zhu. Image parsing: unifying segmentation, detection and recognition. IJCV 63(2), , 2005.Image parsing: unifying segmentation, detection and recognition.

Image parsing Define generative models for text and faces Deformable spline-based templates for characters PCA model for faces

Top-down: propose a model for a given region Bottom-up: verify the consistency of the model with image features Image parsing Data-driven Markov Chain Monte Carlo

Example results