Exercise class 13 : Image Segmentation Many slides from Steve Seitz

Image segmentation Segment – A region of the image which: Belongs to a single object. Looks uniform (gray levels, color…) Have the same attributes (texture…) Moves in a similar way (motion segmentation) … Segmentation is a problem of assigning attributes (pixels) to objects Compact representation of object to groups (also called grouping, perceptual organization, fitting) Problem to segment – huge amount of data (pixels), we don’t know in which of the pixels should we use

Why do we need it ? The first step towards higher level vision (object recognition etc.) Treat different segments differently. (compression, histogram manipulations…) Use it to find regions which are related to a single “model” (motion computation, texture analysis…) Many more applications. We want to treat object as a hole (high level vision – extracting the dog from the background) – very hard In many algorithms we would like to treat differently different part and adjust the parameters to each part separately Define regions that belong to single model (motion model) Some examples: segmenting to shots in video, matching close area in images, locating faces / people in pictures, finding buildings in satellite images, searching in images database – compact representation of images

Image Segmentation Looking at some more examples – there are many possible segmentation with different hierarchy between them Another hard question – what is does the user want to see (level of details) – depends on the application

Image segmentation How do you pick the right segmentation? Bottom up segmentation: - Tokens belong together because they are locally coherent. Top down segmentation: - Tokens grouped because they lie on the same object.

“Correct” segmentation There may not be a single correct answer. Gestalt Laws seek to formalize what is an object proximity, similarity, continuation, closure, common fate Segmentation can be thought of a partition problem Many approaches proposed cues: color, regions, contours, texture, motions automatic vs. user-guided no clear winner Using low level image features to try and extract the required grouping The human vision has usually very good ability to segment. It was shown that the context (top down approach) affects the segmentation The gestalt principles offer explanation to grouping in different examples and situations

Main Approaches Histogram-based segmentation Region-based segmentation Edge detection Region growing Region splitting and merging. Clustering K-means Mean shift Graph Based Clustering Our visual system is proof that automatic methods are possible classical image segmentation methods are automatic

Example: Splitting text and background Thresholding: define a threshold T such that each pixel x,y is “text” iff I(x,y)<T. How do we determine the threshold ? Just choose 128 as a threshold (problematic for dark images) Use the median/mean (both are not good, as most of the paper is white)

Histogram-based segmentation Goal Break the image into K regions (segments) Solve this by reducing the number of gray levels / colors to K and mapping each pixel to the closest gray level / color

Histogram-based segmentation Goal Break the image into K regions (segments) Solve this by reducing the number of gray levels / colors to K and mapping each pixel to the closest gray level / color Possible threshold – minimum on histogram (between larger values) Here’s what it looks like if we use two gray levels

Problems with that approach Noise is considered as text. Holes and discontinuities in the resulting text. Changes in the illumination Solutions: 1. Enhance the result using morphological operations 2. Adaptive threshold. 3. Use spatial information ! A

Adaptive Threshold Use a running window.. Assume a constant illumination in each window. For each window: compute a threshold and use it to segment the pixel in the middle of the window. Problem: This scheme will find text in blank areas! Possible Solution: we can try to recognize that these regions are unimodal.

Segmentation as clustering Consider the image as a set of points in the N-dimensional space: Color Histograms: p=(R(x,y),G(x,y),B(x,y)) in R3. (ignores the spatial information.) Gray level images: p=(x,y,I(x,y)) in R3 Color images: p =(x,y,R(x,y),G(x,y),B(x,y)) in R5 Texture: p= (x,y,vector_of_fetures) … From this stage, we forget the meaning of each coordinate. We deal with arbitrary set of points. Important: We need to find the appropriate space in which the distance between points is meaningful ! Clustering problem is assigning tokens to groups. In the case of images as we said the assignment can be based on low level features such as color, texture, proximity Need to define a distance matrix based on the features

Segmentation using simple clustering model Two natural algorithms Clustering by splitting the entire data set is one cluster and we split it to get a good clustering Clustering by merging each point is a different cluster and we merge similar clusters Usually the simple methods use greedy algorithms Important issues : Define a proper distance between data points Define a proper distance between clusters How many clusters are there? Main difficulty – Defining threshold : when should we stop the algorithm? Very large number of pixels : hard to look for best split / merge.

Segmentation as Clustering How to choose the representative colors? This is a clustering problem! G G R R Objective Each point should be as close as possible to a cluster center Minimize sum squared distance of each point to closest center As opposed to simple greedy algorithms, we want to represent the problem as aiming for an objective function that represents good clustering and look for the best possible representation Clustering problem is assigning tokens to groups. In the case of images as we said the assignment can be based on low level features such as color, texture, proximity Need to define a distance matrix based on the features In the following examples – distance in 2D but this can be thought also as a representation of distances in higher dimensional feature space Instead of using Ad-hook solutions try to define a objective function and maximize/minimize it

Break it down into subproblems Suppose I tell you the cluster centers ci Q: how to determine which points to associate with each ci? A: for each point p, choose closest ci Suppose I tell you the points in each cluster Q: how to determine the cluster centers? We notice that we can split the problem to 2 easy problems : if we know the cluster centers we can easily allocate a pixel to cluster. If we know the allocation of points to cluster we can easily calculate the best cluster centers A: choose ci to be the mean of all points in the cluster

K-means clustering K-means clustering algorithm Randomly initialize the cluster centers, c1, ..., cK Given cluster centers, determine points in each cluster For each point p, find the closest ci. Put p into cluster i Given points in each cluster, solve for ci Set ci to be the mean of points in cluster i If ci have changed, repeat Step 2 Java demo: http://www.elet.polimi.it/upload/matteucc/Clustering/tutorial_html/AppletKM.html Properties Will always converge to some solution Can be a “local minimum” does not always find the global minimum of objective function:

Example: Segmentation using Color + Texture

K-means limitations Converges to a local minima Constant number of clusters, k Cluster’s shape is typically spherical

Mean Shift A non-parametric technique for analyzing complex multimodal feature spaces and estimating the stationary points (modes) of the underlying probability density function without explicitly estimating it. Ukrainitz & Sarel

An Example Feature Space Ukrainitz & Sarel

An Example Feature Space Parametric Density Estimation? Ukrainitz & Sarel

Distribution of identical billiard balls Mean Shift Clustering Region of interest Center of mass Mean Shift vector Objective : Find the densest region Distribution of identical billiard balls Ukrainitz & Sarel

Distribution of identical billiard balls Mean Shift Clustering Region of interest Center of mass Mean Shift vector Objective : Find the densest region Distribution of identical billiard balls Ukrainitz & Sarel

Distribution of identical billiard balls Mean Shift Clustering Region of interest Center of mass Mean Shift vector Objective : Find the densest region Distribution of identical billiard balls Ukrainitz & Sarel

Distribution of identical billiard balls Mean Shift Clustering Region of interest Center of mass Mean Shift vector Objective : Find the densest region Distribution of identical billiard balls Ukrainitz & Sarel

Distribution of identical billiard balls Mean Shift Clustering Region of interest Center of mass Mean Shift vector Objective : Find the densest region Distribution of identical billiard balls Ukrainitz & Sarel

Distribution of identical billiard balls Mean Shift Clustering Region of interest Center of mass Objective : Find the densest region Distribution of identical billiard balls Ukrainitz & Sarel

Synthetic Example Ukrainitz & Sarel

Real Data Example Ukrainitz & Sarel

Mean shift based image segmentation Ukrainitz & Sarel

Mean shift based image segmentation Ukrainitz & Sarel

Segmentation by min (s-t) cut [Boykov 2001] Cut in a graph set of edges whose removal makes a graph disconnected cost of a cut: Find minimum cut gives you a segmentation of the graph

Segmentation by min (s-t) cut [Boykov 2001] min cut s t Graph construction: node for each pixel, link between pixels specify a few pixels as foreground and background create an infinite cost link from each bg pixel to the “t” node create an infinite cost link from each fg pixel to the “s” node compute min cut that separates s from t how to define link cost between neighboring pixels? We can see segmentation as a problem of finding good cut in a graph. All the pixels in the same connected component will be considered as a single cluster Finding - Min cut, we showed in previous lesson (on stitching) that this is a problem of flow in graph (min-cut max flow) and can be solved in polynomial time

Segmentation by min (s-t) cut [Boykov 2001] how to define link cost between neighboring pixels? Assign large weight on edges of pixels that you expect to be in the same group (body / background) F(i)- gray level value in pixel I - normalization factor

Examples for user assistance Min-cut segmentation (example from Micusik B. and Hanbury A )

Automatic graph cut [Shi & Malik] j w(i,j) w i Fully-connected graph node for every pixel link between every pair of pixels, i,j cost w(i,j) for each link w(i,j) measures similarity similarity is inversely proportional to difference in color and position

Automatic graph cut [Shi & Malik] Construct a weighted graph from the image pixels G = (V,E) by taking each pixel and connecting each pair of pixels in an edge F(i) – gray level value in pixel i X(i) – pixel i position

Segmentation by Graph Cuts w A B C Break Graph into Segments Delete links that cross between segments Easiest to break links that have low cost (similarity) similar pixels should be in the same segments dissimilar pixels should be in different segments

Minimum cut Criterion for partition: What is the problem with the min cut? Problem! Weight of cut is directly proportional to the number of edges in the cut. Ideal Cut Cuts with lesser weight than the ideal cut First proposed by Wu and Leahy

Normalized Cut Cut in a graph penalizes large segments Fix by normalizing in size of segments vol (A) = sum of costs of all edges that touch A Finds better cut

Formal definitions Association of two sets: Volume of a set: Normalized cut: A fraction of the total edge connections to all the nodes in the graph

How to minimize Ncut? Transform Ncut equation to a matrix form Solve an eigenvector problem for this matrix

Graph-based Image Segmentation Image (I) Intensity Color Edges Texture Graph Affinities (W) Slide from Timothee Cour (http://www.seas.upenn.edu/~timothee)

Graph-based Image Segmentation Image (I) Intensity Color Edges Texture Graph Affinities (W) Slide from Timothee Cour (http://www.seas.upenn.edu/~timothee)

Graph-based Image Segmentation Image (I) Eigenvector X(W) Intensity Color Edges Texture Graph Affinities (W) Slide from Timothee Cour (http://www.seas.upenn.edu/~timothee)

Graph-based Image Segmentation Image (I) Eigenvector X(W) Discretization Intensity Color Edges Texture Graph Affinities (W) Slide from Timothee Cour (http://www.seas.upenn.edu/~timothee)

Graph-based Image Segmentation G = {V,E} V: graph nodes E: edges connection nodes Pixels Pixel affinity Slides from Jianbo Shi

Graph terminology Affinity matrix: Slides from Jianbo Shi

the size of W is quadratic Affinity matrix Similarity of image pixels to selected pixel Brighter means more similar N pixels M pixels Warning the size of W is quadratic with the number of the image pixels! Reshape N*M pixels N*M pixels

Graph terminology Degree of node: … … Slides from Jianbo Shi

Graph terminology Volume of set: Slides from Jianbo Shi

Graph terminology Cuts in a graph: Slides from Jianbo Shi

Normalized Cut A B A B Volume of set (or association): Define normalized cut: “a fraction of the total edge connections to all the nodes in the graph”: A B A B

Optimization Problem Given a set A of vertices , consider so that 

y’s values are quantized Optimization Problem Replacing x by y: Subject to: 1) (b is depended in W) 2) NP-Hard! y’s values are quantized

Optimization Problem - Relaxation Relax the constranits on y: Subject to The solution is the (second) minimal eigenvector of D-W

Algorithm Define a similarity function between 2 nodes. i.e.: Compute affinity matrix (W) and degree matrix (D). Solve Use the eigenvector with the second smallest eigenvalue to bipartition the graph. Decide if re-partition current partitions.

Example (1) Eigenvectors Segments

Example (2) Original Segments * Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003

Real segmentation example

Ideas ? Steps in solving specific problem Make the assumptions appropriate for this case Size of the warm is relatively constant Warm does not move much between successive frames Lighting is changing between frames and within the frame, but can be considered as constans on small window Warm is not the only dark area in the picture Use your assumption to “tailor” a specific solution

Real segmentation example

Real segmentation example

Examples Using the same thresh for 2 different frames Using adaptive thresh based on gradient values

How to remove ‘wrong areas’ ? Put additional condition on the selected area – to be next to large gradient pixel Remove noise by using morphological operations

Possible Solution Find the warm location in the first frame Use gradient value to determine threshold Remove from the selection pixels that are selected according to the threshold but do not have large gradient in their area Use morphological operation to remove noise Using the warm location in the previous frame, search in the current frame using a simple threshold on window around the warm Update window location based on the new position of warm And still – in practice can know if this works only after trying ….

Thanks and Good Luck!