1. Image Segmentation February 27, 2007

2. Implicit Scheme is considerably better with topological change. Transition from Active Contours: –contour v(t)  front  (t) –contour energy  forces F A F C –image energy  speed function k I Level set: –The level set c 0 at time t of a function  (x,y,t) is the set of arguments { (x,y),  (x,y,t) = c 0 } –Idea: define a function  (x,y,t) so that at any time,  (t) = { (x,y),  (x,y,t) = 0 } there are many such   has many other level sets, more or less parallel to  only  has a meaning for segmentation, not any other level set of 

3. Need to figure out how to evolve the level set function!

4. Usual choice for  : signed distance to the front  (0)  - d(x,y,  ) if (x,y) inside the front  (x,y,0) =  0 “ on “  d(x,y,  ) “ outside “ 0 0 0 0000 00 0 0 0 0 0 0 0 0 000 0 0 000 0 -2 -3 1 1 1 1 1 1 1 1 111 1 1 111 1 1 1 1 1 1 1111 11 1 1 2 2 2 2 2 2 2 22 2 222 2 2 2 2 2 2 2 2 22222 2 2 2 3 3 3 3 3 3 3 3 3 33 333 3 3 3 3 3 3 34 4 4 4 4 4 4 4444 4 4 4 4 4 4 7 5 5 5 5 55 5 5 5 5 6 6 66  (t)  (x,y,t) 0 -2 5 Level Set Framework

5. 0 0 0 0000 00 0 0 0 0 0 0 0 0 000 0 0 000 0 -2 -3 1 1 1 1 1 1 1 1 111 1 1 111 1 1 1 1 1 1 1111 11 1 1 2 2 2 2 2 2 2 22 2 222 2 2 2 2 2 2 2 2 22222 2 2 2 3 3 3 3 3 3 3 3 3 33 333 3 3 3 3 3 3 34 4 4 4 4 4 4 4444 4 4 4 4 4 4 7 5 5 5 5 55 5 5 5 5 6 6 66  (x,y,t)  (x,y,t+1) =  (x,y,t) + ∆  (x,y,t) 1 1 1 1 1 1 1 1 1 1 1 1 1 111 1 111 1 1 1 1 1 11 11 1 -2 -3 -2 -3 2 2 2 2 2 2 2 2 22 2 222 2 2 2 2 2 2 2 2 222 0 0 0 0 0 0 0 0 0 0 0 0 00 0 000 0 0 0 0 0 0 00 00 0 2 2 2 2 3 3 3 3 3 3 3 33 333 3 3 3 3 3 3 34 4 4 4 4 4 4 4444 4 4 4 4 4 4 7 5 5 5 5 55 5 5 5 5 6 6 66 no movement, only change of values the front may change its topology the front location may be between samples Level Set

6. Segmentation with LS: Initialise the front  (0) Compute  (x,y,0) Iterate:  (x,y,t+1) =  (x,y,t) + ∆  (x,y,t) until convergence Mark the front  (t end ) Level Set

7. link between spatial and temporal derivatives, but not the same type of motion as contours! constant “force” (balloon pressure)  (x,y,t+1) -  (x,y,t) extension of the speed function k I (image influence) smoothing “force” depending on the local curvature  (contour influence) spatial derivative of  product of influences Equation for Front Propagation

8. Speed function: –k I is meant to stop the front on the object’s boundaries –similar to image energy: k I (x,y) = 1 / ( 1 + ||  I (x,y) || ) –only makes sense for the front (level set 0) –yet, same equation for all level sets  extend k I to all level sets, defining –possible extension: ^ kIkI ^ k I (x,y) = k I (x’,y’) where (x’,y’) is the point in the front closest to (x,y) ^ ( such a k I (x,y) depends on the front location ) Equation for Front Propagation

9. 0 0 0 0000 00 0 0 0 0 0 0 0 0 000 0 0 000 0 -2 -3 1 1 1 1 1 1 1 1 111 1 1 111 1 1 1 1 1 1 1111 11 1 1 2 2 2 2 2 2 2 22 2 222 2 2 2 2 2 2 2 2 22222 2 2 2 3 3 3 3 3 3 3 3 3 33 333 3 3 3 3 3 3 34 4 4 4 4 4 4 4444 4 4 4 4 4 4 7 5 5 5 5 55 5 5 5 5 6 6 66 1 1 1 1 1 1 1 1 1 1 1 1 1 111 1 111 1 1 1 1 1 11 11 1 -2 -3 -2 -3 2 2 2 2 2 2 2 2 22 2 222 2 2 2 2 2 2 2 2 222 0 0 0 0 0 0 0 0 0 0 0 0 00 0 000 0 0 0 0 0 0 00 00 0 2 2 2 2 3 3 3 3 3 3 3 33 333 3 3 3 3 3 3 34 4 4 4 4 4 4 4444 4 4 4 4 4 4 7 5 5 5 5 55 5 5 5 5 6 6 66 1. compute the speed k I on the front extend it to all other level sets 2. compute  (x,y,t+1) =  (x,y,t) + ∆  (x,y,t) 3. find the front location (for next iteration) modify  (x,y,t+1) by linear interpolation 0 0 0 0000 00 0 0 0 0 0 0 0 0 000 0 0 000 0 -2 -3 1 1 1 1 1 1 1 1 111 1 1 111 1 1 1 1 1 1 1111 11 1 1 2 2 2 2 2 2 2 22 2 222 2 2 2 2 2 2 2 2 22222 2 2 2 3 3 3 3 3 3 3 3 3 33 333 3 3 3 3 3 3 34 4 4 4 4 4 4 4444 4 4 4 4 4 4 7 5 5 5 5 55 5 5 5 5 6 6 66 0 0 0 0 0 0000 00 0 0 0 0 0 0 0 0 000 0 0 000 0  (x,y,t) Algorithm

10. Weaknesses of algorithm 1 –update of all  (x,y,t): inefficient, only care about the front –speed extension: computationally expensive Improvement: –narrow band: only update a few level sets around  –other extended speed: k I (x,y) = 1 / ( 1 + ||  I(x,y)|| ) ^ 0 0 0 0000 00 0 0 0 0 0 0 0 0 000 0 0 000 0 -2 1 1 1 1 1 1 1 1 111 1 1 111 1 1 1 1 1 1 1111 11 1 1 2 2 2 2 2 2 2 22 2 222 2 2 2 2 2 2 2 2 22222 2 2 2 Narrow band extension

11. Caution: –extrapolate the curvature  at the edges –re-select the narrow band regularly: an empty pixel cannot get a value  may restrict the evolution of the front 0 0 0 0000 00 0 0 0 0 0 0 0 0 000 0 0 000 0 -2 1 1 1 1 1 1 1 1 111 1 1 111 1 1 1 1 1 1 1111 11 1 1 2 2 2 2 2 2 2 22 2 222 2 2 2 2 2 2 2 2 22222 2 2 2 Narrow band extension

12. Level sets: –function  : [ 0, I width ] x [ 0, I height ] x N  R ( x, y, t )   (x,y,t) –embed a curve  :  (t) = { (x,y),  (x,y,t) = 0 } –  (0) is provided externally,  (x,y,0) is computed –  (x,y,t+1) is computed by changing the values of  (x,y,t) –changes using a product of influences –on convergence,  (t end ) is the border of the object Issue: –computation time (improved with narrow band) Summary

14. Segmentation: A region in the image 1.with some homogeneous properties (intensity, colors, texture, … ) 2.Cohesion (moved in a similar way, motion segmentation) Active contours will have difficulties with natural images such as Image Segmentation Similarity (intensity difference) Similarity (intensity difference) Proximity Proximity Continuity Continuity

15. Image Segmentation The first step towards higher level vision (object recognition etc. There may not be a single correct answer. Segmentation can be considered as a partition problem. Literature on this topic is tremendous. Many approaches: Cues such as color, regions, contours, texture, motion, etc. Automatic vs. user-assisted

16. Image Segmentation Results

17. 1.Histogram-based segmentation 2.Region-based segmentation Edge detection Region growing, spliting and merging. 3.Clustering K-means 4.Graph based clustering Main Approaches

18. Simple Example (text segmentation) Thresholding How to choose threshold value? (128/256, median/mean etc…)

19. Break images into K regions. Reducing intensity values into K different levels. Threshold value Histogram-based Methods

20. Consider the image as a set of points in N-dimensional feature space: 1.Intensity values or color values [ (x, y, I) or (x, y, r, g, b) ] 2.Texture and other features Segmentation as a clustering problem Work directly in the feature space and cluster these points in the feature space. Require: 1.A good definition of feature space 2.Distance between feature points should be meaningful

22. What is the “correct” grouping?

23. Segmentation as a clustering problem

24. Two Sub-problems

25. K-means clustering

26. Segmentation Result

27. Set of points of the feature space represented as a weighted, undirected graph, G = (V, E) The points of the feature space are the nodes of the graph. Edge between every pair of nodes. Weight on each edge, w(i, j), is a function of the similarity between the nodes i and j. Partition the set of vertices into disjoint sets where similarity within the sets is high and across the sets is low. Graph Partitioning Model

28. Weight measure (reflects likelihood of two pixels belonging to the same object) Weight function Note: the function is based on local similiarity

29. Images as Graphs

31. A graph can be partitioned into two disjoint sets by simply removing the edges connecting the two parts The degree of dissimilarity between these two pieces can be computed as total weight of the edges that have been removed More formally, it is called the ‘cut’ Global Criterion Selection

33. Minimize the cut value No of such partitions is exponential (2 N ) but the minimum cut can be found efficiently Reference: Z. Wu and R. Leahy, “An Optimal Graph Theoretic Approach to Data Clustering: Theory and Its Application to Image Segmentation”. IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 11, pp. 1101-1113, Nov. 1993. Subject to the constraints: Optimization Problem

35. Picking an appropriate criterion to minimize which would result in a “good” segmentation Finding an efficient way to achieve the minimization Challenges

36. Set of points in the feature space with similarity (relation) defined for pairs of points. Problem: Partition the feature points into disjoint sets where similarity within the sets is high and across the sets is low. Construct a complete graph V, and nodes are the points. Edge between every pair of nodes. Weight on each edge, w(i, j), is a function of the similarity between the nodes i and j. A cut (of V) gives the partition. Graph Partitioning Model

37. Minimize the cut value No of such partitions is exponential (2 N ) but the minimum cut can be found efficiently Reference: Z. Wu and R. Leahy, “An Optimal Graph Theoretic Approach to Data Clustering: Theory and Its Application to Image Segmentation”. IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 11, pp. 1101-1113, Nov. 1993. Subject to the constraints: Optimization Problem

38. Weights defined as W ij = exp (-|s i -s j | 2 /2  2 )

39. We must avoid unnatural bias for partitioning out small sets of points Normalized Cut - computes the cut cost as a fraction of the total edge connections to all the nodes in the graph where Normalized Cut J. Shi and J. Malik, “Normalized Cuts and Image Segmentation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888-905, Aug. 2000.

40. Our criteria can also aim to tighten similarity within the groups Minimizing Ncut and maximizing Nassoc are equivalent (2-Nassoc = Ncut) Normalized Cut

41. Tighter cluster Weights defined as W ij = exp (-|s i -s j | 2 /2  2 ) Larger Nassoc(A, B) value reflects tigher cluster. Smaller Nassoc(A, B) value

42. Let e be an indicator vector (of dimension N ): e = 1, if i belongs to A 0,otherwise Assoc(A, A) = e T We Assoc(A, V) = e T De Cut(A, V-A) = e T (D – W)e Matrix Formulation Find two indicator vectors e 1, e 2, such that (e 1 t e 2 =0) is minimized.

43. Exact solution to minimizing normalized cut is an NP-complete problem However, approximate discrete solutions can be found efficiently Normalized cut criterion can be computed efficiently by solving a generalized eigenvalue problem Computational Issues

44. Relaxation

45. 1.Construct the weighted graph representing the image. Summarize the information into matrices, W and D. Edge weight is an exponential function of feature similarity as well as distance measure. 2. Solve for the eigenvector with the second smallest eigenvalue of: Lx=(D – W)x = Dx (L = D-W) Algorithm (for image segmentation)

46. 3. Partition the graph into two pieces using the second smallest eigenvector. Signs tell us exactly how to partition the graph. 4. Recursively run the algorithm on the two partitioned parts. Recursion stops once the Ncut value exceeds a certain limit. This maximum allowed Ncut value controls the number of groups segmented. Algorithm (cont.)

48. Example

50. Results

52. K-way Normalized Cut One can recursively apply normalized cut to obtain desired number of subgraphs. Better still, we can do the following. Let e 1, e 2, … e K denote the indicator vectors of K-partition of the graph. We want to maximize this cost function. The trick is to rewrite this optimization problem

53. K-way Normalized Cut Proposition (Bach-Jordan 2003) C W (e 1, …, e k ) equals to tr ( Y t D -1/2 W D -1/2 Y ) for any N-by-K matrix Y such that (N is the number of data points ) 1.Columns of D -1/2 Y are piecewise constant with respect to the clusters 2.Y has orthonormal columns (Y t Y = I). Why this? Because we reduce (and relax) the problem to a very solvable form (appears frequently) Maximize tr ( Y t M Y) subject to Y t Y = I (with M symmetric!) Answer: Y=K-largest eigenvectors of M.

54. Spectral Clustering (Ng, Weiss&Jordan, 2001) Using spectral method (i.e., eigenstructure of some symmetric matrix) to computr clusters. Given a set of points s 1, s 2,... s n 1.Form affinity matrix A (W before) A ij = exp (-|s i -s j | 2 /2  2 ) if i  j, and A ij =0. 2.D the diagonal matrix whose (i, i)-element is the sum of A’s i-th row, and let L=D -1/2 A D -1/2. 3.Find x 1, x 2,. . ., x k the k largest eigenvectors of L and form X=[x 1 x 2. . . x k ]. 4.Renormalizing each of X’s rows to have unit length 5.Treating each row of Y as a point in R k, cluster them into k clusters via K- means or other algorithm. 6.S i is assigned to cluster j if and only if row i of Y was assigned to cluster j.

55. Spectral Clustering In the ideal case when there is no inter-cluster affinity, the matrix A is block-diagonal (after some row, column permutation). Computed Affinity matrix

56. Spectral Clustering Each block has 1 as its largest eigenvalue and the top K eigenvectors are all 1. X = [ X 1 X 2 X 3 … X K ] X’s rows (after normalization) give you a ‘projection’ of the data points in the sphere in R K.

57. Spectral Clustering

58. K-Means Spectral Technique