1. Graph Cuts based segmentation methods: Daniel Heilper, CS Department, Haifa University

2. Topics: Segmentation as Graph cuts problems. MIN-CUT and Normalized CUT approaches Examples of Cost/Energy functions Performance issues – feasibility and approximations Relations to Clustering and other methods

3. Motivation: Human perception takes into account several perceptual aspects and “blends” them together: Proximity:

4. Human perception Proximity: Closure:

5. Human perception Similarity Continuation

6. Early segmentation methods Boundary based methods - based on local information and cannot guarantee continuous closed edge contours (Derivatives kernels, Harris, Canny) Region based methods + statistics inside the region. - often generates irregular boundaries and small Holes.

7. Boundary based methods Original text Canny:

8. Boundary based methods Derivatives:

9. Region based methods

10. Global optimization approaches based on energy functions Construct a heuristic energy function which take into account both the local information and the global information.

11. Graph Representation: The set of points in an arbitrary feature space are represented as a weighted undirected graph G = (V,E,w) where the nodes of the graph are the points in the feature space, and an edge is formed between every pair of nodes. The weight on each edge, Wij is a function of the similarity between nodes i and j.

12. Graph Representation:

13. Energy cost functions: Energy Function is heuristic for quantization of a combination of Data Features on an N-D Image. Simple Examples: Distance metric – Image is sent as a binary image, graph is represented as points in the plane

14. Energy functions for Graph Cuts In some cases, energy minimization is NP-Hard, or has high polynomial order which make the method impractical Choosing a good energy function is not “Pure” science

15. Energy functions for Graph Cuts The most Natural Energy form is: V - Boundary Penalty caused by choosing labels for 2 pixels D – Regional Penalty for assigning a pixel to some label.

17. First approach: Max-Flow Min-Cut Refresher: Flow Networks:

18. Network flow graphs Given A flow Networks where: is a directed graph is a source (not necessary belongs to G) is a sink (not necessary belongs to G) a non-negative capacity function on the edges (if we can assume )

19. Network flow graphs Function is a valid flow function if holds: Capacity constraints:, counter symmetry: for each, flow conservation:, where, are the edge set that leave V. Similarly are the sedge set that enters V

20. Network flow graphs The value of the flow function is

21. Network flow graphs

23. Max-Flow – Min-Cut Theorem: The maximum amount of flow is equal to the capacity of a minimum cut

24. Interactive Segmentation application using Graph Cuts: Motivation: for some applications: Fully automatic segmentation does not give sufficient results. for example, in medical diagnostics, Doctors handlings a medical scan images where soft tissues, blood vessels, and bones are “blended” together, are very difficult to segment.

25. Above: CAT scan of Pancreas.

26. Interactive Segmentation application using Graph Cuts In a nutshell: User marks some pixels as seeds of “Background” and “Object”, and runs the application several times until a satisfactory result achieved. Interactive inputs: “seeds” pixels the ratio between regional importance VS boundary importance Energy function (Intensities, distances, textures, etc..)

27. Seeds in the flow graph

28. Interactive Segmentation application using Graph Cuts First run, application fully solves max-flow task Each time the application outputs an insufficient result, the user adds more “seeds”, (changes the ratio ) and rerun the application. The segmentation is improved until satisfactory by user. Only the first run is computationally heavy, while the consecutive runs are fast, and can be seen as an addition to capacities, and hence finding additional “augmenting paths” in the flow function

29. Past Interactive tools for segmentation: intelligent scissors live wire “shortest path” between certain pixels using some energy function Difficulties: the user inputs have to be very accurately positioned at the desired boundary can not be easily generalized to 3D data.

30. Energy function: General form of the energy function: Where

31. Energy function: R(A) stands for the sum of the regional penalties of a given labeling Similarly B(A) is the sum of the boundry penalties of the labeling

33. 3D - Data Video sequence is treated as a 3D volume image:

35. Second Method: Normalized Cut: The normalized cut criterion measures both the total dissimilarity between the different groups as well as the total similarity within the groups. The key point is that image partitioning is to be done from the big picture downward, rather like a painter first marking out the major areas and then filling in the details.

36. Normalized Cut Unfortunately, minimizing normalized cut exactly is NP-complete, even for the special case of graphs on grids(Papadimitriou). However, an approximate discrete solution can be found efficiently

37. MINCut problem Why minimum cut isn’t always favorable? the minimum cut criteria favors cutting small sets of isolated nodes in the graph. (for example when cost function is inverse to the distance between nodes)

38. A case where minimum cut gives a bad partition.

39. Definition: Normalized cut where assoc(A,V) is the total connection from nodes in A to all nodes in the graph

40. Normalized cut It is easy to see that single nodded cuts produce MINCut value of at least 1, so minimum normal cut is not likely to be single nodded.

41. Definition: Normalized Association: In the same spirit, we can define a measure for total normalized association within groups for a given partition:

42. NAssoc vs NCut A direct conclusion: NCut(A,B) = 2 – Nassoc(A,B) Ncut is normalized measure of disassociation minimizing Ncut is equivalent to maximizing association

43. Computing the Optimal Partition An approximate discrete solution can be found efficiently by converting the problem into a generalized eigenvector problem

44. NCut as Eigenvector system Matrix representation W is the cost Matrix D is the sum of the costs from node i: The Partition is a binary matrix size |P|XK where |P| is the number of pixels and K is the number of labels

45. NCut as Eigenvector system The Segmentation matrix y: Each column is a label vector NCut can be written as:

46. Recursive two-way Ncut grouping algorithm 1. Given an image or image sequence, set up a weighted graph G=(V, E) and set the weight on the edge connection two nodes to be a measure of the similarity between the two nodes. 2. Solve for eigenvectors with the smallest eigenvalues

47. Recursive two-way Ncut grouping algorithm 3. Use the eigenvector with the second smallest eigenvalue to bipartition the graph 4. Decide if the current partition should be subdivided and recursively repartition the segmented parts if necessary.

48. Experiments on energy functions: where X(i) is the spatial location of node i, and F(i) is a feature vector based on intensity, color, or texture information If can separate the regional weight from the boundary weight using log operation

49. (a) A synthetic image showing a noisy ªstepº image. Intensity varies from 0 to 1, and Gaussian noise with sigma = 0.2 is added. (b) eigenvector with the second smallest eigenvalue (c) resulting partition.

50. Performance and optimality Solving an eigenvector system takes at least (|P|^3) time Using iterative power method we can find the second lowest eigenvector in a O(|P|*|n|) where |P| is #pixels, and |n| is #iterative steps The NCut at the end of the algorithm has proven constant proximity to the MIN NCut

51. Finding the Minimum Normalized Cut VS interactive MINCut finding: Min NCut Pros: Avoids segmenting isolated small sections (VS “blind” calculation of MINCut) This NP-Hard can be well approximated Min NCut cons: Difficult to control and analyse

52. Finding the Minimum Normalized Cut VS interactive MINCut finding interactive MINCut Pros: Allows a fast correction of the result in an interactively manner, and converges Easy to control

53. Energy Minimization Techniques for Graph Cuts Goal: giving a faster algorithm to the MIN- Cut problem or a general K-Labeling partition of an Image Some energy functions makes the MinCut problem to be NP-Hard Other functions reach optimal solution in impractical polynomial time

54. Known algorithms for energy minimization: Simmulated Anealing – converges well but for small temperature delta’s is exponential in time, and for large deltas it converges to local minimum which may be far from the minimum Combinatorial Max flow – Min cut algorithms. Combinatorial algorithms solves the problem in at least O(|P|^3) time (Tarjan & Goldberg). dynamic programming – restricted to 1-D energy function

55. linear time algorithm for finding the Max-Flow-Min-Cut/ Updating the labeling function: two types of large moves: α-expansion and α-β-swap

56. α-Exapnsion Take a subset of pixels which are not labeled to have that label Works for a metric energy function:

57. α-β swap Changing a label α of some pixels to label β Works for semimetric Energy functions:

58. α-Exapnsion

59. α-Exapnsion and {α, β}–Swap

60. these moves allow large number of pixels to change their labels simultaneously Note that a move which assigns a given label α to a single pixel is both an α-β swap and an α-expansion. (α-β swap can be seen as a private case of α-Exapnsion)

61. { α} Expansion 1. Start with an arbitrary labeling f 2. Set success := 0 3. For each label α belongs L 3.1. Find ˆ f = argminE(f0) among f0 within one α-expansion of f 3.2. If E(f*) < E(f), set f := f* and success := 1 4. If success = 1 goto 2 5. Return f

62. {α, β} – Swap algorithm 1. Start with an arbitrary labeling f 2. Set success := 0 3. For each pair of labels {α, β} belongs to L 3.1. Find f* = argminE(f’) among f’ within one α-β swap of f 3.2. If E(f*) < E(f), set f := f* and success := 1 4. If success = 1 goto 2 5. Return f

63. Algorithms summary A cycle is successful if a strictly better labeling is found at any iteration. The algorithms stop after the first unsuccessful cycle since no further improvement is possible. α-Exapnsion much faster than α-β swap : |L| VS o(|L|^2) iterations (L is the labeling)

66. Benchmark results (Results of running the algorithms on the previous image)

67. References: [1] J. Shi and J.Malik, “Normalized Cuts and Image Segmentation,” Proc. CVPR 1997. [2] Boykov, Y., Jolly, M., " Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images." In: International Conference on Computer Vision, Vancouver, BC. (2001) 105–112 [3] Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23 (2001) 1222–1239 [4] Goro Obinata, Ashish Dutta: Vision Systems: Segmentation and Pattern Recognition, I-Tech Education and Publishing, Vienna, Austria