Graph Cut & Energy Minimization Yevgeny Doctor IP Seminar 2008, IDC

Yevgeny Doctor, March 2008, IDC Outline Motivation Graph Cuts Energy Minimization in Computer Vision Solving Energy Minimization with Graph Cuts Results Yevgeny Doctor, March 2008, IDC

Yevgeny Doctor, March 2008, IDC Graph Cut G(V,E) is a finite directed graph and every edge (u,v) has a capacity c(u,v) (a non-negative real number). Assume two vertices, the source s and the sink t, have been distinguished. A cut is a split of the nodes into two sets S and T, such that s is in S and t is in T. t s Yevgeny Doctor, March 2008, IDC

Yevgeny Doctor, March 2008, IDC Min-Cut The capacity of a cut (S,T ) is defined as Min-Cut – finding the cut with the minimal capacity Yevgeny Doctor, March 2008, IDC

Yevgeny Doctor, March 2008, IDC Max Flow Find the maximum flow from s to t Yevgeny Doctor, March 2008, IDC

Yevgeny Doctor, March 2008, IDC Min-Cut Max-Flow Min-Cut and Max-Flow are equivalent problems Fast polynomial algorithms for max flow exist For example: Ford-Fulkerson algorithm – O(E*maxflow) Edmonds-Karp algorithm – O(VE2) Yevgeny Doctor, March 2008, IDC

Energy Minimization Problems In general terms, given some problem, we: Formulate the known constraints Build an “energy function” (aka “cost function”) Look for a solution that minimizes it If we have no further knowledge: The problem can be NP-Hard (requires exponential solution time) Use slow, generic approximation algorithms for optimization problems (such as simulated annealing) Yevgeny Doctor, March 2008, IDC

Yevgeny Doctor, March 2008, IDC EM in Computer Vision Consider a broad class of problems called Pixel Labeling Given some images we want to “say something about the pixels” For each pixel p, give it a label fp from a finite set of labels L, such that we minimize some energy function. Many applications Image Segmentation Image Restoration Stereo and Motion Medical Imaging Multicamera Scene Reconstruction Yevgeny Doctor, March 2008, IDC

Pixel Labeling Example "Graph Cut Matching In Computer Vision", Toby Collins, February 2004 http://homepages.inf.ed.ac.uk/rbf/CVonline/ Yevgeny Doctor, March 2008, IDC

Yevgeny Doctor, March 2008, IDC EM in Computer Vision Consider a specific family of Energy Functions Powerful enough to formulate many useful problems Can be reduced to solving a graph min-cut problem Problems defined with these functions: Can be solved quickly (using max-flow algorithms) In many cases – optimal solution or within a known factor of the optimum Yevgeny Doctor, March 2008, IDC

Energy Function Definition Edata(f) Esmooth(f) Input: set of pixels P, set of labels L, is a neighbourhood system on pixels. Goal: find a labeling that minimizes . . is a function derived from the observed data that measures the cost of assigning label to pixel p. . measures the cost of assigning the labels to adjacent pixels p, q. Used to impose spatial smoothness. Yevgeny Doctor, March 2008, IDC

Energy Function - Edata Component The Edata(f) component: Look at each pixel independently Given it’s current value, what would it cost to label it with each of the labels? Examples: Cost based on a-priori known pixel intensity or color distribution is the observed intensity of pixel p What if we used only this component in E(f)? Label each pixel independently with the most likely (cheap) label Yevgeny Doctor, March 2008, IDC

Optimizing Edata Only – Illustration What would be the problem? For example (object segmentation): Images from “Nearest Neighbor” lecture – here for illustration purposes only We need to add a “smoothness” cost Yevgeny Doctor, March 2008, IDC

Energy Function - Esmooth Component Look at all pairs of neighbor pixels Penalize adjacent pixels with different labels What smoothness cost function to use? Noised diamond image Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih Yevgeny Doctor, March 2008, IDC

Smoothness Cost Functions Potts Interaction Penalty: T – indicator function, K – constant The solution will be piecewise constant, with discontinuities at the boundaries Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih

Smoothness Cost Functions L2 distance: What is the problem? High penalties at object boundaries We want smooth objects, but allow different labels at object boundary – a discontinuity-preserving function. Yevgeny Doctor, March 2008, IDC

Smoothness Cost Functions Truncated L2 distance: K – constant The solution will be piecewise smooth, with discontinuities at the boundaries Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih

Smoothness Cost Functions Normalize for neighbor distance, image contrast: This function penalizes a lot for discontinuities between pixels of similar intensities when |Ip − Iq| < s. However, if pixels are very different, |Ip − Iq| > s, then the penalty is small. Yevgeny Doctor, March 2008, IDC

Solving Energy Minimization with Graph Cuts Many classes of Energy Minimization problems in Computer Vision can be reduced to Graph Cuts Solve multiple-labels problems with binary decisions Yevgeny Doctor, March 2008, IDC

Approximate Energy Minimization “Fast Approximate Energy Minimization via Graph Cuts.” Yuri Boykov, Olga Veksler, Ramin Zabih, 1999 For two classes of interaction potentials V (Esmooth): V is semi-metric on a label space L if for every : V is metric on L if in addition, triangle inequality holds: For example, truncated L2 distance and Potts Interaction Penalty are both metric. Yevgeny Doctor, March 2008, IDC

Solution for Semi-metric Class Swap-Move algorithm: 1. Start with an arbitrary labeling f 2. Set success := 0 3. For each pair of labels 3.1. Find f* = argmin E(f') among f' within one a-b swap of f 3.2. If E(f*) < E(f), set f := f* and success := 1 4. If success = 1 goto 2 5. Return f a-b swap: In the new labeling f’, some pixels that were labeled a in f are now labeled b, and vice versa. Yevgeny Doctor, March 2008, IDC

Solve a-b swap step with Graph Cut Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih, 1999

Solve a-b swap step with Graph Cut Cut and Labeling: Weights: Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih, 1999

Solution for Metric Class Expansion Move algorithm: 1. Start with an arbitrary labeling f 2. Set success := 0 3. For each label 3.1. Find f* = argmin E(f') among f' within one a expansion of f 3.2. If E(f*) < E(f), set f := f* and success := 1 4. If success = 1 goto 2 5. Return f a expansion: In the new labeling some pixels change their label to a. Yevgeny Doctor, March 2008, IDC

Expansion Move Algorithm Example: Guaranteed approximation ratio by the algorithm: Produces a labeling f such that , where f* is the global minimum and . "What Energy Functions Can Be Minimized via Graph Cuts?", Vladimir Kolmogorov and Ramin Zabih, February 2004 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 2. Yevgeny Doctor, March 2008, IDC

Solve a expansion step with Graph Cut Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih, 1999

Solve a expansion step with Graph Cut Cut and Labeling: Weights: Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih, 1999

Yevgeny Doctor, March 2008, IDC Experimental Results Three energy functions: Truncated quadratic – semi-metric (swap-move): Potts model – metric (a-expansion): Truncated L2 distance – metric (a-expansion): Compared with: Best simulated annealing algorithm Yevgeny Doctor, March 2008, IDC

Yevgeny Doctor, March 2008, IDC Convergence Rate Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih, 1999

Yevgeny Doctor, March 2008, IDC Image Restoration Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih, 1999

Yevgeny Doctor, March 2008, IDC Motion and Stereo Yevgeny Doctor, March 2008, IDC Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler, Ramin Zabih, 1999

What Energy Functions Can Be Minimized via Graph Cuts? A precise characterization of the class of functions in F3 that can be minimized using graph cuts A general purpose graph construction for minimizing any function in this class A necessary condition for any function of binary variables to be minimized via graph cuts. "What Energy Functions Can Be Minimized via Graph Cuts?", Vladimir Kolmogorov and Ramin Zabih, February 2004 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 2. Yevgeny Doctor, March 2008, IDC

What Energy Functions Can Be Minimized via Graph Cuts? Define general Energy Function classes: Given F2: F3: F2 is contained in F3 "What Energy Functions Can Be Minimized via Graph Cuts?", Vladimir Kolmogorov and Ramin Zabih, February 2004 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 2. Yevgeny Doctor, March 2008, IDC

What Energy Functions Can Be Minimized via Graph Cuts? Regular F2 functions: "What Energy Functions Can Be Minimized via Graph Cuts?", Vladimir Kolmogorov and Ramin Zabih, February 2004 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 2. Yevgeny Doctor, March 2008, IDC

What Energy Functions Can Be Minimized via Graph Cuts? Graph construction for F2: "What Energy Functions Can Be Minimized via Graph Cuts?", Vladimir Kolmogorov and Ramin Zabih, February 2004 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 2. Yevgeny Doctor, March 2008, IDC

What Energy Functions Can Be Minimized via Graph Cuts? Regular F3 functions: A function E of more than two variables is called regular if all projections of E of two variables are regular And Generally: Let E be a function of binary variables. If E is not regular, then E is not graph-representable. Yevgeny Doctor, March 2008, IDC

GrabCut "Interactive Foreground Extraction using Iterated Graph Cuts" Carsten Rother, Vladimir Kolmogorov, Andrew Blake, 2004 Microsoft Research Cambridge, UK Yevgeny Doctor, March 2008, IDC

Yevgeny Doctor, March 2008, IDC GrabCut Yevgeny Doctor, March 2008, IDC

Yevgeny Doctor, March 2008, IDC GrabCut Transparency: Yevgeny Doctor, March 2008, IDC

Yevgeny Doctor, March 2008, IDC GrabCut Yevgeny Doctor, March 2008, IDC

"Video Object Cut and Paste" Video Cutout "Video Object Cut and Paste" Yin Li, Jian Sun, Heung-Yeung Shum, 2005 Microsoft Research Asia Figure 1: Given a video (a) containing an opaque video object on a complicated background, our system can generate clear alpha mattes and foreground colors (b) for the video object, which can be pasted onto another background. Yevgeny Doctor, March 2008, IDC

Yevgeny Doctor, March 2008, IDC Video Cutout 3D segmentation of spatio-temporal video volume Partition into foreground and background, while preserving temporal coherence Local refinement of initial segmentation Provide brush tools for the user to control the object boundary precisely wherever needed Apply coherent matting to extract the alpha mattes and foreground colors of the object based on the accurate binary segmentation result Yevgeny Doctor, March 2008, IDC

Yevgeny Doctor, March 2008, IDC Video Cutout Yevgeny Doctor, March 2008, IDC

Yevgeny Doctor, March 2008, IDC Video Cutout Yevgeny Doctor, March 2008, IDC

Yevgeny Doctor, March 2008, IDC Video Cutout Yevgeny Doctor, March 2008, IDC

Yevgeny Doctor, March 2008, IDC Video Cutout Figure 7: Clip #2, frame 27. (a) 3D graph cut result is shown by the overlaid dashed line. The flag is a rapidly deforming object, but 3D graph cut can apture the shape very well. (b) Dashed lines indicate the boundaries after both the local refinement and overriding operations. The white pixels record the actions of foreground brushing and the black pixels for background brushing. (c) coherence matting result pasted on a blue screen. Yevgeny Doctor, March 2008, IDC