Graph-based Segmentation 02/25/10 Graph-based Segmentation Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem

Last class Gestalt cues and principles of organization Mean-shift segmentation Good general-purpose segmentation method Generally useful clustering, tracking technique Watershed segmentation Good for hierarchical segmentation Use in combination with boundary prediction

Today’s class Treating the image as a graph Recap Normalized cuts segmentation MRFs Graph cuts segmentation Recap Go over HW2 instructions

Images as graphs Fully-connected graph node for every pixel wij c j Fully-connected graph node for every pixel link between every pair of pixels, p,q similarity wij for each link Source: Seitz

Similarity matrix Increasing sigma

Segmentation by Graph Cuts w A B C Break Graph into Segments Delete links that cross between segments Easiest to break links that have low cost (low similarity) similar pixels should be in the same segments dissimilar pixels should be in different segments Source: Seitz

Cuts in a graph B A Link Cut set of links whose removal makes a graph disconnected cost of a cut: One idea: Find minimum cut gives you a segmentation fast algorithms exist for doing this Source: Seitz

But min cut is not always the best cut...

Cuts in a graph a cut penalizes large segments B A Normalized Cut a cut penalizes large segments fix by normalizing for size of segments volume(A) = sum of costs of all edges that touch A Source: Seitz

Recursive normalized cuts Given an image or image sequence, set up a weighted graph: G=(V, E) Vertex for each pixel Edge weight for nearby pairs of pixels Solve for eigenvectors with the smallest eigenvalues: (D − W)y = λDy Use the eigenvector with the second smallest eigenvalue to bipartition the graph Note: this is an approximation 4. Recursively repartition the segmented parts if necessary http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf Details:

Normalized cuts results

Normalized cuts: Pro and con Pros Generic framework, can be used with many different features and affinity formulations Provides regular segments Cons Need to chose number of segments High storage requirement and time complexity Bias towards partitioning into equal segments Usage Use for oversegmentation when you want regular segments

Graph cuts segmentation

Markov Random Fields Node yi: pixel label Edge: constrained pairs Cost to assign a label to each pixel Cost to assign a pair of labels to connected pixels

Markov Random Fields Example: “label smoothing” grid Unary potential 0: -logP(yi = 0 ; data) 1: -logP(yi = 1 ; data) Pairwise Potential 0 1 0 0 K 1 K 0

Solving MRFs with graph cuts Source (Label 0) Cost to assign to 0 Cost to split nodes Cost to assign to 1 Sink (Label 1)

Solving MRFs with graph cuts Source (Label 0) Cost to assign to 0 Cost to split nodes Cost to assign to 1 Sink (Label 1)

Grab cuts and graph cuts Magic Wand (198?) Intelligent Scissors Mortensen and Barrett (1995) GrabCut User Input Result Basic idea is to combine the information which was used in the 2 well known tools: MW and IS. MS select a region of similar colour according to your input. Fat pen and the boundary snapps to high contrast edges. Grabcut combines it. And this allows us to simplify the user interface considerably by … Regions Boundary Regions & Boundary Source: Rother

Colour Model R R G G Gaussian Mixture Model (typically 5-8 components) Iterated graph cut Foreground & Background Foreground Color enbergy. Iterations have the effect of pulling them away; D is – log likelyhood of the GMM. Background G Background G Gaussian Mixture Model (typically 5-8 components) Source: Rother

Graph cuts Boykov and Jolly (2001) Foreground (source) Image Min Cut Cut: separating source and sink; Energy: collection of edges Min Cut: Global minimal enegry in polynomial time Optimization engine we use Graph Cuts. By now everybody should know what graph cut is. IN case you missed it here is a very brief introduction. To image. 3D view. First task to construct a graph. All pixels on a certain scanline. Just a few of them. Next step introduce artificial nodes. fgd and background. Edges – contrast. Boundary is quite likely between black and white. Min Cut - Minimum edge strength. Background (sink) Source: Rother

Graph cuts segmentation Define graph usually 4-connected or 8-connected Define unary potentials Color histogram or mixture of Gaussians for background and foreground Define pairwise potentials Apply graph cuts Return to 2, using current labels to compute foreground, background models

Moderately straightforward examples Moderately straightforward examples- after the user input automnatically … GrabCut completes automatically GrabCut – Interactive Foreground Extraction 10

Difficult Examples Camouflage & Low Contrast Fine structure Harder Case Initial Rectangle Initial Result You might wonder when does it fail. 4 cases. Low contrats – an edge not good visible GrabCut – Interactive Foreground Extraction 11

Using graph cuts for recognition TextonBoost (Shotton et al. 2009 IJCV)

Using graph cuts for recognition Unary Potentials Alpha Expansion Graph Cuts TextonBoost (Shotton et al. 2009 IJCV)

Limits of graph cuts Associative: edge potentials penalize different labels If not associative, can sometimes clip potentials Approximate for multilabel Alpha-expansion or alpha-beta swaps Must satisfy

Graph cuts: Pros and Cons Very fast inference Can incorporate recognition or high-level priors Applies to a wide range of problems (stereo, image labeling, recognition) Cons Not always applicable (associative only) Need unary terms (not used for generic segmentation) Use whenever applicable

Further reading and resources Normalized cuts and image segmentation (Shi and Malik) http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf N-cut implementation http://www.seas.upenn.edu/~timothee/software/ncut/ncut.html Graph cuts http://www.cs.cornell.edu/~rdz/graphcuts.html Classic paper: What Energy Functions can be Minimized via Graph Cuts? (Kolmogorov and Zabih, ECCV '02/PAMI '04)

Recap of Grouping and Fitting

Line detection and Hough transform Canny edge detector = smooth  derivative  thin  threshold  link Generalized Hough transform = points vote for shape parameters Straight line detector = canny + gradient orientations  orientation binning  linking  check for straightness

Robust fitting and registration Key algorithm RANSAC

Clustering Key algorithm Kmeans

EM and Mixture of Gaussians Tutorials: http://www.cs.duke.edu/courses/spring04/cps196.1/.../EM/tomasiEM.pdfhttp://www-clmc.usc.edu/~adsouza/notes/mix_gauss.pdf

Segmentation Mean-shift segmentation Watershed segmentation Flexible clustering method, good segmentation Watershed segmentation Hierarchical segmentation from soft boundaries Normalized cuts Produces regular regions Slow but good for oversegmentation MRFs with Graph Cut Incorporates foreground/background/object model and prefers to cut at image boundaries Good for interactive segmentation or recognition

Next section: Recognition How to recognize Specific object instances Faces Scenes Object categories Materials