GrabCut Interactive Image (and Stereo) Segmentation Carsten Rother Vladimir Kolmogorov Andrew Blake Antonio Criminisi Geoffrey Cross [based on Siggraph 2004 and ECCV 2004] Welcome. I will present Grabcut – an Interactive tool for foreground extraction.

Photomontage GrabCut – Interactive Foreground Extraction 1 Here are some images from my last hiking trip in England. And would it be great if we could just do the following. Drag rectangle. Create a nice photomontage with an extreamly simple user interface. GrabCut – Interactive Foreground Extraction 1

Talk Outline Hard Image Segmentation: Fore- vs. Background Soft Segmentation: Alpha Matting Stereo Segmentation: Exploit Depth GrabCut – Interactive Foreground Extraction 2

Problem Fast & Accurate ? State the problem. Image object to extract and simple tool which gives high quality results with an alpha matte and even works for the case of camoflage. GrabCut – Interactive Foreground Extraction 3

Intelligent Scissors Mortensen and Barrett (1995) What GrabCut does Magic Wand (198?) Intelligent Scissors Mortensen and Barrett (1995) GrabCut User Input Result Basic idea is to combine the information which was used in the 2 well known tools: MW and IS. MS select a region of similar colour according to your input. Fat pen and the boundary snapps to high contrast edges. Grabcut combines it. And this allows us to simplify the user interface considerably by … Regions Boundary Regions & Boundary GrabCut – Interactive Foreground Extraction 4

Framework Input: Image Output: Segmentation Parameters: Colour ,Coherence Energy: Optimization: Let us phrase this segmentation task in a theoretical framework. We will phrase the segmentation problem as an energy minimization problem. Minimum energy corresponds to maximum probability of a Gibbs distribution. Eventually also over lambda. Boils down to GrabCut – Interactive Foreground Extraction 5

Energy – Probabilistic View Gibbs Distribution of the MRF Maximum a posteriori estimator (MAP): - log same as GrabCut – Interactive Foreground Extraction 6

Graph Cuts - Boykov and Jolly (2001) Foreground (source) Image Min Cut Cut: separating source and sink; Energy: collection of edges Min Cut: Global minimal enegry in polynomial time Optimization engine we use Graph Cuts. By know everybody should know what graph cut is. IN case you missed it here is a very brief introduction. To image. 3D view. First task to construct a graph. All pixels on a certain scanline. Just a few of them. Next step introduce artificial nodes. fgd and background. Edges – contrast. Boundary is quite likely between black and white. Min Cut - Minimum edge strength. Background (sink) GrabCut – Interactive Foreground Extraction 7

? Iterated Graph Cut User Initialisation We extended there work in the following way. IGC – Technique like EM – switch between 2 optimization problems. GC does not consider all other parameters. Init by user. Graph cuts to infer the segmentation K-means for learning colour distributions GrabCut – Interactive Foreground Extraction 8

Guaranteed to converge Iterated Graph Cuts Guaranteed to converge Iterations are not shown to the user; Converges: proof in the paper. 1 2 3 4 Result Energy after each Iteration GrabCut – Interactive Foreground Extraction 9

Colour Model R R G G Gaussian Mixture Model (typically 5-8 components) Iterated graph cut Foreground & Background Foreground Color enbergy. Iterations have the effect of pulling them away; D is – log likelyhood of the GMM. Background G Background G Gaussian Mixture Model (typically 5-8 components) GrabCut – Interactive Foreground Extraction 10

Coherence Model An object is a coherent set of pixels: 25 Also coherent model. Strength of contrast – colour difference. Lambda importance of coherence model. 25 Error (%) over training set: How do we choose ? 25

Parameter Learning (Blake 2004) Gaussian MRF: approximation = Pseudo-Likelihood: Linear regression gives in closed-form GrabCut – Interactive Foreground Extraction 12

Parameter Learning - Problems A Gaussian MRF is not a realistic texture model syntheticGMRF Gaussian? Real Image Gaussian! GrabCut – Interactive Foreground Extraction 13

Moderately simple examples Moderately straightforward examples- after the user input automnatically … GrabCut completes automatically GrabCut – Interactive Foreground Extraction 14

Difficult Examples Camouflage & Low Contrast Fine structure No telepathy Initial Rectangle Initial Result You might wonder when does it fail. 4 cases. Low contrats – an edge not good visible GrabCut – Interactive Foreground Extraction 15

Evaluation – Labelled Database Labeled data base 70 images . availoable online. Different scenarios – simple and ifficult shapes and colour. Available online: http://research.microsoft.com/vision/cambridge/segmentation/ GrabCut – Interactive Foreground Extraction 16

Comparison Boykov and Jolly (2001) GrabCut Error Rate: 0.72% User Input Error Rate: 0.72% Databsed used to compare to BJ. Re-implented their method. Simpler same error rate Result Error Rate: 0.72% Error Rate: 1.32% Error Rate: 1.87% Error Rate: 1.81% Error Rate: 1.25% GrabCut – Interactive Foreground Extraction 17

Trimap Boykov and Jolly Comparison Input Image Ground Truth Trimap Boykov and Jolly Error Rate: 1.36% Bimap GrabCut Error Rate: 2.13% Use our database. Small uncertainty area. Error rate - modestly increase User Interactions - considerable reduced GrabCut – Interactive Foreground Extraction 18

Results Parameter Learning GrabCut – Interactive Foreground Extraction 19

Comparison Magic Wand (198?) Finally – number of tools: certain scenarios. what does the user want to do for segmentation – Graph Cut – user input reduced and quality increases. Magic Wand (198?) Intelligent Scissors Mortensen and Barrett (1995) Graph Cuts Boykov and Jolly (2001) LazySnapping Li et al. (2004) GrabCut Rother et al. (2004) GrabCut – Interactive Foreground Extraction 20

Digital Matting “Mixed pixels”: Combination of fore- and background Alpha Mask: Proportion of fore- and background Natural Matting Problem: Determine alpha,F,B from C Under-determined System: 3 Equations and 7 unknowns GrabCut – Interactive Foreground Extraction 21

Transparency - Taxonomie GrabCut Existing Methods 1. Simple Alpha & Simple Colour 2. Difficult Alpha & Simple Colour Human ? 3. Simple Alpha & Difficult Colour 4. Difficult Alpha & Difficult Colour GrabCut – Interactive Foreground Extraction 22

Border Matting to Hard Segmentation Automatic Trimap Soft Segmentation So far hard segmentation. Hard Segmentation Automatic Trimap Soft Segmentation to GrabCut – Interactive Foreground Extraction 23

Comparison With no regularisation over alpha Input Problem – no regularisation over alpha. Input Bayes Matting Chuang et. al. (2001) Knockout 2 Photoshop Plug-In Shum et. al. (2004): Coherence matting in “Pop-up light fields” GrabCut – Interactive Foreground Extraction 24

Natural Image Matting Solve Mean Colour Foreground Given a pixel with a certain colour C, we have to solve the following eq. which Has the true fgd, and bkg colour and the unknown propotion in terms of alpha. Neighborhood – mean colour F, B. now eq. was under constr now overconstrained. Mean Colour Background Solve Ruzon and Tomasi (2000): Alpha estimation in natural images GrabCut – Interactive Foreground Extraction 25

Border Matting Fit a smooth alpha-profile with parameters Foreground Noisy alpha-profile 1 Mix Back-ground Fit a smooth profile – as in FEATHERING Foreground Mix Background Fit a smooth alpha-profile with parameters GrabCut – Interactive Foreground Extraction 26

Dynamic Programming DP Result using DP Border Matting Regularisation Stripes – orthogonal to the boundary. Minimize energy which says: similar sigmja mu, add a noisy data term Result using DP Border Matting Noisy alpha-profile Regularisation GrabCut – Interactive Foreground Extraction 27

GrabCut Border Matting - Colour Compute MAP of p(F|C,alpha) (marginalize over B) To avoid colour bleeding use colour stealing (“exemplar based inpainting” – Patches do not work) [Chuang et al. ‘01] Grabcut Border Matting GrabCut – Interactive Foreground Extraction 28

Results GrabCut – Interactive Foreground Extraction 29

Stereo Video + Segmentation Left Sequence Right Sequence Criminisi et. al. (2003): 4-Plane DP to handle occlusions properly Disparity Sequence GrabCut – Interactive Foreground Extraction 30

Occusion, left and right Try to get proper speed video GrabCut – Interactive Foreground Extraction 31

Background Substitution Criminisi et. al. (2004): Remove boundary artefacts (SPS algorithm) GrabCut – Interactive Foreground Extraction 32

Object Insertion GrabCut – Interactive Foreground Extraction 33

Focus on Foreground GrabCut – Interactive Foreground Extraction 34

Conclusions & Future Work GrabCut – powerful interactive extraction tool Iterated Graph Cut based on colour and contrast Regularized alpha matting by Dynamic Programming Stereo and Segmentation give supportive information How to solve the difficult hair problem ? [Argawall et.al. 2004] GrabCut – Interactive Foreground Extraction 35