1. GrabCut Interactive Image (and Stereo) Segmentation Carsten Rother Vladimir Kolmogorov Andrew Blake Antonio Criminisi Geoffrey Cross [based on Siggraph 2004 and ECCV 2004]
Welcome. I will present Grabcut – an Interactive tool for foreground extraction.

2. Photomontage GrabCut – Interactive Foreground Extraction 1
Here are some images from my last hiking trip in England. And would it be great if we could just do the following. Drag rectangle. Create a nice photomontage with an extreamly simple user interface.
GrabCut – Interactive Foreground Extraction

3. Talk Outline Hard Image Segmentation: Fore- vs. Background
Soft Segmentation: Alpha Matting
Stereo Segmentation: Exploit Depth
GrabCut – Interactive Foreground Extraction

4. Problem Fast & Accurate ?
State the problem. Image object to extract and simple tool which gives high quality results with an alpha matte and even works for the case of camoflage.
GrabCut – Interactive Foreground Extraction

5. Intelligent Scissors Mortensen and Barrett (1995)
What GrabCut does
Magic Wand (198?)
Intelligent Scissors Mortensen and Barrett (1995)
GrabCut
User Input
Result
Basic idea is to combine the information which was used in the 2 well known tools: MW and IS.
MS select a region of similar colour according to your input. Fat pen and the boundary snapps to high contrast edges.
Grabcut combines it. And this allows us to simplify the user interface considerably by …
Regions
Boundary
Regions & Boundary
GrabCut – Interactive Foreground Extraction

6. Framework Input: Image Output: Segmentation
Parameters: Colour ,Coherence
Energy:
Optimization:
Let us phrase this segmentation task in a theoretical framework.
We will phrase the segmentation problem as an energy minimization problem.
Minimum energy corresponds to maximum probability of a Gibbs distribution.
Eventually also over lambda. Boils down to
GrabCut – Interactive Foreground Extraction

7. Energy – Probabilistic View
Gibbs Distribution of the MRF
Maximum a posteriori estimator (MAP):
- log
same as
GrabCut – Interactive Foreground Extraction

8. Graph Cuts - Boykov and Jolly (2001)
Foreground (source)
Image
Min Cut
Cut: separating source and sink; Energy: collection of edges
Min Cut: Global minimal enegry in polynomial time
Optimization engine we use Graph Cuts. By know everybody should know what graph cut is. IN case you missed it here is a very brief introduction.
To image. 3D view. First task to construct a graph.
All pixels on a certain scanline. Just a few of them.
Next step introduce artificial nodes. fgd and background.
Edges – contrast. Boundary is quite likely between black and white.
Min Cut - Minimum edge strength.
Background (sink)
GrabCut – Interactive Foreground Extraction

9. ? Iterated Graph Cut User Initialisation
We extended there work in the following way. IGC – Technique like EM – switch between 2 optimization problems.
GC does not consider all other parameters. Init by user.
Graph cuts to infer the segmentation
K-means for learning colour distributions
GrabCut – Interactive Foreground Extraction

10. Guaranteed to converge
Iterated Graph Cuts
Guaranteed to converge
Iterations are not shown to the user; Converges: proof in the paper. 1 2 3 4
Result
Energy after each Iteration
GrabCut – Interactive Foreground Extraction

11. Colour Model R R G G Gaussian Mixture Model (typically 5-8 components)
Iterated graph cut
Foreground & Background
Foreground
Color enbergy.
Iterations have the effect of pulling them away;
D is – log likelyhood of the GMM.
Background G
Background G
Gaussian Mixture Model (typically 5-8 components)
GrabCut – Interactive Foreground Extraction

12. Coherence Model An object is a coherent set of pixels: 25
Also coherent model. Strength of contrast – colour difference. Lambda importance of coherence model. 25
Error (%) over training set:
How do we choose ? 25

13. Parameter Learning (Blake 2004)
Gaussian MRF:
approximation =
Pseudo-Likelihood:
Linear regression gives in closed-form
GrabCut – Interactive Foreground Extraction

14. Parameter Learning - Problems
A Gaussian MRF is not a realistic texture model
syntheticGMRF
Gaussian?
Real Image
Gaussian!
GrabCut – Interactive Foreground Extraction

15. Moderately simple examples
Moderately straightforward examples- after the user input automnatically
… GrabCut completes automatically
GrabCut – Interactive Foreground Extraction

16. Difficult Examples Camouflage & Low Contrast Fine structure
No telepathy
Initial Rectangle
Initial Result
You might wonder when does it fail. 4 cases. Low contrats – an edge not good visible
GrabCut – Interactive Foreground Extraction

17. Evaluation – Labelled Database
Labeled data base 70 images . availoable online. Different scenarios – simple and ifficult shapes and colour.
Available online:
GrabCut – Interactive Foreground Extraction

18. Comparison Boykov and Jolly (2001) GrabCut Error Rate: 0.72%
User Input
Error Rate: 0.72%
Databsed used to compare to BJ. Re-implented their method. Simpler same error rate
Result
Error Rate: 0.72%
Error Rate: 1.32%
Error Rate: 1.87%
Error Rate: 1.81%
Error Rate: 1.25%
GrabCut – Interactive Foreground Extraction

19. Trimap Boykov and Jolly
Comparison
Input Image Ground Truth
Trimap Boykov and Jolly
Error Rate: 1.36%
Bimap GrabCut
Error Rate: 2.13%
Use our database. Small uncertainty area.
Error rate - modestly increase
User Interactions - considerable reduced
GrabCut – Interactive Foreground Extraction

20. Results Parameter Learning
GrabCut – Interactive Foreground Extraction

21. Comparison Magic Wand (198?)
Finally – number of tools: certain scenarios.
what does the user want to do for segmentation –
Graph Cut – user input reduced and quality increases.
Magic Wand (198?)
Intelligent Scissors Mortensen and Barrett (1995)
Graph Cuts Boykov and Jolly (2001)
LazySnapping Li et al. (2004)
GrabCut Rother et al. (2004)
GrabCut – Interactive Foreground Extraction

22. Digital Matting “Mixed pixels”: Combination of fore- and background
Alpha Mask: Proportion of fore- and background
Natural Matting Problem: Determine alpha,F,B from C
Under-determined System: 3 Equations and 7 unknowns
GrabCut – Interactive Foreground Extraction

23. Transparency - Taxonomie
GrabCut
Existing Methods
1. Simple Alpha & Simple Colour
2. Difficult Alpha & Simple Colour
Human ?
3. Simple Alpha & Difficult Colour
4. Difficult Alpha & Difficult Colour
GrabCut – Interactive Foreground Extraction

24. Border Matting to Hard Segmentation Automatic Trimap Soft Segmentation
So far hard segmentation.
Hard Segmentation
Automatic Trimap
Soft Segmentation to
GrabCut – Interactive Foreground Extraction

25. Comparison With no regularisation over alpha Input
Problem – no regularisation over alpha.
Input
Bayes Matting Chuang et. al. (2001)
Knockout 2 Photoshop Plug-In
Shum et. al. (2004): Coherence matting in “Pop-up light fields”
GrabCut – Interactive Foreground Extraction

26. Natural Image Matting Solve
Mean Colour Foreground
Given a pixel with a certain colour C, we have to solve the following eq. which
Has the true fgd, and bkg colour and the unknown propotion in terms of alpha.
Neighborhood – mean colour F, B. now eq. was under constr now overconstrained.
Mean Colour Background
Solve
Ruzon and Tomasi (2000): Alpha estimation in natural images
GrabCut – Interactive Foreground Extraction

27. Border Matting Fit a smooth alpha-profile with parameters Foreground
Noisy alpha-profile 1
Mix
Back-ground
Fit a smooth profile – as in FEATHERING
Foreground
Mix
Background
Fit a smooth alpha-profile with parameters
GrabCut – Interactive Foreground Extraction

28. Dynamic Programming DP Result using DP Border Matting Regularisation
Stripes – orthogonal to the boundary. Minimize energy which says: similar sigmja mu, add a noisy data term
Result using DP Border Matting
Noisy alpha-profile
Regularisation
GrabCut – Interactive Foreground Extraction

29. GrabCut Border Matting - Colour
Compute MAP of p(F|C,alpha) (marginalize over B)
To avoid colour bleeding use colour stealing (“exemplar based inpainting” – Patches do not work)
[Chuang et al. ‘01]
Grabcut Border Matting
GrabCut – Interactive Foreground Extraction

30. Results
GrabCut – Interactive Foreground Extraction

31. Stereo Video + Segmentation
Left Sequence
Right Sequence
Criminisi et. al. (2003): Plane DP to handle occlusions properly
Disparity Sequence
GrabCut – Interactive Foreground Extraction

32. Occusion, left and right
Try to get proper speed video
GrabCut – Interactive Foreground Extraction

33. Background Substitution
Criminisi et. al. (2004): Remove boundary artefacts (SPS algorithm)
GrabCut – Interactive Foreground Extraction

34. Object Insertion
GrabCut – Interactive Foreground Extraction

35. Focus on Foreground
GrabCut – Interactive Foreground Extraction

36. Conclusions & Future Work
GrabCut – powerful interactive extraction tool
Iterated Graph Cut based on colour and contrast
Regularized alpha matting by Dynamic Programming
Stereo and Segmentation give supportive information
How to solve the difficult hair problem ?
[Argawall et.al. 2004]
GrabCut – Interactive Foreground Extraction