1. Stephen J. Guy 1

2. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1

3.  Background  Segmentation Intro  Graph Cut for Segmentation (Boykov & Jolly ‘01)  Copying  Lazy Snapping (Li et al. ’04)  GrabCut (Rother et al. ’04)  Other  Texture & Video Synthesis (Kwatra et al. ’04) 3

4.  Goal  Splitting Image into similar, meaningful(?) regions  E.g. Clustering, Edge Detection, *Graph Cuts* X. Ren and J. Malik. Learning a classification model for segmentation. ICCV 2003. “SuperPixels” GrabCut Rother et al. 2004 4

5.  A – Proposed Segmentation  E(A) – Overall Energy  R(A) – Degree to which pixels fits model  B(A) – Degree to which the cuts breaks up similar pixels  - Balance A() and B() Goal: Find Segmentation, A, which minimizes E(A) 5

6.  Image As Graph  Each pixel is a node  Create links between each pixel  More nodes/links possible (e.g. foreground node)  Graph Cuts  Assign energy for each link based on similarity  Break the links with least energy, but  Require each pixel is still connected to exactly one foreground or background node  Benefits  No Topological constraints on regions (can be concave, unconnected, etc.)  Fast (polynomial time) solution – Ford-Fulkerson Algorithm 6

7. 7

8.  Pixel links based on color/intensity similarities  Source/Target links based on histogram models of fore/backgroun d 8

9.  Effect of  Recall: 9

10. 10

11.  Main Steps: 1. Mark strokes as foreground & background 2. Perform Boykov & Jolly style graphcut 3. *Edit Boundary*  Familiar Formulation  Minimize Gibbs Energy:  E 1 (X) – Cluster (K-means) known foreground/background colors to find representative colors  E 2 (X) – Similar to Boykov & Jolly 11

12.  Too Slow to allow for real-time refinement  Solution? Superpixels ! 12

13. 13

14.  Graph Cut is only step 1  Allow user to edit boundary directly  Formulate as Graph Cut  Allow user to guide boundary regions w/ brush 14

15. 15

16.  Compared Lazy Snapping to Photoshop 16

17.  Results  Easy of Use – Lazy Snapping 20% less mistakes  Speed – Lazy Snapping 60% less time  Accuracy – Lazy Snapping 60% less pixels wrong  Feedback  “Almost Magic”  “Much Easier”  Suggestions  Get rid of 2 step process  Make it easy to switch between graph cut strokes and boundary editing 17

18. 18

19. GrabCut Interactive Foreground Extraction using Iterated Graph Cuts Carsten Rother Vladimir Kolmogorov Andrew Blake Microsoft Research Cambridge-UK

20.  New model of foreground & background pixels:  GMM vs. K-mean colors vs. Intensity Histogram  More automation:  No need to specify foreground  Iterative Graph Cuts 20

22. What GrabCut does What GrabCut does User Input Result Magic Wand (198?) Intelligent Scissors Mortensen and Barrett (1995) GrabCut Regions Boundary Regions & Boundary GrabCut – Interactive Foreground Extraction 3

23. 1. Assign Selection box or Lasso as background 2. Assign other pixels as unknown 3. Learn Model of Fore/Background 4. Assign Link Energy, GraphCut (Boykov & Jolly) 5. Use GraphCut results to assign Fore/Background 6. Goto Step 3 7. Allow user to add cleanup strokes 23

24. Graph Cuts Boykov and Jolly (2001) Graph Cuts Boykov and Jolly (2001) GrabCut – Interactive Foreground Extraction 5 Image Min Cut Min Cut Cut: separating source and sink; Energy: collection of edges Min Cut: Global minimal enegry in polynomial time Foreground (source) Background (sink)

25. Iterated Graph Cut Iterated Graph Cut User Initialisation K-means for learning colour distributions Graph cuts to infer the segmentation ? GrabCut – Interactive Foreground Extraction 6

26. 1 2 34 Iterated Graph Cuts Iterated Graph Cuts GrabCut – Interactive Foreground Extraction 7 Energy after each IterationResult Guaranteed to converge

27. Colour Model Colour Model Gaussian Mixture Model (typically 5-8 components) Foreground & Background Background Foreground Background G GrabCut – Interactive Foreground Extraction 8 R G R Iterated graph cut

28. Moderately straightforward examples Moderately straightforward examples … GrabCut completes automatically GrabCut – Interactive Foreground Extraction 10

29. Difficult Examples Difficult Examples Camouflage & Low Contrast No telepathy Fine structure Initial Rectangle Initial Result GrabCut – Interactive Foreground Extraction 11

30. Comparison GrabCut Boykov and Jolly (2001) Error Rate: 0.72% Error Rate: 1.87%Error Rate: 1.81%Error Rate: 1.32%Error Rate: 1.25%Error Rate: 0.72% GrabCut – Interactive Foreground Extraction 13 User Input Result

31. Border Matting Border Matting Hard Segmentation Automatic Trimap Soft Segmentation GrabCut – Interactive Foreground Extraction 16

32. Solve Mean Colour Foreground Mean Colour Background GrabCut – Interactive Foreground Extraction 17 Natural Image Matting Ruzon and Tomasi (2000): Alpha estimation in natural images

33. Noisy alpha-profile Border Matting GrabCut – Interactive Foreground Extraction 19 1 0 Foreground Mix Back- ground Foreground Mix Background Fit a smooth alpha-profile with parameters

34. ResultsResults GrabCut – Interactive Foreground Extraction 21

35. Summary Summary Magic Wand (198?) Intelligent Scissors Mortensen and Barrett (1995) GrabCut Rother et al. (2004) Graph Cuts Boykov and Jolly (2001) LazySnapping Li et al. (2004) GrabCut – Interactive Foreground Extraction 22

36. Boykov&JollyLazy SnappingGrabCut Foreground & Background Model Histogram of Intensities K-means clustering, 64 representative colors K-mean Background Source User Strokes Initial Box (Lasso) + User Strokes Foreground Source User Strokes None or User Strokes Boundary Term Similarity Penalty Segmentation GraphCutSuperpixels + GraphCut Iterative GraphCut Refinement User StrokesBoundary EditingUser Strokes 36

37.  Texture Synthesis Revisited 37

38.  Copy overlapping patches one at a times  Find the seamwhich minimize change in neighboring pixels  Formulate as Graph Cut 38

39.  Extends well into 3D 39