1. Interactive Segmentation with Super-Labels Andrew Delong Western Yuri BoykovOlga VekslerLena GorelickFrank Schmidt TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AA A A A A A

2. Natural Images: GMM or MRF? 2 are pixels in this image i.i.d.?NO!

3. Natural Images: GMM or MRF? 3

4. 4

5. 5

6. Boykov-Jolly / Grab-Cut 6 [Boykov & Jolly, ICCV 2001] [Rother, Kolmogorov, Blake, SIGGRAPH 2004]

7. Boykov-Jolly / Grab-Cut 7 [Boykov & Jolly, ICCV 2001] [Rother, Kolmogorov, Blake, SIGGRAPH 2004]

8. Boykov-Jolly / Grab-Cut 8 [Boykov & Jolly, ICCV 2001] [Rother, Kolmogorov, Blake, SIGGRAPH 2004]

9. Objects within image can be as complex as image itself Where do we draw the line? A Spectrum of Complexity 9 MRF?GMM? Gaussian? object recognition??

10. Single Model Per Class Label 10

11. Multiple Models Per Class Label 11

12. Multiple Models Per Class Label 12

13. Our Energy ¼ Supervised Zhu & Yuille! Zhu & Yuille. PAMI’96; Tu & Zhu. PAMI’02 Unsupervised clustering of pixels 13 boundary length MDL regularizer + color similarity +

14. Our Energy ¼ Supervised Zhu & Yuille! Zhu & Yuille. PAMI’96; Tu & Zhu. PAMI’02 14 boundary length MDL regularizer + color similarity +

15. Interactive Segmentation Example 15

16. Boykov-Jolly / Grab Cut 16 segmentationcolour models

17. Ours 17 segmentationcolour models“sub-labeling”

18. Main Idea Standard MRF: Two-level MRF: 18 object MRF GMMs background MRF image-level MRF object GMMbackground GMM image-level MRF unknown number of labels in each group!

19. The “Super-Pixel” View Complex object ¼ group of super-pixels Interactive segmentation ¼ a“user-constrained super-pixel grouping” 19

20. The “Super-Pixel” View Why not just pre-compute super-pixels? – boundaries may contradict user constraints – user is helpful for making fine distinctions Combine automatic (unsupervised) and interactive (supervised) into single energy 20 Spatially coherent clustering + MDL/complexity penalty + user constraints = 2-level MRF Like Zabih & Kolmogorov, CVPR 2004 Label Costs, CVPR 2010 Like Boykov & Jolly, ICCV 2001

21. Process Overview 21 user constraints propose models from current super-labeling 1 solve 2-level MRF via α-expansion 2 refine all sub-models 3 converged E=503005 E=452288 Boykov-Jolly + unsupervised clustering (random sampling) + iterated multi-label graph cuts (like grab-cut)

22. Our Problem Statement Input: set S of super-labels (e.g. f fg,bg g ) constraints g : P ! S [ f any g 22 fg bg any

23. Our Problem Statement Output: set L of sub-labels sub-labeling f : P ! L model params µ ` for each ` 2L label grouping ¼ : L ! S 23 ¼ ±f¼ ±f f `2`2 `1`1 `3`3 GMM ` 1 white GMM ` 2 dark green light green

24. Our Energy Functional 24 Minimize single energy w.r.t. L, µ, f, ¼ data costssmooth costslabel costs `4`4 `3`3 `1`1 `2`2 forces transition

25. Our Energy Functional 25 Minimize single energy w.r.t. L, µ, f, ¼ data costssmooth costslabel costs pay c 2 `between group’ pay c 1 `within group’

26. Our Energy Functional 26 Minimize single energy w.r.t. L, µ, f, ¼ Penalize number of GMMs used – prefer fewer, simpler models – MDL / information criterion regularize “unsupervised” aspect data costssmooth costslabel costs GMMs

27. More Examples 27 Boykov-Jolly2-level MRF

28. More Examples 28 Boykov-Jolly2-level MRF

29. More Examples 29 Boykov-Jolly 2-level MRF

30. More Examples 30 Boykov-Jolly grad students baby panda 2-level MRF GMM density for blue model

31. Interactive Co-segmentation 31 image collection 2-level MRF Boykov-Jolly (like “iCoseg”, Batra et al., CVPR 2010)

32. More Examples 32 Boykov-Jolly 2-level MRF

33. More Examples 33 Boykov-Jolly 2-level MRF

34. Beyond GMMs 34 GMMs plane GMMs onlyGMMs + planes

35. Synthetic Example 35 GMM Boykov-Jolly (1 GMM each label) GMM 2-level MRF (GMMs only) plane GMM 2-level MRF (GMM + planes) object = two planes in (x,y,grey) space noise = one bi-modal GMM (black;white)

36. Synthetic Example 36 plane GMM black white x 2 planes detected 1 GMM detected y black white

37. As Semi-Supervised Learning Interactive segmentation ¼ a semi-supervised learning – Duchenne, Audibert, Keriven, Ponce, Segonne. Segmentation by Transduction. CVPR 2008. –s - t min cut [Blum & Chawla, ICML’01] – random walker [Szummer & Jaakkola, NIPS’01] 37

38. Conclusions GMM not good enough for image ) GMM not good enough for complex objects Energy-based on 2-level MRF – data costs + smooth costs + label costs Algorithm: iterative random sampling, re-fitting, and ® -expansion. Semi-supervised learning of complex subspaces with ® -expansion 38