1. On Multiple Foreground Cosegmentation Gunhee Kim Eric P. Xing 1 School of Computer Science, Carnegie Mellon University June 18, 2012

2. Problem Statement Algorithm  Overview  Foreground Modeling  Region Assignment Experiments Conclusion Outline 2

3. 3 Image Cosegmentation High-level signal: recurring objects in multiple images Jointly segment multiple images into K foregrounds and background [R06] Rother et al. CVPR2006 [H09] Hochbaum and Singh, ICCV2009 [J10,J12] Joulin et al, CVPR2010, CVPR2012 [B11] Batra et al, IJCV 2011 [M11] Mukherjee et al, CVPR 2011 [V10,V11] Vincente et al, ECCV 2010, CVPR2011 [K11] Kim et al, ICCV 2011. [R06] [J10] [B11] [K11]

4. 4 Popular Cosegmentation Datasets CMU-Cornell iCoseg Dataset [Batra et al. IJCV11] MSRC Dataset [Winn et al. ICCV05] Synthesized Dataset [Rhemann et al. CVPR09]

5. 5 Popular Cosegmentation Datasets CMU-Cornell iCoseg Dataset [Batra et al. IJCV11] MSRC Dataset [Winn et al. ICCV05] Synthesized Dataset [Rhemann et al. CVPR09] Input images are carefully prepared by human so that the objects of interest are salient enough in every single image.

6. 6 General Users’ Photo Sets A part of Apple+picking photo stream from Flickr A series of photos are taken for a specific moment The number of foregrounds (ie. subjects of interest) are finite Follow an ordinary user’s photo-taking pattern. Girl in red (R) Girl in blue (G) Baby (B) Apple bucket (A) (R, G, A) (B, A)(B, A) (R, A)(G, A)(G, A) (R, G, B, A)(R, G, A)(R, G, B, A)(R, A) Each image contains an unknown subset of foregrounds

7. 7 General Users’ Photo Sets A part of Apple+picking photo stream from Flickr A series of photos are taken for a specific moment The number of foregrounds (ie. subjects of interest) are finite Follows an ordinary users’ photo-taking pattern. Girl in red (R) Girl in blue (G) Baby (B) Apple bucket (A) (R, G, A) (B, A)(B, A) (R, A)(G, A)(G, A) (R, G, B, A)(R, G, A)(R, G, B, A)(R, A) Each image contains an unknown subset of foregrounds Has NOT yet explicitly addressed

8. 8 Problem Statement Multiple Foreground Cosegmentation Optionally, a user may assign some examples of foregrounds. Each image contains an unknown subset of foregrounds Given an image set I, K foregrounds of interest Segment each image into regions of {F 1,…,F K+1 } (F K+1 =B) Girl in red (R) Girl in blue (G) Baby (B) Apple bucket (A) (R, G, A) (B, A)(B, A) (R, A)(G, A)(G, A) (R, G, B, A)(R, G, A)(R, G, B, A)(R, A)

9. Problem Statement Algorithm  Overview  Foreground Modeling  Region Assignment Experiments Conclusion Outline 9

10. 10 Overview of Algorithm Iteratively solve Foreground Modeling Region Assignment Learn appearance models of K+1 foregrounds (FGs) Performed in each image separately Allocate the regions of image into one of K+1 FGs Initialization Supervised Assigned by a user Unsupervised Apply diversity ranking of [Kim et al. ICCV11] Iterate until convergence [Kim&Toralba NIPS09] [Rother et al. SIGGRAPH04] ✔

11. Problem Statement Algorithm  Overview  Foreground Modeling  Region Assignment Experiments Conclusion Outline 11

12. 12 Foreground (FG) Model Definition of k-th FG model Any region classifiers or their combination Value Assignment (Testing)Foreground learning (Training) Baby FG model A parametric function Given any region S, return its value (score) of FG k

13. 13 Foreground (FG) Model Definition of k-th FG model A parametric function Given any region S, return its value (score) to FG k Any region classifiers or their combination Gaussian Mixture Model (GMM) Spatial Pyramid + linear SVM (SPM) One of most popular in cosegmentation literatures One of most popular in image classification RGB colors Gray/HSV SIFT

14. Problem Statement Proposed Algorithm  Foreground Modeling  Region Assignment Experiments Conclusion Outline 14

15. 15 Region Assignment Given learned (or initialized) FG Models, RA is individually performed in each image Cow FG Person FG Background Oversegment

16. 16 Region Assignment Given learned (or initialized) FG Models, RA is individually performed in each image Cow FG Person FG Background Region Assignment

17. 17 A Naïve Region Assignment A naïve way: Assign each segment to the FG whose value is the highest Cow FG Person FG Background Cow BG BG BG Cow However, it will NOT work !

18. 18 Why does not A Naïve RA work? Most naïve way: Assign each segment to FG whose value is the highest Cow FG Person FG s1s1 s2s2 Cow won! Person won! My value of { s 1, s 2 } is 18. {s 1,s 2 } My value of s 2 is 5. My value of s 1 is 10. My value of { s 1, s 2 } is 35. My value of s 2 is 20. My value of s 1 is 7.

19. 19 Why does not A Naïve RA work? Most naïve way: Assign each segment to FG whose value is the highest Cow FG Person FG s1s1 s2s2 Cow won! Person won! My value of { s 1, s 2 } is 18. {s 1,s 2 } My value of s 2 is 5. My value of s 1 is 10. My value of { s 1, s 2 } is 35. My value of s 2 is 20. My value of s 1 is 7. Value of { s 1, s 2 } to person FG Value of s 1 to Cow FG Value of s 2 to person FG + > Have to evaluate the combinations of segments

20. 20 Region Assignment by Combinatorial Auction Cow FG Person FG Background Items to sell Bidder 1 (or buyer) Bidder 2 Bidder 3 Each FG is allowed to bid packages of items with their own values. Distribute the segments to maximize the values.

21. 21 Region Assignment as Combinatorial Auction Assign the segments to FGs to maximize the overall values. Winner determination problem (ie. Welfare maximization) Unfortunately, NP-complete and Inapproximable Feasibility: Each segment cannot be assigned more than once [Cramton et al. 2005]. 2 |Si||Si| subsets

22. ✔ 22 Next Goal: Tractable Solution to WDP There are two different ways… 1. Constraints on value functions 2. Constraints on generating bidding packages Allow any combinations of FG models (ie. Region classifiers) If value functions are submodular or subadditive, … [Felzenszwalb et al.2008]

23. 23 Procedure of Region Assignment 1. Each FG creates a set of foreground candidates 3. Solve WDP by choosing feasible FG candidates. Follow a general combinatorial auction scenario. : n FG candidates by FG k : a bundle of segments : its value : FG ID In this step, each FG does not care their winning chances. 2. Finally,

24. 24 Assumption for FG Candidates A FG instance = a sub-tree of G i A FG instance in an image = a set of adjacent segments. Any FG candidate = a sub-tree of G i

25. 25 Generating FG Candidates by Beam Search Beam width D Computation time: O(D|S i | 2 ) Number of FG candidates per FG: |B i k | = O(D|S i |) For each size of candidates, we keep only D high valued ones. 2. O t all possible subgraphs by adding a single edge to elements of O. 1. O every single segment 3. Compute values v o v k (o) for all and keep only top D high valued ones to O. 4. Iterate ( |S i |-1) times. For each foreground k,

26. 26 Example of Candidate Set 19.74 39.42 29.30 18.2247.0639.47 39.61 19.85 19.88 18.78 16.35 59.49 Back- ground Apple bucket Baby

27. 27 Solving WDP 19.74 39.42 29.30 18.2247.0639.47 39.61 19.85 19.88 18.78 16.35 59.49 Back- ground Apple bucket Baby Solve WDP ! The number of FG candidates is (K+1)D|S i |. Even a faster algorithm… By search in polynomial time

28. 28 Solving WDP [Theorem] Dynamic program can solve WDP in O(|B i ||S i |) worst time if every candidate in B i can be represented by a connected subgraph of a tree T i *. [Sandholm et al.2003] Each FG candidate is a tree, but the aggregation is not. 19.74 39.42 29.30 18.22 47.0639.47 39.61 19.85 19.88 18.78 16.35 59.49 Back- ground Apple bucket Baby Candidate 2 Candidate 1

29. 29 Inferring the Most Probable Tree from FG Candidate Set 19.74 39.42 29.30 18.22 47.0639.47 39.61 19.85 19.88 18.78 16.35 59.49 Back- ground Apple bucket Baby Solve the following MLE solution : all possible spanning trees where w 2 = 5 Candidate 2 Candidate 1 w 1 = 20 Reject the bids that are not a subtree of

30. 30 Inferring the Most Probable Tree from Candidate Set 19.74 39.42 29.30 18.22 47.0639.47 39.61 19.85 19.88 18.78 16.35 59.49 Back- ground Apple bucket Baby Solve the following MLE solution : all possible spanning trees where MLE solution = MST by Kruskal’s algorithm in O(|B i ||S i | 2 ) Almost identical to Chow-Liu tree structure learning

31. 31 Finally, Solve WDP [Theorem] Dynamic program can solve WDP in O(|B i ||S i |) worst time if every candidate in B i can be represented by a connected subgraph of a tree T i *. [Sandholm et al.2003] optimal assignment segmentation CABOB algorithm [Sandholm et al.2005]

32. Problem Statement Proposed Algorithm  Foreground Modeling  Region Assignment Experiments Conclusion Outline 32

33. 33 Two Experiments FlickrMFC dataset ImageNet dataset Goal: To achieve multiple foreground cosegmentation New benchmark dataset (14 groups, 20 images, fully-labeled) ex. Cow group: {person, car, cow (brown, black)} Goal: Scalability ex. green lizard

34. 34 Quantitative Evaluation Segmentation accuracies Metric MFC-S: (supervised) our method MFC-U: (unsupervised) our method COS : Submodular optimization [Kim et al. ICCV11] DC: Discriminative clustering [Joulin. CVPR10] LDA: LDA-based localization [Russell et al. CVPR06] MNcut: Normalized cuts [Cour et al. CVPR05]

35. 35 Cosegmentation Examples FlickrMFC ImageNet Australian terrier Lion

36. Problem Statement Proposed Algorithm  Foreground Modeling  Region Assignment Experiments Conclusion Outline 36

37. 37 Conclusion Each image contains a unknown subset of foregrounds Web-oriented applications Code and FlickrMFC dataset will be available in this month! Fast and distributable (ex. Linear with M,K, polynomial in |S i | ) Can be used for other tasks (ex.detection) beyond segmentation. Multiple foreground cosegmentation Combinatorial auction-based region assignment

38. 38 Take-Home Message Combinatorial optimization for Image Segmentation! New region descriptors & classifiers every year Multiple fast machines are available. May avoid difficult ML algorithms or high-order models Similar line of thought: Our ICCV 2011 paper Submodular optimization Simply enumerate the cases (or hypotheses) not in a brute- force but in a smart way.