Object Recognition by Integrating Multiple Image Segmentations Caroline Pantofaru, Cordelia Schmid, Martial Hebert ECCV 2008 E

2 Types of object recognition Cat BikeFace Goal: Accurate object recognition and object segmentation of deformable object classes

3 Too many possible shapes

4 Shape proposals Silhouette masks of (semi-) rigid objects. –Edge-driven: [Shape context,Belongie et al., PAMI’02] –Feature-driven: [Marszalek and Schmid, CVPR’06] Each pixel (patch) separately: –Noise [Sivic et al., ICCV’05] [Ferrari et al. ECCV’04] –Constrained shape and appearance [Borenstein and Malik, CVPR’06] Fixed-shape parts: semi-rigid objects. –[Mori et al., CVPR’04]

5 Bottom-up grouping: Image Segmentation [BP at CVPR’06] [BMVC’07] [Hoiem et al., IJCV’07][Russell et al., CVPR’06][Todorovic and Ahuja, CVPR’06][Ren and Malik, ICCV’03]…

6 Classifying single regions Image Object map Learn single region feature classifier Bottom-up image segmentation Training data … … Bottom-up image segmentations Single region classification by SVM Region features: centroid histogram of quantized hue features Region-based Context Feature aggregate hue histogram over the image aggregate RCF over the entire image “Good” segmentation: Repeatable Regions are “large enough” for feature computation but stay within object boundaries.

7 Human segmentations are inconsistent Berkeley segmentation database –[Martin et al. ICCV’01]

8 [MS: Mean shift, Comaniciu and Meer, PAMI’02] [FH: Felzenszwalb and Huttenlocher, IJCV’04] [MS+FH: Pantofaru and Hebert, CMU’05] Automatic segmentations are very inconsistent

9 Types of regions and boundaries Object boundary Color discontinuity Segmentation artifact Over- segmentation Object partUnder- segmentation

10 Multiple segmentations Mean Shift [Comaniciu and Meer, PAMI’02] Normalized cuts with boundary estimates [Shi and Malik, PAMI’00; Fowlkes et al., CVPR’03] Graph-based segmentation [Felzenszwalb and Huttenlocher, IJCV’04] Intersections of Regions

11 Multiple segmentations – Related work Ways to generate multiple segmentations: –Hierarchical [Hoiem et al., IJCV’07] [Borenstein and Malik, CVPR’06] [Todorovic and Ahuja, ICCV’07] –One algorithm, different numbers of regions [Russell et al., CVPR’06] [Malisiewicz and Efros, BMVC’07] –Multiple algorithms, multiple parameters [ECCV’08] Ways to use multiple segmentations: –Pick best region [Russell et al., CVPR’06] –Use segments as parts [Todorovic and Ahuja, ICCV’07] –Soft segmentation, use shape [Borenstein and Malik, CVPR’06] –Integrate information from all segmentations [ECCV’08] [Hoiem et al., IJCV’07]

12 Ideas and Assumptions Ideas: –Groups of pixels consistently clustered should be consistently classified. –The set of regions provides robust features for classification. Assumptions: –Object edges are a subset of the region outlines. –Each pixel is contained in some region that is large enough for descriptive feature computation. 12

13 Multiple Segmentations Intersections of Regions … …

14 Image Multiple segmentations: Results Confidence Ground truthMultiple segmentations Good single segmentation Poor single segmentation MSRC 21-class data set –[Shotton et al., ECCV’06]

15 Multiple segmentations: Results ImageConfidence Ground truthGood single segmentation Poor single segmentation Multiple segmentations

16 Multiple segmentations: Results Pixel accuracy on the MSRC 21-class data set Shotton’06Verbeek’07Good single seg Poor single seg Multiple segs Class- averaged 57.7%64.0%59.8%49.6%60.3% Overall72.2%73.5%72.2%63.3%74.3% [Shotton et al., ECCV’06] [Verbeek and Triggs, CVPR’07 (Different data split)]

17 Multiple Segmentations: Results class avg pixel avg bldggrasstreecowsheepskyplanewaterfacecar Shotton *Verbeek Worst seg Best seg All segs bikeflowersignbirdbookchairroadcatdogbodyboat Shotton *Verbeek Worst seg Best seg All segs

18 Are all of the segmentations useful? PASCAL VOC2007 segmentation challenge data set

19 Region reliability Intersections of Regions … [Homogeneity: Hoiem et al., IJCV’07] Classifier: Boosted decision trees 

20 Result of using multiple segmentations Ground TruthMultiple Segs Individual segmentations PASCAL VOC2007 segmentation challenge data set

21 Results on PASCAL VOC07 data set Image Ground truthGood single segmentation Poor single segmentation Multiple segmentations Good single seg Bad single segMultiple segsMultiple segs w/ region weights Class-averaged

22 Spatial information – Related work Silhouettes/Shape –[Levin and Weiss, ECCV’06] [Kumar et al., CVPR’05] [Borenstein and Malik, CVPR’06] –Rigid or semi-rigid objects Relationships between parts –[Mori et al., CVPR’04] [Todorovic and Ahuja, ICCV’07] –Semi-rigid objects –Regions are whole parts Pair-wise region information in a random field –[ECCV’08] [BMVC’07] [Hoiem et al., IJCV’07] [He et al., ECCV’06] –Deformable objects –Combine over-segmented regions

23 Pair-wise Region Model Reward confidence in the label: Penalize discontinuity: Solve using Graph Cuts [Kolmogorov and Zabih, ECCV’02] Energy minimization: IofRs …

24 Pair-wise Region Model Results Single regionRandom field

25 PASCAL VOC2007 segmentation challenge Without spatial info With spatial info Class avg Person Bus Cat Background Dog Horse Pair-wise Region Model Results Fully supervised training Multiple segmentations 21-class problem

26 Training with weak labels Augment the training set. 422 fully labeled images weakly labeled images. Weak labels: bounding boxes or image labels.

27 Results of training with weak labels Image Seg + Image Training Seg + BBox Training Seg Training Ground Truth PASCAL VOC2007 segmentation challenge data set

28 Conclusions A study into the use of multiple image segmentations to propose spatial support for object recognition and object segmentation. djs Using multiple segmentations provides robustness to poor single segmentations. Learning which regions to trust is difficult. (Ideas from today’s other papers?) Pair-wise information can help in certain situations. Always evaluate your algorithms and share all of the results! Don’t add complexity if it isn’t useful!

29 SIFT words Region-based Context Features (RCF) 2/3 1/ [Lowe, IJCV’04] [BP at CVPR’06]