Presentation is loading. Please wait.

Presentation is loading. Please wait.

Shape Sharing for Object Segmentation

Similar presentations


Presentation on theme: "Shape Sharing for Object Segmentation"— Presentation transcript:

1 Shape Sharing for Object Segmentation
Jaechul Kim and Kristen Grauman University of Texas at Austin

2 Problem statement Category-independent object segmentation:
Generate object segments in the image regardless of their categories.

3 Related work Top-down, category-specific approach
e.g., Active Contours (IJCV 1987), Borenstein and Ullman (ECCV 2002), Levin and Weiss (ECCV 2006), Kumar et.al. (CVPR 2005) Use generic knowledge on object shapes e.g., Levinshtein et.al. (ICCV 2009) Learn local shapes e.g., Ren et.al. (ECCV 2006), Opelt et.al. (CVPR 2006) Category-independent multiple segmentations e.g., Malisiewicz and Efros (BMVC 2007), Carreira and Sminchisescu (CVPR 2010), Endres and Hoiem (ECCV 2010)

4 Spectrum of existing approaches
horse shape priors color, textures, edges… How to model top-down shape in a category-independent way? Bottom-up Class-specific + coherent mid-level regions + applicable to any image - prone to over/under-segment + robustness to low-level cues - typically viewpoint specific - requires class knowledge! e.g., e.g., Malisiewicz and Efros (BMVC 2007), Arbelaez et al. (CVPR 2009) Carreira and Sminchisescu (CVPR 2010) Endres and Hoiem (ECCV 2010) Active Contours (IJCV 1987) Borenstein and Ullman (ECCV 2002) Levin and Weiss (ECCV 2006) Kumar et.al. (CVPR 2005)

5 Our goal Segment even unfamiliar objects with category-independent top-down cues Cow? Sheep? Top-down segmentation with shape prior We don’t want to care what is in the image.

6 Our idea: Shape sharing
Semantically close Semantically disparate Object shapes are shared among different categories. Shapes from one class can be used to segment another (possibly unknown) class: Enable category-independent shape priors

7 Basis of approach: transfer through matching
Exemplar image Test image ground truth object boundaries Partial shape match Global shape projection Transfer category-independent shape prior

8 … + Approach: Overview 1. Shape projection via local shape matches
Exemplars Test image + Shape prior Color model Segmentation model per each group 2. Aggregating the shape projections Graph-cut Segmentation hypotheses 3. Multiple figure-ground segmentations with shape prior

9 Approach: Shape projection
Aggregation Segmentation Test image Exemplars Vs. BPLRs Superpixels Boundary-Preserving Local Regions (BPLR): Distinctively shaped Dense Repeatable [Kim & Grauman, CVPR 2011]

10 Approach: Shape projection
Aggregation Segmentation Test image Exemplars Shape projections via similarity transform of BPLR matches Matched Exemplar 1 Matched Exemplar 2 Shape hypotheses

11 Approach: Refinement of projections
Aggregation Segmentation Exemplar jigsaw Initial projection Refined shape Align with bottom-up evidence Include superpixels where majority of pixels overlap projection

12 Approach: Aggregating projections
Aggregation Segmentation Grouping based on overlap Exploit partial agreement from multiple exemplars

13 + Approach: Segmentation Figure-ground segmentation using graph-cut
Projection Approach: Segmentation Aggregation Segmentation + Shape prior Color model Segmentation model per each group Figure-ground segmentation using graph-cut

14 Approach: Graph-cut n-links Define a graph over image pixels:
Projection Approach: Graph-cut Aggregation Segmentation n-links Define a graph over image pixels: node = pixel edge = cost of a cut between pixels Energy function to minimize: data term smoothness term

15 + Approach: Segmentation Bg color histogram Bg NA Fg color histogram
Projection Approach: Segmentation Aggregation Segmentation Bg color histogram Bg NA Fg color histogram Fg + Shape likelihood Color likelihood Data term Smoothness term Graph-cut optimization

16 … Approach: Multiple segmentations
Projection Approach: Multiple segmentations Aggregation Segmentation Compute multiple segmentations by varying foreground bias: Parameter controlling data term bias Output: Carreira and Sminchisescu, CPMC: Automatic Object Segmentation Using Constrained Parametric Min-Cuts PAMI 2012.

17 Experiments … Exemplar database:
PASCAL 2010 segmentation task training set (20 classes, 2075 objects) Test datasets: PASCAL 2010 segmentation task validation set (20 classes, 964 images) Berkeley segmentation dataset (natural scenes and objects, 300 images) Evaluation metric: Best covering score w.r.t # of segments Baselines: CPMC [Carreira and Sminchisescu, PAMI 2012] Object proposals [Endres and Hoiem, ECCV 2012] gPb+owt+ucm [Arbelaez et al., PAMI 2011] Ground truth 0.92 0.75 0.71 Best covering score: 0.92

18 Segmentation quality Approach Covering (%) Num of segments
Shape sharing (Ours) 84.3 1448 CPMC [Carreira and Sminchisescu] 81.6 1759 Object proposals [Endres and Hoiem] 81.7 1540 gPb-owt-ucm [Arbelaez et al.] 62.8 1242 PASCAL 2010 dataset Approach Covering (%) Num of segments Shape sharing (Ours) 75.6 1449 CPMC [Carreira and Sminchisescu] 74.1 1677 Object proposals [Endres and Hoiem] 72.3 1275 gPb-owt-ucm [Arbelaez et al.] 61.6 1483 Berkeley segmentation dataset *Exemplars = PASCAL

19 When does shape sharing help most?
Gain as a function of color easiness and object size Easy to segment by color Hard to segment by color Compared to CPMC [Carreira and Sminchisescu., PAMI 2012]

20 Which classes share shapes?
Semantically disparate Animals Unexpected pose variations Vehicles

21 Objects with diverse colors
Example results (good) Shape sharing (ours) 0.889 0.859 0.903 0.935 CPMC (Carreira and Sminchisescu) 0.599 0.638 0.630 0.694 Objects with diverse colors

22 Objects confused by surrounding colors
Example results (good) Shape sharing (ours) 0.966 0.875 0.999 0.928 CPMC (Carreira and Sminchisescu) 0.508 0.533 0.526 0.685 Objects confused by surrounding colors

23 Example results (failure cases)
Shape sharing (ours) 0.220 0.199 0.713 0.406 CPMC (Carreira and Sminchisescu) 0.818 0.934 0.973 0.799

24 Shape sharing: highlights
Top-down shape prior in a category-independent way Non-parametric transfer of shapes across categories Partial shape agreement from multiple exemplars Multiple hypothesis approach Most impact for heterogeneous objects Code is available:

25 Approach: Refinement and pruning
Projection Approach: Refinement and pruning Aggregation Segmentation Exemplar jigsaw Initial projection Refined shape Pruned out Exemplar

26 Segmentation quality Quality of initial shape projections Approach
Covering (%) Num of segments Exemplar-based merge (Ours) 77.0 607 Neighbor merge [1] 72.2 5005 Bottom-up segmentation [2] 62.8 1242 [1] Malisiewicz and efros, BMVC 2007. [2] Arbelaez et.al., PAMI 2011. Initial shape prior Exemplar

27 Object proposal, PASCAL
Impact of shapes Shape sharing’s gain in recall as a function of overlap CPMC, PASCAL CPMC, BSD Object proposal, PASCAL Object proposal, BSD

28 Category-independent vs. dependent
Approach Covering (%) Category-specific 84.7 Category-independent (default) 84.3 Strictly category-independent 83.9 CPMC 81.6 Object proposals 81.7 Comparison of category-independent shape prior and category-specific variants


Download ppt "Shape Sharing for Object Segmentation"

Similar presentations


Ads by Google