Recognition using Regions (Demo) Sudheendra V

Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed transform [ Arbel´aez1et al. (2009)] Non-uniform Multiple scales Discovering object regions using LDA –ground truth segments –multiple segmentation –single segmentation –hierarchical segmentation

Multiple Segmentations Normalized cuts –segmentation as graph partitioning nodes -> pixels, edge between neighboring pixels edge weight -> affinity between pixels partition graph into K components –parameters number of partitions K –properties similar sized partitions (normalized) preserves region boundaries for large enough K –multiple segmentations vary number of partitions K resize image to different resolutions affinity matrix

Normalized cuts (examples) K = 4 K = 6 K = 7

Normalized cuts (examples) K = 3 K = 5 K = 7 Extra edge “normalized” regions

Normalized cuts (examples) K = 3 K = 5 K = 7

Multiple Segmentations Watershed transform –contours are detected using texture, edge cues and oriented watershed transform used to determine contour scale –parameters thresholding scale for contours, k –properties variable sized regions preserves region boundaries –multiple segmentations vary thresholding scale k resize image to different resolutions Contours at multiple scalesThreshold at a scale

Watershed (examples) K = 200 K = 180 K = 160 Threshold at different contour scales K to generate multiple segmentations

Watershed (examples) K = 195 K = 183 K = 169 Threshold at different contour scales K to generate multiple segmentations

Watershed (Hierarchical segmentation) Thresholding scales in an increasing sequence produces a hierarchical segmentation K = 175 K = 155 K = 190 K = 140

Watershed (Hierarchical segmentation) Thresholding scales in an increasing sequence produces a hierarchical segmentation K = 175 K = 155 K = 200 K = 145

Ncuts vs Watershed Ncuts Watershed Comparison of multiple segmentations generated using Ncuts vs Watershed

Ncuts vs Watershed Ncuts Watershed Comparison of multiple segmentations generated using Ncuts vs Watershed

Outline Generating multiple segmentations –Normalized cuts [Ren & Malik (2003)] Uniform regions –Watershed transform [ Arbel´aez1et al. (2009)] Non-uniform Multiple scales Discovering object regions using LDA –ground truth segments –multiple segmentation –single segmentation –hierarchical segmentation

Multiple Segmentations

Discovering object regions using LDA Approach Parameters – number of topics to discover Generate multiple segmentations Extract local features (SIFT) Bag of words rep for each segment Use LDA to discover topics based on word co- occurrence Rank segments based on similarity to topic

Dataset Note that the paper uses a larger set containing ~ 4000 images (MSRC_v0) MSRC_v2 dataset 23 categories 591 images 1648 objects Distribution of categories in MSRC_v2

Implementation details –Dense sift on edge points and 3 different scales –2000 visual words –8 segmentations using different parameters –~ 40k segments in total –LDA takes ~ 10 mins

Ground Truth Segments ground truth segments are directly used number of topics set to 25 (~ num categories) Top 20 segments in terms of similarity of word distribution to a topic

Ground Truth Segments Number of topics = 25

Ground Truth Segments Number of topics = 50

Ground Truth Segments Number of topics = 75

Quantitative results Overlap score on top 20 segments Average precision (area under precision-recall curve) Ground Truth Segments

Multiple segmentations Normalized cuts –k = {3, 5, 7, 9} –2 resolutions

Multiple segmentations

Multiple segmentations vs. Ground truth

Quantitative results Overlap score on top 20 segments Average precision (area under precision-recall curve) Multiple segmentations

Effect of number of images Overlap score on top 20 segments topics are easier to discover with more object instances

Single segmentation Multiple segmentations vs. Single segmentation Single segmentation returns partial objects for some classes

Single segmentation Multiple segmentations vs. Single segmentation Single segmentation returns partial objects for some classes

Single segmentation Multiple segmentations vs. Single segmentation

Quantitative results Overlap score on top 20 segments Average precision (area under precision-recall curve) Single/Multiple segmentations

Hierarchical segmentation Hierarchical segmentationvs. Multiplesegmentation (Ncuts) Watershed threshold set such there are 12 leaf nodes and entire hierarchical tree is used by LDA

Hierarchical segmentation Hierarchical segmentationvs. Multiplesegmentation (Ncuts)

Hierarchical segmentation Hierarchical segmentationvs. Multiplesegmentation (Ncuts)

Quantitative results Overlap score on top 20 segments Hierarchical segmentation Contour-based watershed method does better for objects with few internal contours (grass, sky) is worse for objects with large number of contours (flower, airplane)

Conclusion Generating multiple segmentations –ncuts and watershed provide different tradeoffs –bottom-up segmentation needs different parameters for different objects Discovering objects using LDA –number of topics matters quite a bit –topics are easier to discover with more examples –multiple segmentation does better than because different objects require different parameters –contour-based watershed method does better for objects with few internal contours