Boundary Preserving Dense Local Regions Jaechul Kim and Kristen Grauman Univ. of Texas at Austin

Local feature detection A crucial building block for many applications Image retrieval Object recognition Image matching Key issue: How to detect local regions for feature extraction?

Related work Interest point detectors Dense sampling e.g., Matas et al. (BMVC 02), Jurie and Schmid (CVPR 04), Mikolajczyk and Schmid (IJCV 04) Dense sampling e.g., Nowak et al. (ECCV 06) Segmented regions and Superpixels e.g., Ren and Malik (ICCV 03) , Gu et al. (CVPR 09), Todorovic and Ahuja (CVPR 08), Malisiewicz and Efros (BMVC 07), Levinshtein et al. (ICCV 09) Hybrid e.g., Tuytelaars (CVPR 10), Koniusz and Mikolajczyk (BMVC 09)

What makes a good local feature detector? Desired properties: - Repeatable - Boundary-preserving - Distinctively shaped Existing methods lack one or more of these criteria, e.g., Lack repeatability Lack distinctive shape, straddle boundaries Segments Dense sampling Interest points

Our idea: Boundary Preserving Local Regions (BPLRs) Boundary preserving, dense extraction Segmentation-driven feature sampling and linking Repeatable local features capturing objects’ local shapes

Approach: Overview Sampling elements Linking elements Initial elements for each segment are sampled based on distance transform of the segment A segment Sampled elements Linking elements A single graph structure reflecting main shapes and segment layout Min. spanning tree Grouping elements Grouping neighboring elements into BPLR Neighbor elements BPLR

Approach: Sampling Input image Segment Sampled elements Linking Grouping Zoom-in view x An ”element” x Input image Segment Sampled elements from “all” segments Distance transform Dense regular grid Sampled elements

Sampling Approach: Linking Linking Grouping Minimum spanning tree Sampled elements’ locations (i.e., elements’ centers) Global linkage structure

Role of spanning tree linkage Sampling Role of spanning tree linkage Linking Grouping Min spanning tree prefers to link closer elements + Multiple sampling Due to distance transform-based sampling same-segment elements more likely linked Due to multiple segmentations elements in overlapping segments more likely linked

Approach: Grouping Intersection of topology and Euclidean neighbor Sampling Approach: Grouping Linking Grouping Descriptor Intersection of topology and Euclidean neighbor Reference element’s location BPLR Intersection of topology and Euclidean neighbor Reference element’s location Neighbor elements Reference element’s location Zoom-in view Reference element’s location Euclidean neighbor elements’ location Euclidean neighbor Reference element’s location Topological neighbor elements’ location Topological neighbor Example detections of BPLRs (Subset shown for visibility) Reference element’s location

Example matches of BPLRs Leak object boundary

Experiments 20-200 segments  ~7000 BPLRs in 400 x 300 image Tasks: 2-5 seconds to extract BPLRs per an image PHOG + gPb descriptor used Tasks: Repeatability Localization Foreground segmentation Object classification Baselines: Dense sampling (+ SIFT) MSER (+ SIFT) [1] Semi-local regions (+ SIFT) [2,3] Segmented regions (+ PHOG) [4] Superpixels [5] [1] Matas et al., BMVC 02. [2] Quack et al., ICCV 07. [3] Lee and Grauman, IJCV 09. [4] Arbelaez et al., CVPR 09. [5] Ren and Malik, ICCV 03.

Example feature extractions Proposed BPLRs (Subset shown for visibility) Segmented regions Superpixels Interest regions (MSERs) Dense sampling

Repeatability for object categories Bounding Box Hit Rate – False Positive Rate [Quack et al. 2007] Applelogo Test image Giraffe Bottle Train images Swan Mug True match False positive Comparison to baseline region detectors on ETHZ shape classes

Localization accuracy Bounding Box Overlapping Score – Recall Applelogo Giraffe Bottle Compute overlapping score by projecting the training exemplar’s bounding box into the test image Swan Mug Comparison to baseline region detectors on ETHZ shape classes

Localization accuracy Test image Database images with best matches to test BPLRs

Foreground segmentation Replacing superpixels with BPLRs in GrabCut segmentation Approach Accuracy(%) BPLR + GrabCut (Ours) 85.6 Superpixel + GrabCut 81.5 Superpixel ClassCut (Alexe et al., ECCV 10) 83.6 Superpixel Spatial Topic Model (Cao et al., ICCV 07) 67.0 Foreground segmentation in Caltech-28 dataset

Object classification Nearest-neighbor results on Caltech-101 benchmark Feature Accuracy(%) BPLR + PHOG (Ours) 61.1 Dense + SIFT 55.2 Segment + PHOG 37.6 Dense + PHOG 27.9 Comparison of features using the same Naïve Bayes NN [Boiman et al. 2008] classifier.

Conclusion Dense local detector that preserves object boundaries Capture object’s local shape in a repeatable manner Feature sampling and linking driven by segmentation Generic bottom-up extraction Code available: http://vision.cs.utexas.edu/projects/bplr/bplr.html