MIT CSAIL Vision interfaces Approximate Correspondences in High Dimensions Kristen Grauman* Trevor Darrell MIT CSAIL (*) UT Austin…

MIT CSAIL Vision interfaces Key challenges: robustness IlluminationObject pose Clutter Viewpoint Intra-class appearance Occlusions

MIT CSAIL Vision interfaces Key challenges: efficiency Thousands to millions of pixels in an image 3,000-30,000 human recognizable object categories Billions of images indexed by Google Image Search 18 billion+ prints produced from digital camera images in million camera phones sold in 2005

MIT CSAIL Vision interfaces Local representations Superpixels [Ren et al.] Shape context [Belongie et al.] Maximally Stable Extremal Regions [Matas et al.] Geometric Blur [Berg et al.] SIFT [Lowe] Salient regions [Kadir et al.] Harris-Affine [Schmid et al.] Spin images [Johnson and Hebert] Describe component regions or patches separately

MIT CSAIL Vision interfaces How to handle sets of features? Each instance is unordered set of vectors Varying number of vectors per instance

MIT CSAIL Vision interfaces Partial matching Compare sets by computing a partial matching between their features.

MIT CSAIL Vision interfaces Pyramid match overview optimal partial matching

MIT CSAIL Vision interfaces Computing the partial matching Optimal matching Greedy matching Pyramid match for sets with features of dimension

MIT CSAIL Vision interfaces Pyramid match overview Place multi-dimensional, multi-resolution grid over point sets Consider points matched at finest resolution where they fall into same grid cell Approximate optimal similarity with worst case similarity within pyramid cell No explicit search for matches! Pyramid match measures similarity of a partial matching between two sets:

MIT CSAIL Vision interfaces Pyramid match Number of newly matched pairs at level i Measure of difficulty of a match at level i Approximate partial match similarity [Grauman and Darrell, ICCV 2005]

MIT CSAIL Vision interfaces Pyramid extraction, Histogram pyramid: level i has bins of size

MIT CSAIL Vision interfaces Counting matches Histogram intersection

MIT CSAIL Vision interfaces Example pyramid match

MIT CSAIL Vision interfaces Example pyramid match

MIT CSAIL Vision interfaces Example pyramid match

MIT CSAIL Vision interfaces Example pyramid match pyramid match optimal match

MIT CSAIL Vision interfaces x Randomly generated uniformly distributed point sets with m= 5 to 100, d=2 Approximating the optimal partial matching

MIT CSAIL Vision interfaces PM preserves rank…

MIT CSAIL Vision interfaces and is robust to clutter…

MIT CSAIL Vision interfaces Learning with the pyramid match Kernel-based methods –Embed data into a Euclidean space via a similarity function (kernel), then seek linear relationships among embedded data –Efficient and good generalization –Include classification, regression, clustering, dimensionality reduction,… Pyramid match forms a Mercer kernel

MIT CSAIL Vision interfaces ComplexityKernel Pyramid match Match [Wallraven et al.] Time (s) Accuracy Category recognition results ETH-80 data set Mean number of features

MIT CSAIL Vision interfaces s / match5 s / match Category recognition results Pyramid match kernel over spatial features with quantized appearance 2004 Time of publication 6/05 12/05 3/06 6/06

MIT CSAIL Vision interfaces But rectangular histogram may scale poorly with input dimension… Build data-dependent histogram structure… New Vocabulary-guided PM [NIPS 06]: Hierarchical k-means over training set Irregular cells; record diameter of each bin VG pyramid structure stored O(k L ); stored once Individual Histograms still stored sparsely Vocabulary-guided pyramid match

MIT CSAIL Vision interfaces Vocabulary-guided pyramid match Uniform bins Tune pyramid partitions to the feature distribution Accurate for d > 100 Requires initial corpus of features to determine pyramid structure Small cost increase over uniform bins: kL distances against bin centers to insert points Vocabulary- guided bins

MIT CSAIL Vision interfaces Vocabulary-guided pyramid match n ij (X) : hist. X level i cell j w ij : weight for hist. X level i cell j (1)~= diameter of cell (2)~= d ij (X) + d ij (Y) (d ij (H)=max dist of H’s pts in cell i,j to center) c h (n) : child h of node n c 2 (n 11 ) Mercer kernel Upper bound w ij * (# matches in cell j level i - # matches in children) W * # new level i

MIT CSAIL Vision interfaces Results: Evaluation criteria Quality of match scores How similar are the rankings produced by the approximate measure to those produced by the optimal measure? Quality of correspondences How similar is the approximate correspondence field to the optimal one? Object recognition accuracy Used as a match kernel over feature sets, what is the recognition output?

MIT CSAIL Vision interfaces Match score quality Uniform bin pyramid match Vocabulary- guided pyramid match ETH-80 images, sets of SIFT features d=8d=128 d=8 Dense SIFT (d=128) k=10, L=5 for VG PM; PCA for low-dim feats

MIT CSAIL Vision interfaces ETH-80 images, sets of SIFT features Match score quality

MIT CSAIL Vision interfaces Spearman correlation Correlation coefficient to measure how well two ordinal rankings agree rank value in true ordering corresponding rank assigned by approximate ordering

MIT CSAIL Vision interfaces Bin structure and match counts Data-dependent bins allow more gradual distance ranges d=8d=13 d=68 d=3 d=113 d=128

MIT CSAIL Vision interfaces Approximate correspondences Use pyramid intersections to compute smaller explicit matchings.

MIT CSAIL Vision interfaces Approximate correspondences Use pyramid intersections to compute smaller explicit matchings. optimal per bin random per bin

MIT CSAIL Vision interfaces Correspondence examples

MIT CSAIL Vision interfaces ETH-80 images, sets of SIFT descriptors Approximate correspondences

MIT CSAIL Vision interfaces ETH-80 images, sets of SIFT descriptors Approximate correspondences

MIT CSAIL Vision interfaces Impact on recognition accuracy VG-PMK as kernel for SVM Caltech-4 data set SIFT descriptors extracted at Harris and MSER interest points

MIT CSAIL Vision interfaces Sets of features elsewhere diseases as sets of gene expressions documents as bags of words methods as sets of instructions