Ziming Zhang *, Ze-Nian Li, Mark Drew School of Computing Science, Simon Fraser University, Vancouver, B.C., Canada {zza27, li, Learning Image Similarities via Probabilistic Feature Matching 1 *This work was done when the author was in SFU.

Outline 2 Introduction Probabilistic-matching based similarity learning Probabilistic feature matching function Probabilistic feature matching learning Experiments Conclusion

Introduction 3 Object-based image similarity Ideally two images are supposed to have a higher similarity if they contain similar objects. Feature matching A natural way to measure the image similarities Many different criteria In this talk, we simply match features only based on their appearance information

4 >

Introduction 5 Several relevant feature matching approaches Summation kernel Max-selection kernel Optimal assignment kernel

Introduction 6

7 Our approach is a generalization of a family of SIMILARITY learning approaches, including the three above. Similarity matrix ≠ Kernel A kernel matrix can be considered as a special similarity matrix (i.e. symmetric positive semi-definite) Classification with Support Vector Machines (SVM)

Probabilistic-matching based similarity learning 8 How to learn these feature matching probabilities?

Feature Matching Function 9 Given two images X={x 1,…,x |X| } and Y={y 1,…,y |Y| }, a feature matching function α can be defined as Explanations of matching processes in the Summation Kernel, Max-selection Kernel, and Optimal Assignment Kernel using feature matching functions K sum : K max : K OA :

Probabilistic Feature Matching Function 10 Our probabilistic feature matching function α is defined in the vector space covered by the following convex set: Total matching probability of one feature Total matching probability of all features Each matching probability

Probabilistic Feature Matching Learning 11 Data-dependent optimization Image similarity Distribution sparseness (or Regulizer)

Probabilistic Feature Matching Learning 12 Theorems Consider max f(x) over x X, where f(x) is convex, and X is a closed convex set. If the optimum exists, a boundary point of X is the optimum. If a convex function f(x) attains its maximum on a convex polyhedron X with some extreme points, then this maximum is attained at an extreme point of X. Relation to K sum, K max, and K OA K sum : C=+∞ and H={i,j} K max : C=0 and H={i}, and C=0 and H={j} K OA : C=0 and H={i,j}

Probabilistic Feature Matching Learning 13 Proposition For two images X and Y, both the sparseness of α and their similarity will decrease monotonically with increasing the parameter C.

Probabilistic Feature Matching Learning 14

Experiments 15 Datasets: Graz Descriptor SIFT + dense sampling Image Representation 3*3 spatial Bag-of-Word histograms with 200 codewords Feature similarity: RBF- kernel with χ 2 distance 50 runs Graz-01 Graz-02

Experiments 16 Graz-01 (a) PFM 1 with H={i,j} (b) PFM 2 with H={i} or H={j} (c) PFM 3 with H= ф

Experiments 17 BikePersonAverage SPK [1] 86.3± ± PDK [2] 90.2± ± PFM 1 (C=0)90.6± ± PFM 2 (C=5)89.6± ± PFM 3 (C=+∞)89.6± ± Table 1. Comparison results between different approaches on Graz-01 (%) [1] Lazebnik et. al., “Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories,” in CVPR’06. [2] Ling and Soatto, “Proximity distribution kernels for geometric context in category recognition,” in ICCV’07.

Experiments 18 Graz-02 (a) PFM 1 with H={i,j}(b) PFM 2 with H={i} or H={j} (c) PFM 3 with H= ф

Experiments 19 BikePersonCarAverage Boost.+SIFT [3] Boost.+comb. [3] PDK+SIFT [2] PDK+hybrid [2] PFM 1 +SIFT (C=5) PFM 1 +SIFT (C=10) PFM 1 +SIFT (C=+∞) Table 2. Comparison results between different approaches on Graz-02 (%) Opelt et. al., “Generic object recognition with boosting,” PAMI, 2006.

Conclusion 20 Probabilistic feature matching scheme A generalization of a rich family of probabilistic feature matching approaches Easy to control the sparseness of the matching probability distributions and their corresponding image similarities

21 Thank you !!!