Learning Globally-Consistent Local Distance Functions for Shape-Based Image Retrieval and Classification Computer Vision, ICCV IEEE 11th International Conference on Andrea Frome, EECS, UC Berkeley Yoram Singer, Google, Inc Fei Sha, EECS, UC Berkeley Jitendra Malik, EECS, UC Berkeley

Outline Introduction Training step Testing step Experiment & Result Conclusion

Outline Introduction Training step Testing step Experiment & Result Conclusion

What we do? Goal – classify an image to a more appropriate category Machine learning Two steps – Training step – Testing step

Outline Introduction Training step Testing step Experiment & Result Conclusion

Flow chart: training Generate features each image from dataset, ex: SIFT or geometric blur Input distances to SVM for training, evaluate W Compute distance dji, dki

Flow chart: training Generate features each image from dataset, ex: SIFT or geometric blur Input distances to SVM for training, evaluate W Compute distance dji, dki

Choosing features Dataset: Caltech101 Patch-based Features – SIFT Old school – Geometric Blur It’s a notion of blurring The measure of similarity between image patches The extension of Gaussian blur

Geometric blur

Flow chart: training Generate features each image from dataset, ex: SIFT or geometric blur Input distances to SVM for training, evaluate W Compute distance dji, dki

Triplet dji is the distance from image j to i It’s not symmetric, ex: dji ≠ dij dki > dji djidki

How to compute distance L2 norm dji, 1 m features dji, 1 distance vector dji Image j Image i

Example Given 101 category, 15 images each category 101*15 Feature j 101*15 distance vector Image j vs training data

Flow chart: training Generate features each image from dataset, ex: SIFT or geometric blur Input distances to SVM for training, evaluate W Compute distance dji, dki

Machine learning: SVM Support Vector Machine Function: Classify prediction Supervised learning Training data are n dimension vector

Example Male investigate – Annual income – Free time Have girlfriend?

Ex: Training data

space free income vector

Mathematical expression(1/2)

Mathematical expression(2/2)

Support vector Model free income

But the world is not so ideal.

Real world data

Hyper-dimension

Error cut

SVM standard mathematical expression Trade-off

In this paper Goal: to get the weight vector W 101*15 feature Image weight wj of W wj, 1 wj

Visualization of the weights

How to choose Triplets? Reference Image – Good friend - In the same class – Bad friend - In the different class Ex: 101category, 15 images per category – 14 good friends & 15*100(1500) bad friends – 15*101(1515) reference images – total of about 31.8 million triplets

Mathematical expression(1/2) Idealistic: Scaling: Different: The length of Weight i 00 triplet

Mathematical expression(2/2) Empirical loss: Vector machine:

Dual problem

Dual variable Iterate the dual variables:

Early stopping Satisfy KTT condition – In mathematics, a solution in nonlinear programming to be optimal.mathematicsnonlinear programming Threshold – Dual variable update falls below a value

Outline Introduction Training step Testing step Experiment & Result Conclusion

Flow chart: testing Query an image i Output the most appropriate category Calculate Dxi, x is all training data, except itself.

Flow chart: testing Query an image i Output the most appropriate category Calculate Dxi, x is all training data, except itself.

Query image? Goal: classify the query image to an appropriate class Using the remaining images in the dataset as the query image

Flow chart: testing Query an image i Output the most appropriate category Calculate Dxi, x is all training data, except itself.

Distance function(1/2) Query image i Image i feature 101*15 distance vector Image i vs all training data dxi, 1

Distance function(2/2) 101*15 Image I vs all the training data Dji

Flow chart: testing Query an image i Output the most appropriate category Calculate Dxi, x is all training data, except itself.

How to choose the best image? Modified 3-NN classifier no two images agree on the class within the top 10 – Take the class of the top-ranked image of the 10

Outline Introduction Training step Testing step Experiment & Result Conclusion

Experiment & Result Caltech 101 Feature – Geometric blur (shape feature) – HSV histograms (color feature) 5, 10, 15, 20 training images per category

Confusion matrix for 15

Outline Introduction Training step Testing step Experiment & Result Conclusion

Learning Globally-Consistent Local Distance Functions for Shape-Based Image Retrieval and Classification