A Statistical Approach to Speed Up Ranking/Re-Ranking Hong-Ming Chen Advisor: Professor Shih-Fu Chang

Outline Flow chart of the overall work The idea of using statistical approach to do re- ranking – By feature locations relationship O(n 2 ) time complexity – By orientation relationship O(n) time complexity The re-rank accuracy is as good as RANSAC Experimental result evaluation

Flow Chart 1 – ranking components construction Dataset: Ukbench [1] [1] D. Nistér and H. Stewénius. Scalable recognition with a vocabulary tree. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, pages , June [2] Code Book Hierarchical k-means [1][2] Bag of Word histograms of the database images Query image Bag of Word histogram of the query image Respond top-N result

Flow Chart 2 – re-ranking components construction Respond top-N result Re-rank by RANSAC [3] [3] Peter Kovesi, Centre for Exploration Targeting School of Earth and Environment The University of Western Australia Re-rank by proposed statistical approach Result evaluation

1. Feature Locations Relationship SIFT features [4] are: – Invariant to translation, rotation and scaling – Partially invariant to local geometric distortion For an ideal similar image pair: – Only translation, rotation and scaling – The ratio of corresponding distance pairs should be constant. [4] David G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, 60, 2 (2004) P 1a P 1b P 2a Image AImage B dist1 dist2

1. Feature Locations Relationship SIFT features [4] are: – Invariant to translation, rotation and scaling – Partially invariant to local geometric distortion For a similar image pair with view angle difference: – Translation, rotation and scaling – Local geometric distortion, and wrong feature points matching – The ratio of corresponding distance pairs is near constant. [4] David G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, 60, 2 (2004) P 1a P 1b P 2a Image AImage B dist1 dist2

Example ukbench00000ukbench00001 Mean = 0.85 Variance = Total amount of match points: 554 Mean: scaling Variance: matching error, the smaller the better

1. Feature Locations Relationship Assumption after observation: – A similar image pair: a distribution with small distribution variance – A dissimilar image pair: a distribution with large distribution variance

Analysis of feature locations relationship Relationship of match pair numbers and average variances between similar image pairs and dissimilar image pairs Red: dissimilar image pairs Blue: similar image pairs

2. Feature orientation Relationship SIFT features [4] are: – Invariant to translation, rotation and scaling – Partially invariant to local geometric distortion For similar image pairs: – The rotation degree of P 1a -> P1b should be EQUAL to the rotation degree of P 2a -> P 2b [4] David G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, 60, 2 (2004) P 1a P 1b P 2a Image AImage B

Example ukbench00000 ukbench00001 Shift about pi/4 The rotation degree is about 50, Distance measured by histogram intersection

2. Feature orientation Relationship Assumption after observation: – A similar image pair: small orientation histogram distance – A dissimilar image pair: large orientation histogram distance

Analysis of Feature orientation Relationship Relationship of match pair numbers and average orientation intersection difference between similar image pairs and dissimilar image pairs Red: dissimilar image pairs Blue: similar image pairs

Why I zoom in the small-match-number portion of the diagrams?

Dataset and features discussion Ukbench dataset analysis: – 2550 classes, 4 images/class – Similar image pairs combination: C(4, 2) * 2550 = pairs High percentage of similar image pairs having small amount of match points. (with default ratio value = 0.6) The re-ranking criteria should have outstanding performance especially only having small match points amount. Match points #Accumulated #/%Match points #Accumulated #/% % % % % % % % % % % % %

Comparison of two re-ranking approach Match point # Similar image pairsDissimilar image pairs Variance of Scaling Distribution Orientation histogram difference Variance of Scaling Distribution Orientation histogram difference meanvarmeanvarmeanvarmeanvar overall

Comparison of two re-ranking approach Match point # Similar image pairsDissimilar image pairs Variance of Scaling Distribution Orientation histogram difference Variance of Scaling Distribution Orientation histogram difference meanvarmeanvarmeanvarmeanvar overall High variance of the variance of Scaling Distribution, even though the mean of it is quite distinctive.

Comparison of two re-ranking approach Match point # Similar image pairsDissimilar image pairs Variance of Scaling Distribution Orientation histogram difference Variance of Scaling Distribution Orientation histogram difference meanvarmeanvarmeanvarmeanvar overall The variance of orientation histogram difference are very small (with respect to its mean value) and stable.

Comparison of two re-ranking approach Match point # Similar image pairsDissimilar image pairs Variance of Scaling Distribution Orientation histogram difference Variance of Scaling Distribution Orientation histogram difference meanvarmeanvarmeanvarmeanvar overall Overall, the orientation histogram difference can clearly separate similar/dissimilar image pairs, because of its large distance of mean value and quite small variance.

Comparison of two re-ranking approach Match point # Similar image pairsDissimilar image pairs Variance of Scaling Distribution Orientation histogram difference Variance of Scaling Distribution Orientation histogram difference meanvarmeanvarmeanvarmeanvar overall When match points are more than 5, the orientation histogram difference can roughly separate similar and dissimilar image pairs.

Comparison of two re-ranking approach Match point # Similar image pairsDissimilar image pairs Variance of Scaling Distribution Orientation histogram difference Variance of Scaling Distribution Orientation histogram difference meanvarmeanvarmeanvarmeanvar overall When match points are more than 10, the orientation histogram difference can clearly separate similar and dissimilar image pairs.

Experimental results discussion 1. the impact of k values (cluster centers) K=1000K=4096K=10000K=50625K= Recall = 1 (33%) Recall = 2 (66%) Recall = 3 (100%) K=1000 K=4096 K=10000 K=50625 K=100000

Experimental results discussion 2. the impact of looking up code book by different approach: – A. by tracing the vocabulary tree [1]: efficient, but the result is not optimal – B. by scanning the whole code book: very slow, but guarantees a optimal BoW result with respect to the K centers K=1000(by tree)K=1000K=10000(by tree)K=10000 Recall = 1 (33%) Recall = 2 (66%) Recall = 3 (100%) K=1000: decoded by tree K=1000: decoded directly K=10000: decoded by tree K=10000: decoded directly

K=1000 Ground truth Rotation Scale var + rotation RANSAC Scale var Original Ground truthRotationScale var + rotation RANSACScale varOriginal Re-rank depth =20

K=50625 Ground truth Rotation Scale var + rotation RANSAC Scale var Original Ground truthRotationScale var + rotation RANSACScale varOriginal Re-rank depth =20

Experimental result -- all Re-rank depth = 20 distributionK=1000K=4096K=10000K=50625 Recall = 1 (33%) Recall = 2 (66%) Recall = 3 (100%) rotationK=1000K=4096K=10000K=50625 Recall = 1 (33%) Recall = 2 (66%) Recall = 3 (100%) K=1000K=4096K=10000K=50625K= Recall = 1 (33%) Recall = 2 (66%) Recall = 3 (100%)

RANSACK=1000K=4096K=10000K=50625 Recall = 1 (33%) Recall = 2 (66%) Recall = 3 (100%) Ground TruthK=1000K=4096K=10000K=50625 Recall = 1 (33%) Recall = 2 (66%) Recall = 3 (100%) Dist+RoK=1000K=4096K=10000K=50625 Recall = 1 (33%) Recall = 2 (66%) Recall = 3 (100%)

Time Complexity Analysis RANSAC: O(Kn): – K: random subset tried – n: input data size – no upper bound on the time it takes to compute the parameters Distribution of Feature Location distance relationship: – O(n 2 ) : distribution consists of all distance relationships – O(n): when n (match point number) is large enough, we can subsample “reliable enough” amount of samples to form the distribution The distance of orientation histograms of matched SIFT features: – O(n) : to generate rotation angle histograms of matched SIFT features – Constant time for compute rotation angles – Only little overhead with respect to searching match points

Future work We have: – 1. Scale information – 2. Orientation information – 3. Trivial to find translation – A good initial guess for precise homography matrix estimation? Applied the current approach to quantized SIFT features: – Using a code word to represent a interesting point, rather than applying 128 dimension vector Moving from exact 1-1 mapping to many-to-many mapping. – I’ve tried to solve this problem. However, there are now no satisfying results at this stage.

Reference [1] D. Nistér and H. Stewénius. Scalable recognition with a vocabulary tree. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, pages , June [2] [3] Peter Kovesi, Centre for Exploration Targeting School of Earth and Environment The University of Western Australiahttp:// [4] David G. Lowe, "Distinctive image features from scale- invariant keypoints," International Journal of Computer Vision, 60, 2 (2004)