An Approximate Nearest Neighbor Retrieval Scheme for Computationally Intensive Distance Measures Pratyush Bhatt MS by Research(CVIT)

Nearest Neighbor Retrieval Representation of an object – Fixed Length – Variable Length (Dis)similarity Function (Distance Measure) Neighborhood Nearest neighbor retrieval problem can now be formalized as retrieving objects similar to a given object, where similarity is in accordance to a given similarity function

Need of NN retrieval schemes If the search space is small, Sequential search is applied for accurate results. With increase in memory, volume of data stored online has increased. – Sequential search is time consuming – Need to index data for fast retrieval – Birth of NN search algorithms

Computationally Expensive Distance Measures X 1 more similar to X 2 than X 3 visually L1 distance between X 1 and X 2 is more than X 1 and X 3

Time complexity is super-linear to length of input. Edit Distance: O(d 2 ) Chamfer Distance

Nearest Neighbor Classification Computationally expensive distance functions – Nearest neighbor classifiers often impractical for real applications. – Takes over 20 minutes to classify a single object on a modern PC using an optimized C++ implementation. – Larger the available training data, better will be the accuracy but at the cost of high computation time.

Motivation(Approx. NN) Why to find similarity with all samples when decision is based on top K ? How to find top K without finding similarity with all the samples ? Solution: Compute K1 > K Approx. NN and find best K in that list. In order to compute K1-NN, use fewer explicit matches.

Problem Statement Improve nearest neighbor retrieval and classification performance in spaces with computationally expensive distance measures. Generate an expansible approximate nearest neighbor list for a given query. Dealing with points in non-metric space.

Metric Space

Tree based (KD-tree, R-Tree) Hashing Based

KD Tree for Metric Space

Non-Metric Space

Wrist Rotation

Applications Classification based on K-NN – K is determined empirically Identification – Stop when similarity is above a fixed threshold Retrieval Applications – Optimizing network usage in peer-to-peer computer networks. – Content-based retrieval systems Concept of similarity is abstract – Need to generate expansible list – Learn from user feedbacks

Challenges Accuracy depends on the similarity function used to compute Approx. NN List can not be pre-computed, should be generated on-the-fly Should be incremental to support scalability Non-metric space – Prohibit application of triangular inequality

Manifold Theory Given set of high dimensional points, where Find embedding set in a low dimensional space – Preserves local similarity – Assumption: Data is well distributed

Run MDS on similarity matrix to get Embedding of new sample is given by mean of column of squared matrix requires computation of similarity of new sample to points

Related Work FastMap, Random Reference Objects, Random Line Projections, VP-Trees – Finds embedding of the query by computing only a few exact distances – Assumption: Triangular Inequality BoostMap: Used AdaBoost to combine many simple 1-D embedding FastMapLipschitz Embedding

Problem Formulation Goal : Compute approx. NN of a query point q, from a set S of N points in accordance to similarity function, F. Solution : – Split the data into a multi-level hierarchy – Exploit local similarity property to direct the search from top to bottom

Hierarchical Local Maps(HLM)

1. FIND SIMILARITY OF QUERY WITH POINTS AT TOPMOST LEVEL 2. IDENTIFY NEAREST NEIGHBORS3. GET CHILDREN OF NEAREST NEIGHBOR HOW TO FIND SIMILARITY OF QUERY WITH THESE SAMPLES WITHOUT EXPLICITLY CALCULATING IT ? 4. PROJECT POINTS ON THE MANIFOLD 5. PROJECT QUERY ON THE MANIFOLD 6. FIND NEAREST NEIGHBOURS OF QUERY IN THIS METRIC SPACE 7. IDENTIFY THESE POINTS IN THE TREE 8. TRAVERSE DOWN IN SAME FASHION

Results Unipen Handwriting Database – digit examples – Divided into training and testing set with 2:1 ratio – Distance Measure : DTW

Number of DTW Computations for K nearest neighbor retrieval

Classification Accuracy on UNIPEN Dataset using exact and approximate k-NN Classification Accuracy K DTW HLM Difference Average No. of DTW computations done by HLM : 160 Expected No. of DTW computations done in brute-force : 5315

Biometric ( Special Case) High Inter-Class Variation Low Intra-Class Variation Low variation in inter-class distances Indexing for identification How to apply HLM in such cases ?? – Local similarity structure becomes degenerate destroying any manifold structure

Biometric Data

High dimensional data Relative Contrast :

Iris Feature Vector Length 1000 If same class- < 100 bit differ Else equal probability of each bit to match or not match – On average 500 bit differ Imposter Scores would be around 0.5

Biometric Data Such Distribution is bad for Indexing

Softness/Hardness Measure of overlap between genuine and imposter classes Soft Biometric (Face, Body Silhouette) – Poor Classification Accuracy – Better indexing – Correlates to multi dimensional point Hard Biometric (Iris, Fingerprint) – Good Classification Accuracy – Poor indexing – Correlates to high dimensional point Need a balance between two

Dataset CASIA Iris Image Database V3.0 – 855 images corresponding to 285 users in training and testing set – 3 samples per eye Simlarity Function – Hamming Distance – Euclidean Distance (Softer Metric) Average gray value of a block resulted in 160D feature vector Penetration Rate – Percentage of data on which we ran biometric matcher. False Reject Rate – Percentage of identification instances in which false rejection occurs

HLM on CASIA Iris Dataset

Synthetic Dataset Class center sampled from 1D gaussian (0,1) Generate d-dimensional by sampling d times Points of same class sampled from gaussian with mean as class centers and varying variance. Total Number of classes: 500 Points in same class – Training : 10 – Testing : 5

Indexing performance varying number of dimensions

Indexing performance varying within class to between class variance ratio

Contributions A representation scheme for objects in a dataset that allows for fast retrieval of approximate nearest neighbors in non- euclidean space. Search mechanism combined with filter and refine approach is proposed that minimizes the number of exact distance computations for computationally expensive distance measure. Study performance of our scheme on biometric data and study the parameters affecting its performance.

Conclusion and Future Work Local Similarity Property is well exploited by HLM Incremental and Scalable Softer biometric in filtering step combined with hard biometric in refine step would drastically reduce computation time Optimal construction of HLM Defining a measure for similarity function that allows hierarchical representation. Learn a function to find degree of indexibility – Extract parameters from data distribution and similarity function

Thank You Related Publication: Pratyush Bhatt and Anoop Namboodiri “Hierarchical Local Maps for Robust Approximate Nearest Neighbour Computation” Proceedings of the 7th International Conference on Advances in Pattern Recognition (ICAPR 2009), Feb. 4-6, 2009, Kolkatta, India.