Grassmannian Hashing for Subspace searching

Slides:



Advertisements
Similar presentations
Object Recognition Using Locality-Sensitive Hashing of Shape Contexts Andrea Frome, Jitendra Malik Presented by Ilias Apostolopoulos.
Advertisements

Aggregating local image descriptors into compact codes
Nearest Neighbor Search in High Dimensions Seminar in Algorithms and Geometry Mica Arie-Nachimson and Daniel Glasner April 2009.
Principal Component Analysis Based on L1-Norm Maximization Nojun Kwak IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008.
Overcoming the L 1 Non- Embeddability Barrier Robert Krauthgamer (Weizmann Institute) Joint work with Alexandr Andoni and Piotr Indyk (MIT)
Big Data Lecture 6: Locality Sensitive Hashing (LSH)
Searching on Multi-Dimensional Data
MIT CSAIL Vision interfaces Towards efficient matching with random hashing methods… Kristen Grauman Gregory Shakhnarovich Trevor Darrell.
Hongliang Li, Senior Member, IEEE, Linfeng Xu, Member, IEEE, and Guanghui Liu Face Hallucination via Similarity Constraints.
Similarity Search in High Dimensions via Hashing
A NOVEL LOCAL FEATURE DESCRIPTOR FOR IMAGE MATCHING Heng Yang, Qing Wang ICME 2008.
Mixed-Resolution Patch- Matching (MRPM) Harshit Sureka and P.J. Narayanan (ECCV 2012) Presentation by Yaniv Romano 1.
Low Complexity Keypoint Recognition and Pose Estimation Vincent Lepetit.
Data Structures and Functional Programming Algorithms for Big Data Ramin Zabih Cornell University Fall 2012.
VLSH: Voronoi-based Locality Sensitive Hashing Sung-eui Yoon Authors: Lin Loi, Jae-Pil Heo, Junghwan Lee, and Sung-Eui Yoon KAIST
Dimensionality reduction. Outline From distances to points : – MultiDimensional Scaling (MDS) – FastMap Dimensionality Reductions or data projections.
Coherency Sensitive Hashing (CSH) Simon Korman and Shai Avidan Dept. of Electrical Engineering Tel Aviv University ICCV2011 | 13th International Conference.
DIMENSIONALITY REDUCTION BY RANDOM PROJECTION AND LATENT SEMANTIC INDEXING Jessica Lin and Dimitrios Gunopulos Ângelo Cardoso IST/UTL December
Fast and Compact Retrieval Methods in Computer Vision Part II A. Torralba, R. Fergus and Y. Weiss. Small Codes and Large Image Databases for Recognition.
Given by: Erez Eyal Uri Klein Lecture Outline Exact Nearest Neighbor search Exact Nearest Neighbor search Definition Definition Low dimensions Low dimensions.
1 Jun Wang, 2 Sanjiv Kumar, and 1 Shih-Fu Chang 1 Columbia University, New York, USA 2 Google Research, New York, USA Sequential Projection Learning for.
Approximate Nearest Subspace Search with Applications to Pattern Recognition Ronen Basri, Tal Hassner, Lihi Zelnik-Manor presented by Andrew Guillory and.
Sparse Solutions for Large Scale Kernel Machines Taher Dameh CMPT820-Multimedia Systems Dec 2 nd, 2010.
Face Recognition Based on 3D Shape Estimation
Similarity Search in High Dimensions via Hashing Aristides Gionis, Protr Indyk and Rajeev Motwani Department of Computer Science Stanford University presented.
1 An Empirical Study on Large-Scale Content-Based Image Retrieval Group Meeting Presented by Wyman
Nearest Neighbor Retrieval Using Distance-Based Hashing Michalis Potamias and Panagiotis Papapetrou supervised by Prof George Kollios A method is proposed.
Cliff Rhyne and Jerry Fu June 5, 2007 Parallel Image Segmenter CSE 262 Spring 2007 Project Final Presentation.
Dimensionality Reduction
Face Processing System Presented by: Harvest Jang Group meeting Fall 2002.
Approximate Nearest Subspace Search with applications to pattern recognition Ronen Basri Tal Hassner Lihi Zelnik-Manor Weizmann Institute Caltech.
FLANN Fast Library for Approximate Nearest Neighbors
Image Analogies Aaron Hertzmann (1,2) Charles E. Jacobs (2) Nuria Oliver (2) Brian Curless (3) David H. Salesin (2,3) 1 New York University 1 New York.
Indexing Techniques Mei-Chen Yeh.
Person-Specific Domain Adaptation with Applications to Heterogeneous Face Recognition (HFR) Presenter: Yao-Hung Tsai Dept. of Electrical Engineering, NTU.
Nearest Neighbor Paul Hsiung March 16, Quick Review of NN Set of points P Query point q Distance metric d Find p in P such that d(p,q) < d(p’,q)
Fast Similarity Search for Learned Metrics Prateek Jain, Brian Kulis, and Kristen Grauman Department of Computer Sciences University of Texas at Austin.
80 million tiny images: a large dataset for non-parametric object and scene recognition CS 4763 Multimedia Systems Spring 2008.
CSE 185 Introduction to Computer Vision Face Recognition.
Similarity Searching in High Dimensions via Hashing Paper by: Aristides Gionis, Poitr Indyk, Rajeev Motwani.
Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality Piotr Indyk, Rajeev Motwani The 30 th annual ACM symposium on theory of computing.
A Fast LBG Codebook Training Algorithm for Vector Quantization Presented by 蔡進義.
Optimal Data-Dependent Hashing for Nearest Neighbor Search Alex Andoni (Columbia University) Joint work with: Ilya Razenshteyn.
October 3, 2013Computer Vision Lecture 10: Contour Fitting 1 Edge Relaxation Typically, this technique works on crack edges: pixelpixelpixel pixelpixelpixelebg.
Clustering Data Streams
Computational Physics (Lecture 10)
Fast nearest neighbor searches in high dimensions Sami Sieranoja
Instance Based Learning
Efficient Image Classification on Vertically Decomposed Data
کاربرد نگاشت با حفظ تنکی در شناسایی چهره
Fast Preprocessing for Robust Face Sketch Synthesis
Sublinear Algorithmic Tools 3
Lecture 11: Nearest Neighbor Search
Enhanced-alignment Measure for Binary Foreground Map Evaluation
Efficient Image Classification on Vertically Decomposed Data
K Nearest Neighbor Classification
Randomized Algorithms CS648
Efficient Distribution-based Feature Search in Multi-field Datasets Ohio State University (Shen) Problem: How to efficiently search for distribution-based.
Fast Sequence Alignments
Locality Sensitive Hashing
University of Crete Department Computer Science CS-562
CS5112: Algorithms and Data Structures for Applications
Minwise Hashing and Efficient Search
President’s Day Lecture: Advanced Nearest Neighbor Search
Topological Signatures For Fast Mobility Analysis
Ronen Basri Tal Hassner Lihi Zelnik-Manor Weizmann Institute Caltech
Memory-Based Learning Instance-Based Learning K-Nearest Neighbor
Scalable light field coding using weighted binary images
Random Neural Network Texture Model
Outline Texture modeling - continued Markov Random Field models
Presentation transcript:

Grassmannian Hashing for Subspace searching The topic of the talk is Coherency Sensitive Hashing This is a joint work with Shai Avidan from the Tel-Aviv University Grassmannian Hashing for Subspace searching *Xu Wang, †Stefan Atev, ‡John Wright, *Gilad Lerman *University of Minnesota †Proto Labs Inc. ‡Columbia University

Fast search for big dataset The problem we tackle is that of finding a Nearest Neighbor Field between a pair of related images. That is, for each patch of an image ‘A’, one needs to find the most similar patch in an image ‘B’. This is a dense type of problem, where we consider sliding window kind of patches in both images. For this reason, exact Nearest Neighbor calculation is very difficult so people settle for approximations. The way we evaluate the accuracy of the mapping is by taking the average distance between matching patches. -------------------------------------------------------------- Computing Approximate Nearest Neighbor Fields (ANNF) is an important building block in many computer vision and graphics applications such as texture synthesis [10], image editing [18] and image denoising [7] This is a challenging task because the number of patches in an image is in the millions and one needs to find Approximate Nearest Neighbors (ANN) for each patch in real or near real time. Fast search for big dataset Approximate nearest subspace Dataset : a set of subspaces 𝑋 Query : a subspace 𝑄 𝑄 Relaxation: Nearest Approximate nearest Sublinear time algorithm

Capture the appearance of faces What do we need this kind of NN Field for? One good example is in the Non-Local-Means denoising algorithm. In this algorithm, each pixel is replaced by an average of all other pixels in the image, where the weight in the averaging is taken according to the distance between the containing patches. For computational reasons, one could only use a small neighborhood around each pixel. However, if we had an NNf between the image and itself, we could be able to incorporate in the averaging other patches from across the image. -------------------------------------------------------------- Computing Approximate Nearest Neighbor Fields (ANNF) is an important building block in many computer vision and graphics applications such as texture synthesis [10], image editing [18] and image denoising [7] This is a challenging task because the number of patches in an image is in themillions and one needs to find Approximate Nearest Neighbors (ANN) for each patch in real or near real time. Why Subspaces? Capture the appearance of faces One person One subspace 𝑅 𝑛×𝑘 Different lighting, etc.

Why Subspaces? Face recognition What do we need this kind of NN Field for? One good example is in the Non-Local-Means denoising algorithm. In this algorithm, each pixel is replaced by an average of all other pixels in the image, where the weight in the averaging is taken according to the distance between the containing patches. For computational reasons, one could only use a small neighborhood around each pixel. However, if we had an NNf between the image and itself, we could be able to incorporate in the averaging other patches from across the image. -------------------------------------------------------------- Computing Approximate Nearest Neighbor Fields (ANNF) is an important building block in many computer vision and graphics applications such as texture synthesis [10], image editing [18] and image denoising [7] This is a challenging task because the number of patches in an image is in themillions and one needs to find Approximate Nearest Neighbors (ANN) for each patch in real or near real time. Why Subspaces? Face recognition 𝑅 𝑛×𝑘

Nearest neighbor field (NNF) What do we need this kind of NN Field for? One good example is in the Non-Local-Means denoising algorithm. In this algorithm, each pixel is replaced by an average of all other pixels in the image, where the weight in the averaging is taken according to the distance between the containing patches. For computational reasons, one could only use a small neighborhood around each pixel. However, if we had an NNf between the image and itself, we could be able to incorporate in the averaging other patches from across the image. -------------------------------------------------------------- Computing Approximate Nearest Neighbor Fields (ANNF) is an important building block in many computer vision and graphics applications such as texture synthesis [10], image editing [18] and image denoising [7] This is a challenging task because the number of patches in an image is in themillions and one needs to find Approximate Nearest Neighbors (ANN) for each patch in real or near real time. Why Subspaces? Nearest neighbor field (NNF) Patch match Image A Image B Application of NNF: Image completion Image denoising Object Removal by Examplar Based Inpainting. Criminisi et al’, CVPR 2003 Non Local Means. Baudes, Coll, Morel. CVPR 2005

Capture local intensities of an image What do we need this kind of NN Field for? One good example is in the Non-Local-Means denoising algorithm. In this algorithm, each pixel is replaced by an average of all other pixels in the image, where the weight in the averaging is taken according to the distance between the containing patches. For computational reasons, one could only use a small neighborhood around each pixel. However, if we had an NNf between the image and itself, we could be able to incorporate in the averaging other patches from across the image. -------------------------------------------------------------- Computing Approximate Nearest Neighbor Fields (ANNF) is an important building block in many computer vision and graphics applications such as texture synthesis [10], image editing [18] and image denoising [7] This is a challenging task because the number of patches in an image is in themillions and one needs to find Approximate Nearest Neighbors (ANN) for each patch in real or near real time. Why Subspaces? Capture local intensities of an image 𝑘𝑏𝑦𝑘 patch = dimensional point 𝑘×𝑘 𝑅 𝑘×𝑘 Image A

Capture local intensities of an image What do we need this kind of NN Field for? One good example is in the Non-Local-Means denoising algorithm. In this algorithm, each pixel is replaced by an average of all other pixels in the image, where the weight in the averaging is taken according to the distance between the containing patches. For computational reasons, one could only use a small neighborhood around each pixel. However, if we had an NNf between the image and itself, we could be able to incorporate in the averaging other patches from across the image. -------------------------------------------------------------- Computing Approximate Nearest Neighbor Fields (ANNF) is an important building block in many computer vision and graphics applications such as texture synthesis [10], image editing [18] and image denoising [7] This is a challenging task because the number of patches in an image is in themillions and one needs to find Approximate Nearest Neighbors (ANN) for each patch in real or near real time. Why Subspaces? Capture local intensities of an image Consider all patches (Red, Green) inside the bigger patch (Blue) ! 𝑅 𝑘×𝑘

Approximate Nearest Subspace The problem we tackle is that of finding a Nearest Neighbor Field between a pair of related images. That is, for each patch of an image ‘A’, one needs to find the most similar patch in an image ‘B’. This is a dense type of problem, where we consider sliding window kind of patches in both images. For this reason, exact Nearest Neighbor calculation is very difficult so people settle for approximations. The way we evaluate the accuracy of the mapping is by taking the average distance between matching patches. -------------------------------------------------------------- Computing Approximate Nearest Neighbor Fields (ANNF) is an important building block in many computer vision and graphics applications such as texture synthesis [10], image editing [18] and image denoising [7] This is a challenging task because the number of patches in an image is in the millions and one needs to find Approximate Nearest Neighbors (ANN) for each patch in real or near real time. Approximate Nearest Subspace ( )-Approximate nearest subspace search 𝑟,𝑐 Dataset : a set of subspaces 𝑋 Grassmannian distance Query : a subspace 𝑄 ( )-Approximation 𝑟,𝑐 𝑟: distance to the “nearest neighbor” 𝑐: approximation factor 𝑐𝑟 -neighbor 𝑟 -neighbor Approximate Nearest Neighbors: towards removing the curse of dimensionality. Piotr Indyk et al., STOC 1998

Previous works Two methods for subspace-related searching: The traditional way to go with computing NNFs is by searching for the nearest patches in appearance space. Throwing all the patches into high dimensional space, this becomes an ordinary Nearest Neighbor Search. The common indexing techniques for this are KD-Trees and Locality Sensitive Hashing. The nice thing about them is that they can be as accurate as needed, but this comes at a high cost in runtime. ------------------------------------------------------------------------------ In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate Previous works Two methods for subspace-related searching: Hashing hyperplane queries [Jain et al., 2010] Approx. nearest subspace (BHZ) [Basri et al., 2011] Subspace of dimension d Point Increase dimension Introduce distortion Orthogonal projection matrix of size D-by-D Subspaces in 𝑅 𝐷 Points in 𝑅 𝐷×𝐷

Methods for Nearest Neighbor Search The traditional way to go with computing NNFs is by searching for the nearest patches in appearance space. Throwing all the patches into high dimensional space, this becomes an ordinary Nearest Neighbor Search. The common indexing techniques for this are KD-Trees and Locality Sensitive Hashing. The nice thing about them is that they can be as accurate as needed, but this comes at a high cost in runtime. ------------------------------------------------------------------------------ In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate Methods for Nearest Neighbor Search Two popular methods: KD-trees [Arya et al., 1998] Locality Sensitive Hashing (LSH) [Indyk et al., 1998] Grassmannian Distance Locality Sensitive Hashing Grassmannian Hashing (GH)

The LSH for Points Project each point v on a random segmented line Coherency Sensitive Hashing (or CSH), starts from the LSH framework. Here, looking at high dimensional points in space, each point v is projected onto a random line. The projection is described by the following formula. ‘a’ is the random line, ‘r’ is the width of the bins that the line is divided into and ‘b’ is a random shift of the bins along the line. Each point v is assigned the number of the bin that it is projected into. This process is repeated, projecting the point on several random lines and the final code ‘g’ is the concatenation of the bin numbers that a point is assigned under each projection. Once we have these codes, each point can be hashed into a hash table and it can be shown that close points in space will be hashed together with high probability. In our case, these points are the image’s patches and the main modification we make is in replacing the random lines ‘a_i’ by the leading vectors of the 2-dimensional Walsh-Hadamard basis. These projections can be computed rapidly at only 2 operations per pixel and produce a very effective dimension reduction of the space of patches. The notion of locality sensitive hashing (LSH) was first introduced by Indyk and Motwani [13]. Given a set of points in a metric space, LSH function families have the property that points that are close to each other have a higher probability of colliding (under random members of the family) compared to points that are far apart. The first usage of LSH for nearest neighbor search in high dimen- sions worked in high dimensional binary Hamming space [11]. Our algorithm will follow the general lines of an LSH- based approximate nearest neighbor search scheme later proposed by Datar et al. [8]. In the rest of this section we outline their algorithm. At the base of the algorithm is a family H of LSH func- tions and the ANN search algorithm consists of two stages: indexing and search (query). In the indexing stage, prim- itive hash functions from H are used to create an index in which similar points map into the same hash bins with high probability. M such primitive hash functions are con- catenated to create a code which amplifies the gap between the collision probability of far away points and the colli- sion probability of nearby points. Such a code creates a single hash table, by evaluating it on all data-set points. In the search stage, a query point is hashed into a table bin, from which the nearest of residing data-set points is cho- sen. In order to decrease the probability of falling into an empty bin (with no data-set points), multiple (L) random codes are used to create L hash tables, which are searched sequentially at search stage. Datar et al.[8] show that the above scheme results in significantly improved efficiency compared to previous methods in the case of L2 distances, which are the ones of interest in our case. The LSH for Points Project each point v on a random segmented line 𝑙 Line 𝑙 Bin number 1 2 3 4 5 6 ℎ 𝑙 𝑣 Approximate nearest neighbor: towards removing the curse of dimensionality. Indyk et al., STOC 1998

The LSH for Points Hash function LSH table 𝑔 𝑣 = ℎ 1 𝑣 ,..., ℎ 𝑛 𝑣 Coherency Sensitive Hashing (or CSH), starts from the LSH framework. Here, looking at high dimensional points in space, each point v is projected onto a random line. The projection is described by the following formula. ‘a’ is the random line, ‘r’ is the width of the bins that the line is divided into and ‘b’ is a random shift of the bins along the line. Each point v is assigned the number of the bin that it is projected into. This process is repeated, projecting the point on several random lines and the final code ‘g’ is the concatenation of the bin numbers that a point is assigned under each projection. Once we have these codes, each point can be hashed into a hash table and it can be shown that close points in space will be hashed together with high probability. In our case, these points are the image’s patches and the main modification we make is in replacing the random lines ‘a_i’ by the leading vectors of the 2-dimensional Walsh-Hadamard basis. These projections can be computed rapidly at only 2 operations per pixel and produce a very effective dimension reduction of the space of patches. The notion of locality sensitive hashing (LSH) was first introduced by Indyk and Motwani [13]. Given a set of points in a metric space, LSH function families have the property that points that are close to each other have a higher probability of colliding (under random members of the family) compared to points that are far apart. The first usage of LSH for nearest neighbor search in high dimen- sions worked in high dimensional binary Hamming space [11]. Our algorithm will follow the general lines of an LSH- based approximate nearest neighbor search scheme later proposed by Datar et al. [8]. In the rest of this section we outline their algorithm. At the base of the algorithm is a family H of LSH func- tions and the ANN search algorithm consists of two stages: indexing and search (query). In the indexing stage, prim- itive hash functions from H are used to create an index in which similar points map into the same hash bins with high probability. M such primitive hash functions are con- catenated to create a code which amplifies the gap between the collision probability of far away points and the colli- sion probability of nearby points. Such a code creates a single hash table, by evaluating it on all data-set points. In the search stage, a query point is hashed into a table bin, from which the nearest of residing data-set points is cho- sen. In order to decrease the probability of falling into an empty bin (with no data-set points), multiple (L) random codes are used to create L hash tables, which are searched sequentially at search stage. Datar et al.[8] show that the above scheme results in significantly improved efficiency compared to previous methods in the case of L2 distances, which are the ones of interest in our case. The LSH for Points Hash function 𝑔 𝑣 = ℎ 1 𝑣 ,..., ℎ 𝑛 𝑣 LSH table 2 3 4 5 6 𝑔 𝑣 6 7 8 9

The LSH for Subspaces (GH) Coherency Sensitive Hashing (or CSH), starts from the LSH framework. Here, looking at high dimensional points in space, each point v is projected onto a random line. The projection is described by the following formula. ‘a’ is the random line, ‘r’ is the width of the bins that the line is divided into and ‘b’ is a random shift of the bins along the line. Each point v is assigned the number of the bin that it is projected into. This process is repeated, projecting the point on several random lines and the final code ‘g’ is the concatenation of the bin numbers that a point is assigned under each projection. Once we have these codes, each point can be hashed into a hash table and it can be shown that close points in space will be hashed together with high probability. In our case, these points are the image’s patches and the main modification we make is in replacing the random lines ‘a_i’ by the leading vectors of the 2-dimensional Walsh-Hadamard basis. These projections can be computed rapidly at only 2 operations per pixel and produce a very effective dimension reduction of the space of patches. The notion of locality sensitive hashing (LSH) was first introduced by Indyk and Motwani [13]. Given a set of points in a metric space, LSH function families have the property that points that are close to each other have a higher probability of colliding (under random members of the family) compared to points that are far apart. The first usage of LSH for nearest neighbor search in high dimen- sions worked in high dimensional binary Hamming space [11]. Our algorithm will follow the general lines of an LSH- based approximate nearest neighbor search scheme later proposed by Datar et al. [8]. In the rest of this section we outline their algorithm. At the base of the algorithm is a family H of LSH func- tions and the ANN search algorithm consists of two stages: indexing and search (query). In the indexing stage, prim- itive hash functions from H are used to create an index in which similar points map into the same hash bins with high probability. M such primitive hash functions are con- catenated to create a code which amplifies the gap between the collision probability of far away points and the colli- sion probability of nearby points. Such a code creates a single hash table, by evaluating it on all data-set points. In the search stage, a query point is hashed into a table bin, from which the nearest of residing data-set points is cho- sen. In order to decrease the probability of falling into an empty bin (with no data-set points), multiple (L) random codes are used to create L hash tables, which are searched sequentially at search stage. Datar et al.[8] show that the above scheme results in significantly improved efficiency compared to previous methods in the case of L2 distances, which are the ones of interest in our case. The LSH for Subspaces (GH) 𝑠 𝑠−1 Compute the elevation angle between subspace and a random line θ * * * 𝑉 2 1 𝑙 𝑙 𝑙 Bin number of θ θ 𝑉 ℎ 𝑙 𝑉 Divide by 𝑠 𝑠 1 2 * * * * Range 0, π 2 π 2 θ

The LSH for Subspaces (GH) Coherency Sensitive Hashing (or CSH), starts from the LSH framework. Here, looking at high dimensional points in space, each point v is projected onto a random line. The projection is described by the following formula. ‘a’ is the random line, ‘r’ is the width of the bins that the line is divided into and ‘b’ is a random shift of the bins along the line. Each point v is assigned the number of the bin that it is projected into. This process is repeated, projecting the point on several random lines and the final code ‘g’ is the concatenation of the bin numbers that a point is assigned under each projection. Once we have these codes, each point can be hashed into a hash table and it can be shown that close points in space will be hashed together with high probability. In our case, these points are the image’s patches and the main modification we make is in replacing the random lines ‘a_i’ by the leading vectors of the 2-dimensional Walsh-Hadamard basis. These projections can be computed rapidly at only 2 operations per pixel and produce a very effective dimension reduction of the space of patches. The notion of locality sensitive hashing (LSH) was first introduced by Indyk and Motwani [13]. Given a set of points in a metric space, LSH function families have the property that points that are close to each other have a higher probability of colliding (under random members of the family) compared to points that are far apart. The first usage of LSH for nearest neighbor search in high dimen- sions worked in high dimensional binary Hamming space [11]. Our algorithm will follow the general lines of an LSH- based approximate nearest neighbor search scheme later proposed by Datar et al. [8]. In the rest of this section we outline their algorithm. At the base of the algorithm is a family H of LSH func- tions and the ANN search algorithm consists of two stages: indexing and search (query). In the indexing stage, prim- itive hash functions from H are used to create an index in which similar points map into the same hash bins with high probability. M such primitive hash functions are con- catenated to create a code which amplifies the gap between the collision probability of far away points and the colli- sion probability of nearby points. Such a code creates a single hash table, by evaluating it on all data-set points. In the search stage, a query point is hashed into a table bin, from which the nearest of residing data-set points is cho- sen. In order to decrease the probability of falling into an empty bin (with no data-set points), multiple (L) random codes are used to create L hash tables, which are searched sequentially at search stage. Datar et al.[8] show that the above scheme results in significantly improved efficiency compared to previous methods in the case of L2 distances, which are the ones of interest in our case. The LSH for Subspaces (GH) Intuition π 2 ℓ θ ≈ θ Random line 1 2 θ θ 1 2 θ 1 θ ≈ θ 1 3 θ 2 θ ≈ θ 2 3 θ 3 θ 3 1 2 * * * s-1 s * further details in the paper

The GH Algorithm Summary Let’s now look at the outline of the algorithm. We start with a preprocessing stage, projecting all patches on the WH vectors. Then we iterate the following: First hash the patches into the table Then, for each patch in ‘A’, create the candidate patches in ‘B’ and pick the best ----------------------------------------------------------------------- In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate The GH Algorithm Summary Preprocessing phase: Pick Random lines compute the bin numbers for each subspace in dataset Searching phase: compute the bin numbers of the query Return one of the subspaces having same hash value

Sublinearity guarantee of GH Let’s now look at the outline of the algorithm. We start with a preprocessing stage, projecting all patches on the WH vectors. Then we iterate the following: First hash the patches into the table Then, for each patch in ‘A’, create the candidate patches in ‘B’ and pick the best ----------------------------------------------------------------------- In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate Sublinearity guarantee of GH Theorem If (i) or (ii) , then 𝑑 1 = 𝑑 2 =𝑑,𝑐> 𝑑 , 𝑑 2 =1 ρ 𝐷, 𝑑 1 , 𝑑 2 ,𝑟,𝑐,θ <1. 𝑑 1 : dataset dimension 𝑑 2 : query dimension 𝑐: approximation factor

Extreme cases of GH Case I (dimensions of Q and X equal one) Let’s now look at the outline of the algorithm. We start with a preprocessing stage, projecting all patches on the WH vectors. Then we iterate the following: First hash the patches into the table Then, for each patch in ‘A’, create the candidate patches in ‘B’ and pick the best ----------------------------------------------------------------------- In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate Extreme cases of GH Case I (dimensions of Q and X equal one) If and is fixed, then 𝑑 1 = 𝑑 2 =1,𝑟= α 𝐷 α>0 ,𝑐𝑟=𝑂 1 , θ lim 𝐷→∞ ρ 𝐷, 𝑑 1 , 𝑑 2 ,𝑟,𝑐,θ ≤ 1 1+ 𝑒 α 2 2 Case II (query Q is in the dataset X) With proper chosen and , for any 𝑟 𝑐 ϵ>0 ρ 𝐷,𝑑,𝑑,𝑟,𝑐,θ <ϵ.

Monte-Carlo Integration of Let’s now look at the outline of the algorithm. We start with a preprocessing stage, projecting all patches on the WH vectors. Then we iterate the following: First hash the patches into the table Then, for each patch in ‘A’, create the candidate patches in ‘B’ and pick the best ----------------------------------------------------------------------- In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate Monte-Carlo Integration of ρ 𝐷=10 𝑑=1 θ= π 10, π 7, π 4 ρ ρ≈0.85 𝑐=1.1 𝑟

Monte-Carlo Integration of Let’s now look at the outline of the algorithm. We start with a preprocessing stage, projecting all patches on the WH vectors. Then we iterate the following: First hash the patches into the table Then, for each patch in ‘A’, create the candidate patches in ‘B’ and pick the best ----------------------------------------------------------------------- In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate Monte-Carlo Integration of ρ 𝐷=10 𝑑=1 ρ≈0.62 θ= π 10, π 7, π 4 ρ 𝑐=1.5 𝑟

Experiments – Face Recognition Another experiment we ran is what we call ‘image reconstruction’. Given a pair of images, we compute the NNF between them and then, try to reconstruct image ‘A’, given only image ‘B’ and the NN mapping. The reconstruction was done by taking each pixel to be the simple average of all pixels it is mapped to by its containing patches. This kind of reconstruction is a basic building block many patch based algorithms. The advantage of having the original image in the experiment is that you can compare it to the reconstructed image, both visually and quantitatively. ------------------------------------------------------------- In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate Experiments – Face Recognition Multi-Pie dataset Images for 239 persons 80 frontview images for each person Cropped image of size 23*19

Experiments – Face Recognition Another experiment we ran is what we call ‘image reconstruction’. Given a pair of images, we compute the NNF between them and then, try to reconstruct image ‘A’, given only image ‘B’ and the NN mapping. The reconstruction was done by taking each pixel to be the simple average of all pixels it is mapped to by its containing patches. This kind of reconstruction is a basic building block many patch based algorithms. The advantage of having the original image in the experiment is that you can compare it to the reconstructed image, both visually and quantitatively. ------------------------------------------------------------- In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate Experiments – Face Recognition Experiment procedure 80 images per person Dataset: 36 images to construct a 5-dim subspace 239 subspaces in dataset Query: images to construct a -dim subspace 239 queries for each ranges from 1 to 10 𝑑 𝑑 𝑑 𝑑

Experiments – Face Recognition Another experiment we ran is what we call ‘image reconstruction’. Given a pair of images, we compute the NNF between them and then, try to reconstruct image ‘A’, given only image ‘B’ and the NN mapping. The reconstruction was done by taking each pixel to be the simple average of all pixels it is mapped to by its containing patches. This kind of reconstruction is a basic building block many patch based algorithms. The advantage of having the original image in the experiment is that you can compare it to the reconstructed image, both visually and quantitatively. ------------------------------------------------------------- In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate Experiments – Face Recognition

Experiments – Image Approximation Another experiment we ran is what we call ‘image reconstruction’. Given a pair of images, we compute the NNF between them and then, try to reconstruct image ‘A’, given only image ‘B’ and the NN mapping. The reconstruction was done by taking each pixel to be the simple average of all pixels it is mapped to by its containing patches. This kind of reconstruction is a basic building block many patch based algorithms. The advantage of having the original image in the experiment is that you can compare it to the reconstructed image, both visually and quantitatively. ------------------------------------------------------------- In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate Experiments – Image Approximation Test Image A Base Image B GH BHZ Reconstructed Image A' ? Original patch Projected patch

Experiments – Image Approximation Most our experiments are on a new data-set we created. It includes 133 pairs of images taken from hi-definition movie trailers. These are quite challenging image pairs, since they typically include both camera and object motion. The data-set is available on-line. ----------------------------------------------------------------------- In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate Experiments – Image Approximation Berkeley Segmentation Dataset 100 gray 481×321 images

Experiments – Image Approximation Another experiment we ran is what we call ‘image reconstruction’. Given a pair of images, we compute the NNF between them and then, try to reconstruct image ‘A’, given only image ‘B’ and the NN mapping. The reconstruction was done by taking each pixel to be the simple average of all pixels it is mapped to by its containing patches. This kind of reconstruction is a basic building block many patch based algorithms. The advantage of having the original image in the experiment is that you can compare it to the reconstructed image, both visually and quantitatively. ------------------------------------------------------------- In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate Experiments – Image Approximation Original

Experiments – Image Approximation Another experiment we ran is what we call ‘image reconstruction’. Given a pair of images, we compute the NNF between them and then, try to reconstruct image ‘A’, given only image ‘B’ and the NN mapping. The reconstruction was done by taking each pixel to be the simple average of all pixels it is mapped to by its containing patches. This kind of reconstruction is a basic building block many patch based algorithms. The advantage of having the original image in the experiment is that you can compare it to the reconstructed image, both visually and quantitatively. ------------------------------------------------------------- In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate Experiments – Image Approximation BHZ

Experiments – Image Approximation Another experiment we ran is what we call ‘image reconstruction’. Given a pair of images, we compute the NNF between them and then, try to reconstruct image ‘A’, given only image ‘B’ and the NN mapping. The reconstruction was done by taking each pixel to be the simple average of all pixels it is mapped to by its containing patches. This kind of reconstruction is a basic building block many patch based algorithms. The advantage of having the original image in the experiment is that you can compare it to the reconstructed image, both visually and quantitatively. ------------------------------------------------------------- In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate Experiments – Image Approximation GH

Approximation-Error SSIM measures similarity between two images SSIM is equal to 1 if two images are identical Red line is GH Blue line is BHZ Image quality assessment: from error visibility to structural similarity. Wang et al’, IEEE, 2004

Run time Green line is Linear Red line is BHZ Blue line is GH

Summary – Grassmannian Hashing In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate A new algorithm for subspace search With “state of art” accuracy and speed Stable running time Complexity analysis in worst case scenario Remarks Currently doesn't work for affine subspaces Can choose good hash table parameters using results of Indyk et. al. Can it be applied to feature space?

Thank You In the past, it was customary to compute ANNF with traditional approximate nearest neighbor tools such as Lo- cality Sensitive Hashing (LSH) [13] or KD-trees [1, 16]. These tools perform well in terms of accuracy but are not as fast as one would hope. Recently, a novel method, termed PatchMatch [4], proved to outperform those methods by up to two orders of magnitude, making applications that rely on ANNF run at interactive rate Thank You