Approximate Nearest Subspace Search with Applications to Pattern Recognition Ronen Basri, Tal Hassner, Lihi Zelnik-Manor presented by Andrew Guillory and Ian Simon

The Problem Given n linear subspaces S i :

The Problem Given n linear subspaces S i : And a query point q:

The Problem Given n linear subspaces S i : And a query point q: Find the subspace S i that minimizes dist(S i,q).

Why? object appearance variation = subspace – fast queries on object database

Why? object appearance variation = subspace – fast queries on object database Other reasons?

Approach Solve by reduction to nearest neighbor. – point-to-point distances

Approach Solve by reduction to nearest neighbor. – point-to-point distances not actual reduction

Approach Solve by reduction to nearest neighbor. – point-to-point distances In higher-dimensional space. not actual reduction

Point-Subspace Distance Use squared distance.

Point-Subspace Distance Use squared distance.

Point-Subspace Distance Use squared distance. Squared point-subspace distance can be represented as a dot product.

The Reduction Let: Remember:

The Reduction Let: Then: Remember:

The Reduction

constant over query

The Reduction ?constant over query

The Reduction Z T Z = I ?constant over query

The Reduction Z T Z = I Z is d-by-(d-k), columns orthonormal. ?constant over query

The Reduction Z T Z = I Z is d-by-(d-k), columns orthonormal. ?constant over query

The Reduction For query point q:

The Reduction For query point q: Can we decrease the additive constant?

Observation 1 All data points lie on a hyperplane.

Observation 1 All data points lie on a hyperplane. Let: Now the hyperplane contains the origin.

Observation 2 After hyperplane projection: All data points lie on a hypersphere.

Observation 2 After hyperplane projection: All data points lie on a hypersphere. Let: Now the query point lies on the hypersphere.

Observation 2 After hyperplane projection: All data points lie on a hypersphere. Let: Now the query point lies on the hypersphere.

Reduction Geometry What is happening?

Reduction Geometry What is happening?

Finally Additive constant depends only on dimension of points and subspaces. This applies to linear subspaces, all of the same dimension.

Extensions subspaces of different dimension – lines and planes, e.g. – Not all data points have the same norm. Add extra dimension to fix this.

Extensions subspaces of different dimension – lines and planes, e.g. – Not all data points have the same norm. Add extra dimension to fix this. affine subspaces – Again, not all data points have the same norm.

Approximate Nearest Neighbor Search Find point x with distance d(x, q) <= (1 + ε) min i d(x i,q) Tree based approaches: KD-trees, metric / ball trees, cover trees Locality sensitive hashing This paper uses multiple KD-Trees with (different) random projections

KD-Trees Decompose space into axis aligned rectangles Image from Dan Pelleg

Random Projections Multiply data with a random matrix X with X(i,j) drawn from N(0,1) Several different justifications – Johnson-Lindenstrauss (data set that is small compared to dimensionality) – Compressed Sensing (data set that is sparse in some linear basis) – RP-Trees (data set that has small doubling dimension)

Results Two goals – show their method is fast – show nearest subspace is useful Four experiments – Synthetic Experiments – Image Approximation – Yale Faces – Yale Patches

Image Reconstruction

Yale Faces

Questions / Issues Should random projections be applied before or after the reduction? Why does the effective distance error go down with the ambient dimensionality? The reduction tends to make query points far away from the points in the database. Are there better approximate nearest neighbor algorithms in this case?