Ryan O'Donnell (CMU) Yi Wu (CMU, IBM) Yuan Zhou (CMU)

Locality Sensitive Hashing [Indyk-Motwani '98] objectssketchesh : H : family of hash functions h s.t. “similar” objects collide w/ high prob. “dissimilar” objects collide w/ low prob.

Abbreviated history

Min-wise hash functions [Broder '98] A B word 1?word 2?word 3?word d? Jaccard similarity: Invented simple H s.t. Pr [h(A) = h(B)] =

Indyk-Motwani '98 Defined LSH. Invented very simple H good for {0, 1} d under Hamming distance. Showed good LSH implies good nearest-neighbor-search data structs.

Charikar '02, STOC Proposed alternate H (“simhash”) for Jaccard similarity. Patented by. GoogleGoogle GoogleGoogle

Many papers about LSH

PracticeTheory Free code base [AI’04] Sequence comparison in bioinformatics Association-rule finding in data mining Collaborative filtering Clustering nouns by meaning in NLP Pose estimation in vision [Tenesawa–Tanaka ’07] [Broder ’97] [Indyk–Motwani ’98] [Gionis–Indyk–Motwani ’98] [Charikar ’02] [Datar–Immorlica– –Indyk–Mirrokni ’04] [Motwani–Naor–Panigrahi ’06] [Andoni–Indyk ’06] [Neylon ’10] [Andoni–Indyk ’08, CACM]

Given: (X, dist), r > 0, c > 1 distance space“radius”“approx factor” Goal: Family H of functions X → S ( S can be any finite set) s.t. ∀ x, y ∈ X, ≥ p ≤ q ≥ q.5 ≥ q.25 ≥ q.1 ≥ q ρ ≤ r≥ cr

Theorem [IM’98, GIM’98] Given LSH family for (X, dist), can solve “(r,cr)-near-neighbor search” for n points with data structure of size: O(n 1+ρ ) query time: Õ(n ρ ) hash fcn evals.

Example X = {0,1} d, dist = Hamming r = εd, c = dist ≤ εd or ≥5εd H = { h 1, h 2, …, h d }, h i (x) = x i [IM’98] “output a random coord.”

Analysis = q = q ρ (1 − 5 ε ) 1/5 ≈ 1 − ε. ∴ ρ ≈ 1/5 (1 − 5 ε ) 1/5 ≤ 1 − ε. ∴ ρ ≤ 1/5 In general, achieves ρ ≤ 1/c, ∀ c ( ∀ r).

Optimal upper bound ( {0, 1} d, Ham ), r > 0, c > 1. S ≝ {0, 1} d ∪ { ✔ }, H ≝ {h ab : dist(a,b) ≤ r} h ab (x) = ✔ if x = a or x = b x otherwise = 0 positive = > 0.5 > 0.1 > 0.01 > ≤ r≥ cr

The End. Any questions?

Wait, what? Theorem [IM’98, GIM’98] Given LSH family for (X, dist), can solve “(r,cr)-near-neighbor search” for n points with data structure of size: O(n 1+ρ ) query time: Õ(n ρ ) hash fcn evals.

Wait, what? Theorem [IM’98, GIM’98] Given LSH family for (X, dist), can solve “(r,cr)-near-neighbor search” for n points with data structure of size: O(n 1+ρ ) query time: Õ(n ρ ) hash fcn evals. q ≥ n -o(1) ("not tiny")

More results For R d with ℓ p -distance: when p = 1, 0 < p < 1, p = 2 [IM’98][DIIM’04][AI’06] For Jaccard similarity: ρ ≤ 1/c [Bro’98] For {0,1} d with Hamming distance: −o d (1) (assuming q ≥ 2 −o(d) ) [MNP’06] immediately for ℓ p -distance

Our Theorem For {0,1} d with Hamming distance: −o d (1) (assuming q ≥ 2 −o(d) ) immediately for ℓ p -distance ( ∃ r s.t.) Proof also yields ρ ≥ 1/c for Jaccard.

Proof

Noise-stability is log-convex. :

Proof A definition, and two lemmas. :

Definition: Noise stability at e -т Fix any arbitrary function h : {0,1} d → S. Pick x ∈ {0,1} d at random: x =h(x) = s Flip each bit w.p. (1-e -2т )/2 independenttly y = h(y) = s’ def:

Lemma 1: Lemma 2: For x y, when τ ≪ 1. K h (τ) is a log-convex function of τ. (for any h) τ dist(x, y) = o(d) w.v.h.p. ≈ Proof: Chernoff bound and Taylor expansion. Proof uses Fourier analysis of Boolean functions. 0 τ log K h (τ)

Fourier transformation Theorem. f : {0, 1} d -> R can be uniquely written as where Proof. is an orthonormal basis of {f : {0, 1} d -> R}. Basis fcns. Fourier coef.

Lemma 2:K h (τ) is a log-convex function of τ. Proof: Let h i (x) = 1 h(x)=i.

=

Lemma 2:K h (τ) is a log-convex function of τ. Proof: Let h i (x) = 1 h(x)=i. log-convex fcns. non-neg comb. of

Lemma 1: Lemma 2: For x y, when τ ≪ 1. K h (τ) is a log-convex function of τ. (for any h) τ ≈ Theorem: LSH for {0,1} d requires. 0 τ log K h (τ) dist(x, y) = o(d) w.v.h.p.

Proof: Say H is an LSH family for {0,1} d with params (εd + o(d), cεd - o(d), q ρ, q). r (c − o(1) ) r def: ( Non-neg. lin. comb. of log-convex fcns. ∴ K H (τ) is also log-convex. ) w.v.h.p., dist(x,y) ≈ (1 - e -т )d ≈ тd ∴ K H (ε) ≳ q ρ K H (cε) ≲ q in truth, q+2 −Θ(d) ; we assume q not tiny

∴ K H (ε) ≳ K H (cε) ≲ ∴ K H (0) = ln 1 qρqρ q 0 ρ ln q ln q K H (τ) is log-convex 0 τ ln K H (τ) cεcε ln q ε ∴ 1 c ρ ln q ≤ ln q 1 c

The End. Any questions?