Ryan O’Donnell (CMU, IAS) joint work with Yi Wu (CMU, IBM), Yuan Zhou (CMU)

Locality Sensitive Hashing [Indyk–Motwani ’98] objectssketchesh : H : family of hash functions h s.t. “similar” objects collide w/ high prob. “dissimilar” objects collide w/ low prob.

Abbreviated history

A Broder ’97, Altavista B word 1?word 2?word 3?word d? Jaccard similarity: Invented simple H s.t. Pr [h(A) = h(B)] =

Indyk–Motwani ’98 (cf. Gionis–I–M ’98) Defined LSH. Invented very simple H good for {0, 1} d under Hamming distance. Showed good LSH implies good nearest-neighbor-search data structs.

Charikar ’02, STOC Proposed alternate H (“simhash”) for Jaccard similarity.

Many papers about LSH

PracticeTheory Free code base [AI’04] Sequence comparison in bioinformatics Association-rule finding in data mining Collaborative filtering Clustering nouns by meaning in NLP Pose estimation in vision [Tenesawa–Tanaka ’07] [Broder ’97] [Indyk–Motwani ’98] [Gionis–Indyk–Motwani ’98] [Charikar ’02] [Datar–Immorlica– –Indyk–Mirrokni ’04] [Motwani–Naor–Panigrahi ’06] [Andoni–Indyk ’06] [Neylon ’10] [Andoni–Indyk ’08, CACM]

Given: (X, dist), r > 0, c > 1 distance space“radius”“approx factor” Goal: Family H of functions X → S ( S can be any finite set) s.t. ∀ x, y ∈ X, ≥ p ≤ q ≥ q.5 ≥ q.25 ≥ q.1 ≥ q ρ

Theorem [IM’98, GIM’98] Given LSH family for (X, dist), can solve “(r,cr)-near-neighbor search” for n points with data structure of size: O(n 1+ρ ) query time: Õ(n ρ ) hash fcn evals.

Example X = {0,1} d, dist = Hamming r = d, c = dist ≤ d or ≥ 5d H = { h 1, h 2, …, h d }, h i (x) = x i [IM’98] “output a random coord.” ( S = {0,1})

Analysis = q = q ρ (1 − 5) 1/5 ≈ 1 −. ∴ ρ ≈ (1 − 5) 1/5 ≤ 1 −. ∴ ρ ≤ In general, achieves ρ ≤ ∀ c ( ∀ r).

Optimal upper bound ( {0, 1} d, Ham ), r > 0, c > 1. S ≝ {0, 1} d ∪ { ✔ }, H ≝ {h ab : dist(a,b) ≤ r} h ab (x) = ✔ if x = a or x = b x otherwise 0 positive = > 0.5 > 0.1 > 0.01 >

Wait, what? [IM’98, GIM’98] Theorem: Given LSH family for (X, dist), can solve “(r,cr)-near-neighbor search” for n points with data structure of size: Õ(n 1+ρ ) query time: Õ(n ρ ) hash fcn evals

Wait, what? [IM’98, GIM’98] Theorem: size: Õ(n 1+ρ ) query time: Õ(n ρ ) hash fcn evals

More results For R d with ℓ p -distance: when p = 1, 0 < p < 1, p = 2 [IM’98][DIIM’04][AI’06] For Jaccard similarity: ρ ≤ 1/c For {0,1} d with Hamming distance: [Bro’97] −o d (1) (assuming q ≥ 2 −o(d) ) [MNP’06] immediately for ℓ p -distance

Our Theorem For {0,1} d with Hamming distance: −o d (1) (assuming q ≥ 2 −o(d) ) immediately for ℓ p -distance ( ∃ r s.t.) Proof also yields ρ ≥ 1/c for Jaccard.

Proof:

Noise-stability is log-convex.

Proof: A definition, and two lemmas.

Fix any arbitrary function h : {0,1} d → S. Pick x ∈ {0,1} d at random: x =h(x) = s Continuous-time (lazy) random walk for time τ y = h(y) = s’ def:

Lemma 1: Lemma 2: From which the proof of ρ ≥ 1/c follows easily. For x y, τ when τ ≪ 1. K h (τ) is a log-convex function of τ. (for any h) 0 1 τ

Continuous-Time Random Walk : Repeatedly — waits Exponential(1) seconds, — dings. (Reminder: T ~ Expon(1) means Pr[T > u] = e −u.) In C.T.R.W. on {0,1} d, each coord. gets its own independent alarm clock. When i th clock dings, coord. i is rerandomized.

x = y = time τ Pr[coord. i never updated] =Pr[Exp(1) > τ]= e − τ ∴ Pr[x i ≠ y i ] = ⇒ Lemma 1: dist(x,y) ≈

Lemma 2:K h (τ) is a log-convex function of τ. Remark:True for any reversible C.T.M.C. Recall:For f : {0,1} d → ℝ, Given hash function h : {0,1} d → S, for each s ∈ S, introduce h s : {0,1} d → {0,1}, h s (x) = 1 {h(x)=s}

Proof of Lemma 2: is log-convex. log-convexnon-neg. lin. comb. of

Lemma 1: Lemma 2: Theorem: LSH for {0,1} d requires For x y, τ is a log-convex function of τ.

Proof: Say H is an LSH family for {0,1} d with params. r (c − o(1) ) r def: ( Non-neg. lin. comb. of log-convex fcns. ∴ K H (τ) is also log-convex. ) w.v.h.p., dist(x,y) ≈ ∴ K H () ≳ q ρ K H (c) ≲ q in truth, q+2 −Θ(d) ; we assume q not tiny

∴ K H () ≳ K H (c) ≲ ∴ K H (0) = ln 1 qρqρ q 0 ρ ln q ln q K H (τ) is log-convex 0 τ ln K H (τ) c ln q ∴

Super-tedious, super-straightforward Make Lemma 1 precise. (Chernoff) Make precise. (Taylor) Choose = (c, q, d) very carefully. Theorem: Meaningful iff q ≥ 2 −o(d) ; i.e., not tiny.