1. Efficient Approximation of Edit Distance Robert Krauthgamer, Weizmann Institute of Science SPIRE 2013 TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A A A AAA A AA

2. Efficient Approximation of Edit Distance Generic Search Engine Given two strings x  n, y  m : ed(x,y) = minimum number of character operations (insertion/deletion/substitution) that transform x to y. Edit Distance (Levenshtein distance) Applications: Computational Biology Text processing Web search Examples: ed( banana, ananas ) = 2 ed(00000, 1111) = 5 For simplicity: m = n. 2

3. Efficient Approximation of Edit Distance Dynamic Programming Algorithm Compute ed(x,y) for input x,y   n  O(n 2 ) time by dynamic programming [WF’74]  O(n 2 /log 2 n) time when |  | · O(1) [MP’80] banana a n a n a s 2 56 1 1 1 1 1 1 2 22 22 22 2 2 2 2 2 3 33 3 3 33 34 4 4 4 4 5 5 D(i,j)= min D(i-1, j-1), if x[i]=y[j] D(i, j-1) + 1 D(i-1, j) + 1 D(i,j) = ed( x[1:i], y[1:j] ) Faster algorithms? 3

4. Focus of This Talk Approximating edit distance  Multiplicatively:ed(x,y) · output · A ¢ ed(x,y)  Decision version: ed(x,y) · r or ed(x,y) > A ¢ r Different computational models  RAM, Sampling and query complexity, Sketching, (Streaming)  Interactions (is it surprising?), Techniques Variants of the problem Efficient Approximation of Edit Distance 4

5. RAM Model: Sampling Idea 1: quickly estimate ed(x,y) by sampling a few positions Intuition:  If ed(x,y) is small, then “many” large blocks should “match”  “Test” this by reading few (randomly chosen) blocks  Apply this idea recursively (inside blocks) Theorem [Batu-Ergun-Kilian-Magen-Raskhodinkova-Rubinfeld-Sami’03] : Factor n c “weak” approximation in sublinear time. Obstacles:  “Block match” means both “similar pattern” and “similar location”  Argue that if and only if ed(x,y) is small then … Can only distinguish ed(x,y) · n/(8A) from ed(x,y)>n/8. Efficient Approximation of Edit Distance Best approximation in (near) linear time? 5

6. Learn from Past Success Suppose x,y are permutations  Every symbol of  appears exactly once Consider transpositions=block moves (“block edit distance”)  No Insert/delete (unreasonable), no substitution (not needed) Example: bed(0123456789, 0457689123)=2 An easy estimate (based on breakpoints)  Compute S x = {all length 2 substrings of x} = {x[i:i+1] | i=1,…,n-1} Lemma: bed(x,y) · ½ |S x ΔS y | · 3 bed(x,y) Proof idea: Fix x (wlog identity), let y=  Each block move “creates” at most 3 new breakpoints  Break y at breakpoints, and move (rearrange) the blocks to get x Can compute |S x ΔS y | in linear time!!  Best approximation known in poly-time is 1.375 [Elias-Hartman’06] Efficient Approximation of Edit Distance A B C D Open: better approximation in linear-time? 6

7. Reduction to Hamming Distance |S x ΔS y | = Hamming distance between their characteristic vectors  In fact, each vector has |  | 2 =n 2 coordinates, but only n-1 are non-zero We thus obtain f : Permutations   { 0,1} n 2 such that 8 x,y,bed(x,y) · ½ ||f(x)-f(y)|| 1 · 3 bed(x,y). Such a reduction from one metric space (BED on permutations) to another (L 1 ) is called an embedding. This one has distortion D=3.  Known lower bound: distortion into L 1 must be ¸ 4/3 [Polak-K.’12] Efficient Approximation of Edit Distance More benefits of “good” embeddings? A sweet spot of fruitful interaction between Math/Geometry (“comparing” metric spaces using embeddings) and CS/Algorithms (solving new problems by “reducing” to old ones) 7

8. Sketching Model Idea: “summarize” each string separately, then estimate ed(x,y) only from the short sketches s(x),s(y). Possible at all?? YES for Hamming distance, and even L 1 /L 2 [Indyk-Motwani’98, Kushilevitz-Ostrosvky-Rabani’00]  Approximation factor A=1+ε using sketch size O(ε -2 ) bits  It’s essentially a “dimension reduction” [Johnson-Lindenstrauss’86]  Achieved by projection on (inner product with) random direction in space Consequently, YES also for block edit distance on permutations: Applies whenever there is an embedding into L 1 !! Efficient Approximation of Edit Distance BED on perm. HammingO(ε -2 ) bits sketch f s distort. D=3 approx. 1+ε 8

9. Applications of Sketching Input: large database M, with |M| strings of length n each. Output: all pairwise distances or closest pair (BED on perm)  Naively: in time O(|M| 2 n)  Sketching [3+ε approx., decision version]: sketch each string, then estimate all pairs in time O(|M|n + |M| 2 /ε 2 ) Practical viewpoint: filteration, i.e., fast pruning of “bad” pairs Works similarly for Nearest Neighbor Search (NNS):  Reduce NNS for permutations under BED, to NNS for Hamming (L 1 ) Efficient Approximation of Edit Distance Q1. More embeddings? Q2. Sketching directly? Q3. Lower bounds? 9

10. Efficient Approximation of Edit Distance Embedding ED on Permutations Theorem [Charikar-K.’06]: Edit distance on permutations of length n embeds into L 1 with distortion O(log n). Proof.Define where Lemma 1: ||f(P)-f(Q)|| 1 ≤ O(log n) ed(P,Q)‏  Suppose Q is obtained from P by moving one symbol, say ‘s’  General case then follows by applying triangle inequality on P,P’,P’’,…,Q  Total contribution of coordinates s 2 {a,b} is 2  k (1/k) ≤ O(log n)‏ other coordinates is  k k(1/k – 1/(k+1)) ≤ O(log n)‏ Intuition: sign(f a,b (P)) is indicator for “a appears before b” in P Thus, |f a,b (P)-f a,b (Q)| “measures” if {a,b} is an inversion in P vs. Q 10

11. Efficient Approximation of Edit Distance Embedding ED on Permutations (2) Recall where Lemma 1: ||f(P)-f(Q)|| 1 ≤ O(log n) ed(P,Q)‏ Lemma 2: ||f(P)-f(Q)|| 1 ¸ ½ ed(P,Q)  Assume wlog that P=identity  Edit Q into an increasing sequence (thus into P) using quicksort: Choose a random pivot, Delete all characters inverted wrt to pivot Repeat recursively on left and right portions  Now argue ||f(P)-f(Q)|| 1 ¸ E [ #quicksort deletions ] ¸ ½ ed(P,Q) QED Surviving subsequence is increasing  ed(P,Q) ≤ 2 #deletions For every inversion (a,b) in Q: Pr[a deleted “by” pivot b] ≤ 1/|Q -1 [a]-Q -1 [b]+1| ≤ 2 |f a,b (P) – f a,b (Q)| 11

12. Embedding Edit Distance Theorem [Ostrovsky-Rabani’05]: Edit distance on all strings (not only permutations) embeds into L 1 with distortion 2 Õ(√log n).  Previously, distortion n c was known [BarYossef-Jayram-K.-Kumar’04, Batu-Ergun-Sahinalp’06]  Clever recursive method to match blocks much more accurately Penalizes both pattern and location errors  Not very fast (quadratic time), but influenced later work on near-linear time algorithms [Andoni-Onak’09, Andoni-Onak-K.’10] Immediate consequences:  NNS algorithms for edit distance  Sketching Efficient Approximation of Edit Distance 12

13. Lower Bounds Theorem [Khot-Naor’05, K.-Rabani’06]: Embedding edit distance into L 1 requires distortion Ω(log n)  Main technique: Fourier analysis [Kahn-Kalai-Linial’88]  L 1 embedding $ sparsest-cuts $ Boolean functions f:{0,1} n  {0,1} Stronger assertion: O(1)-size sketches for edit distance require Ω̃(log n) approximation, even only for permutations [Andoni-K.’06]  Actually tradeoff between approximation and sketch-size  Techniques: communication complexity and Fourier analysis reduce the problem to sketches that are linear functions (of their input x) Efficient Approximation of Edit Distance Q2’. Sketching vs embedding? 13

14. RAM Model: Asymmetric Sampling Idea 1’: Read all of y, and sampled positions of x Motivations:  Better chances to “obtain” information  Which y’s are easier/harder?  Sampling issues: Focus on query complexity bounds (tight?) Adaptive vs non-adaptive queries Queries depend on y?  Use dynamic programming in time O(n 1+ε )? Efficient Approximation of Edit Distance x y 14

15. Efficient Approximation of Edit Distance Asymmetric Sampling Results [Andoni-Onak-K.’10] Problem: Decide ed(x,y) ≥ n/10 vs ed(x,y) ≤ n/(10A) Complexity = #queries into x (unlimited access to y) n 1-ε A Θ(log n) Θ(log 2 n) Θ(log 3 n) Θ(log t n) #queries n 1/2-ε n 1/2 n 1/3 n 1/4 n 1/t-ε n 1/(t+1) Approximation A:(log n) O(1/ε) # Queries:O(n ε ) Ω(n ε/loglog n ) [n 1/(t+1), n 1/t-ε ] O(log t n) Ω(log t n) 15

16. Efficient Approximation of Edit Distance Overview of Upper Bound Theorem 1: Can distinguish ed(x,y) ≥ n/10 vs ed(x,y) ≤ n/(10A) for A=(log n) O(1/ε) approximation with n ε queries into x (for any ε>0). Proof structure: 1. Characterize edit by “tree-distance” T xy Parameter b≥2 (degree) T xy ≈ ed(x,y) up to 6b*log n factor 2. Prune the tree to subsample x x1x1 x2x2 xnxn b sampled positions in x 16

17. Efficient Approximation of Edit Distance Step 1: Tree Distance Partition x into b blocks, recursively, for h=log b n levels x[1:n] x[1:⅓n]x[⅔n:n] … x[1] x[2]x[3] x[⅓n:⅔n] y[1:n] y[u:u+⅓n] x[s:s+ ⅓ n] T i (s,u) = tree-distance between x[s:s+ℓ i ] and y[u:u+ℓ i ] where ℓ i is the block-length at level i 17

18. Efficient Approximation of Edit Distance Tree Distance: Recursive Definition Recall T i (s,u) = tree-distance between x[s:s+ℓ i ] and y[u:u+ℓ i ] Base case: T h (s,u)=Hamming(x[s],y[u]) Output: T xy =T 0 (s=1,u=1) x[s:s+ℓ i ] y[u:u+ℓ i ] r0r0 x y 18

19. Efficient Approximation of Edit Distance Tree Approximates Edit Distance Lemma: T xy ≈ed(x,y) up to 6b*log b n factor. Hierarchical decomposition inspired by earlier approaches [BEKMRRS’03, OR’05]  All had approximation recurrence of the type A(n) = c*A(n/b) + b for c≥2  Solves to A(n) ≥ 2 √log n factor for every choice of b Our characterization has no multiplicative loss (c=1): A(n) = A(n/b) + b  Analysis inspired by algorithms for smoothed instances [Andoni-K.’08] 19

20. Efficient Approximation of Edit Distance Step 2: Compute the Tree Distance For b=2, tree-distance gives O(log n) approximation!  BUT know only how to compute T-distance in Õ(n 2 ) time Instead, for b=( log n) 1/ε, can prune the tree to n O(ε) nodes, and approximate T-distance within factor 1+ε Pruning: subsample (log n) O(1) children out of each node  Works only when ed(x,y) ≥  (n)  Generally, must subsample the tree non-uniformly, using the Precision Sampling Lemma b sampled positions in x 20

21. Efficient Approximation of Edit Distance Key tool: non-uniform sampling Goal:  For unknown a 1, a 2, …a n  [0,1]  Estimate their sum, up to an additive constant error  Using only “weak” estimates ã 1, ã 2, …ã n Sum Estimator Adversary 0. fix distribution U 1. Fix a 1,a 2,…a n (unknown) 2. pick “precisions” u i (our algorithm: u i ~U[0,1] i.i.d.) 3. provide ã 1,ã 2,…ã n s.t. |a i -ã i |<1/u i 4. report S̃=S̃(ã 1,…, u 1,…) with |S̃ – ∑a i ̃| < 1. 21

22. Efficient Approximation of Edit Distance Precision Sampling Goal: estimate ∑a i from {ã i } s.t. |a i -ã i |<1/u i. Precision Sampling Lemma: Can achieve WHP  additive error 1 and multiplicative error 1.5  with expected precision E u_i~U [u i ]=O(log n). Inspired by a technique from [IW’05] for streaming (F k moments)  In fact, PSL gives simple & improved algorithms for F k moments, cascaded (mixed) norms, ℓ p -sampling problems [AKO’11] Also distant relative of Priority Sampling [DLT’07] 22

23. Efficient Approximation of Edit Distance Precision Sampling for Edit Distance Apply Precision Sampling to the tree from the characterization recursively at each node If a node has very weak precision, can trim the entire sub-tree 23

24. Fast Approximation Algorithm Theorem [Andoni-Onak-K.’10]: Can approximate ed(x,y) within factor (log n) O(1/ε) using n ε queries to x and in time n 1+ε (for any ε>0).  Exponential improvement over previous factor 2 Õ(√log n) [Andoni-Onak’09]  Asymmetric sampling approach, implemented faster by data structure tricks  Sampling is non-adaptive, independent of y Efficient Approximation of Edit Distance 24

25. Smoothed Instances Smooth Instance (x,y) constructed by:  Start with arbitrary x*,y* 2 {0,1} n and their optimal alignment A*  Replace each position w/probability p by random bit, but respect A* Theorem [Andoni-K.’08]: Can approximate ed(x,y) within constant factor, in smoothed runtime that is (whp) near-linear n 1+ε.  Some extensions to sublinear time Techniques:  Match blocks of length L=O(1/p ¢ log n) that have edit distance · εL. A known heuristic technique (e.g. PatternHunter) To find block matches quickly, we use naive NNS algorithm  Because of smoothing, blocks are likely to be distinct (and even far), so modulo overlaps between blocks, we “effectively” have permutations Efficient Approximation of Edit Distance Open: Better time n ¢ polylog(n)? Approximation independent of p? 25

26. Variants of Edit Distance Edit distance with block operations  Admits O(log n ¢ log*n) approximation in near-linear time, via embedding into L 1 [Muthukrishnan-Sahnialp’00,Cormode-Muthukrishnan’02]  Open: Distortion lower bounds? Better approximation in polytime? Edit distance between trees (generalizes strings)  Basic operations: insert/delete/relabel vertex  Can be computed in O(n 3 ) time [Demaine-Mozes-Rossman- Weimann’07]  Open: Embedding? Edit distance with “rich” alphabet  Can model shape matching [Klein-Tirthapura-Sharvit-Kimia’00]  Challenge: Cost of basic operation varies with symbols Efficient Approximation of Edit Distance 26

27. Conclusion Having multiple computational models is fruitful  New ideas, techniques, viewpoints, applications  can come full circle  Lower bounds —in certain models — highlight limitations of methods  Explore which instances are easy/hard “Asymmetric algorithms” can work well for symmetric problems Connections to other fields (sampling, embeddings, communication complexity, Fourier analysis) and computational problems (NNS) Had much progress, but still many gaps, and much more to go Efficient Approximation of Edit Distance Thank You! 27