Large-scale Similarity Join with Edit-distance Constraints ---BY Yu Haiyang 1/30

Outline  Background  The introduction of Pass-Join-K  Combining Pass-Join-K with Hadoop 2/30

Background  Similarity join: Find all similar pairs from two sets. Data Cleaning. Query Relaxation Spellchecking 3/30

Background  How to define similarity? Jaccard distance( 词袋模型 ) Cosine distance Edit distance 4/30

Background  Edit distance The minimum number of edit operations (insertion, deletion, and substitution) to transform one string to another. BabyBodySubstitution BodBodyInsertion 5/30

Background  How does the edit distance compare with other two? Accuracy: {“abcdefg”,”gfedcba”} Verification time: O(m+n) -> O(mn) 6/30

Background  Find similar pairs We have two string sets,one is {vldb,sigmod,….},the other is {pvldb,icde,…}. Find some candidate pairs, and then verify these pairs. {,,,,,….} Yes No 7/30

Background  So we have to: Finding candidate pairs. There are O(N 2 ) if we do not prune some pairs. verifying these pairs. O(mn) 8/30

Outline  Background  The introduction of Pass-Join-K  Combining Pass-Join-K with Hadoop 9/30

Introduction of Pass-Join-K  Partition-based pruning technique We suppose the threshold tau = 2, K= 1 and we have a pair 10/30

Introduction of Pass-Join-K  Partition-based pruning technique We suppose the threshold tau = 2, K=2and we have a pair 11/30

Introduction of Pass-Join-K  Some obvious pruning techniques Length –based: threshold = 2, Shift-based: abcd cdef 12/30

Introduction of Pass-Join-K  Partition Scheme We have seen that the longer the substrings are, the harder they could be marched. So we break the string into tau+k parts and each part while its length equals length/(tau+k) or length/(tau+k)+1. 13/30

Introduction of Pass-Join-K  Partition Scheme 14/30

Introduction of Pass-Join-K  Substring Selection Here we suppose tau = 3 and k = 1; abcdefghijk a b d e f g h k 15/30

Introduction of Pass-Join-K  Substring Selection Here we suppose tau = 3 and k = 1; abcdefghijk a b d e f g h k 16/30

Introduction of Pass-Join-K  Substring Selection Here we suppose tau = 3 and k = 1; abcdefghijk a b d e f gh k 17/30

Introduction of Pass-Join-K  Substring Selection Here we suppose tau = 3 and k = 1; abcdefghijk abdefghk 18/30

Introduction of Pass-Join-K  Substring Selection Here we suppose tau = 3 and k = 1; abcdefghijk a b d e f g h k 19/30

Introduction of Pass-Join-K  Substring Selection So what we do is to deduce the number of substrings. More pruning techniques, please read our paper: 《 Pass-Join-K 多分段匹配的相 似性连接算法》 20/30

Introduction of Pass-Join-K  Verification DP( Dynamic programming)  D(m,n)=max(D(m,n-1)+1,D(m-1,n)+1,D(m-1,n- 1)+flag) where flag = 1 when s m =r n, s and r are both strings. 21/30

Introduction of Pass-Join-K  Verification Here we suppose tau = 3 and k = 1; abcdefghijk def e f g h k Tau left = 3 Tau right = 3-3=0 22/30

Outline  Background  The introduction of Pass-Join-K  Combining Pass-Join-K with Hadoop 23/30

Combining Pass-Join-K with Hadoop  Big data Big file Large number of files 24/30

Combining Pass-Join-K with Hadoop  Inverted index tree in hadoop (abc, 1, 11,r,IFlag) (def,2,11,r,IFlag) (ghi,3,11,r,IFlag) (jk,4,11,r,IFlag) abcdefghijk rr rr L 11 25/30

Combining Pass-Join-K with Hadoop  Substrings in hadoop Suppose tau = 3, k = 1, and s = “abdefghk”, length(s) = 8. We have to generate some records such as (a,1,5,s,SFlag),(a,2,6,s,SFlag)(a,3,7,s,SFlag),(a b,1,8,s,SFlag),…,(ab,1,11,s,SFlag),… 26/30

Combining Pass-Join-K with Hadoop  Data flows in hadoop 27/30

Combining Pass-Join-K with Hadoop  Big data Big file Large number of files 28/30

/30 Combining Pass-Join-K with Hadoop  [segmentString, segmentNumber, stringLength, FLAG], [DirNumber, ID]

/30 