Similarity Join Wu Yang

Main work MS--A Primitive Operator for Similarity Joins in Data Cleaning ICDE 2006 Google--Scaling Up All Pairs Similarity Search www2007 University of New South Wales & NICTA Australia – Chuan Xiao ， Wei Wang ， Xuemin Lin PPJoin ： Efficient Similarity Joins for Near Duplicate Detection ， WWW2008 ， EdJoin:An Efficient Algorithm for Similarity Joins With Edit Distance Constraints ， VLDB 2008 Approximate Entity Extraction with Edit Distance Constraints. SIGMOD 2009 Top-k Set Similarity Joins. ICDE

3 Outline Motivation Algorithms Experiments Thinking

Near Duplicate Data On one end, a winded Pete Sampras tried to summon enough energy to give the New York fans another memorable win to talk about it on the subway ride home. On the other side, Roger Federer wore a sly grin like he knew age was about to catch up to the former world No. 1 - the man who owns the record of 14 Grand Slams he wants. 03/11/2008 | 11:28 AM By JAY COHEN, AP Sports Writer Mar 11, 4:23 am EDT

App: deduplication / App: Identify spams Plagiarism Copyright protection Replicate Web collections

App: data integration / record linkage Efficient Similarity Joins for Near Duplicate Detection

7 Applications For Web search engines: Perform focused crawling Increase the quality and diversity of query results Identify spams. For Web mining: Perform document clustering Find replicate Web collections Detect plagiarism SPAM TEMPLATE Sir/Madam, We happily announce to you the draw of the EURO MILLIONS SPANISH LOTTERY INTERNATIONAL WINNINGS PROGRAM PROMOTIONS held on the 27TH MARCH 2008 in SPAIN. Your company or your personal address attached to ticket number with serial main number drew lucky star winning numbers which consequently won in the 2ND category, you have therefore been approved for a lump sum pay out of Euros. (NINE HUNDRED AND SIXTY THOUSAND EUROS). CONGRATULATIONS!!! Sincerely yours, Q. What are the advantages of RAID5 over RAID4? A. 1. Several write requests could be processed in parallel, since the bottleneck of a unique check disk has been eliminated. 2. Read requests have a higher level of parallelism. Since the data is distributed over all disks, read requests involve all disks, whereas in systems with a dedicated check disk the check disk never participates in read. Q. What are the advantages of RAID5 over RAID4? A. 1. Several write requests could be processed in parallel, since the bottleneck of a single check disk has been eliminated. 2. Read requests have a higher level of parallelism on RAID5. Since the data is distributed over all disks, read requests involve all disks, whereas in systems with a check disk the check disk never participates in read.

Algorithms Data set Similarity function Algorithms

Data set dblp.raw texas.raw trec.raw uniref.500K.raw

10 Similarity Function Common similarity functions: Jaccard: Cosine: Overlap: Jaccard can be equivalently converted to Overlap x = {A,B,C,D,E} y = {B,C,D,E,F} 4/6 = /5 = 0.8 4

Similarity Function Hamming distance =|(x-y)U(y-x)| Edit distance

Algorithms - classication

Algorithms object Similarity between sets Binary similarity functions Contains, intersects Numerical similarity functions Overlap, Jaccard, dice, cosine Similarity between strings Treat strings as sets Jaccard (on q-grams), edit distance

algorithms SSJoin All-pairs PPJoin, PPJoin+ Top-k Set Similarity Joins

SSJoin Based on sets Why string to set? Cited from Efficient Exact Set-Similarity Joins --MS Generalizes to many string similarity funcs Powerful primitive Sets ≈ Relations Leverage relational data processing

SSjoin find {(r, s) | r ∈ R, s ∈ S, overlap(r, s) ≥ t} A fundamental “operator” can handle other similarity functions (Jaccard, cosine, Hamming, dice, edit distance, …) via transformation Efficient Similarity Joins for Near Duplicate Detection

Prefix Filtering-based similarity join 1. SSJoin[Chaudhuri et al, ICDE06] Formalize the prefix-filtering principle 2. All pairs [Bayardo et al, WWW07] Use prefix-filtering in an asymmetric way 3. PPJoin+[Xiao et al, WWW08] Employs prefix-filtering, position filtering and suffix Filtering- based Similarity Joins

ALL Pairs

Prefix + Positional Information We use prefix filter (All-Pairs [www07] ) as basic framework Intuition tokens sorted -> rank, or position of tokens within a record estimate tighter upper bounds of overlap between x and y with positional information Contributions index construction index not only tokens, but their positions in the record  ppjoin algorithm candidate generation probe tokens in suffix, compare the positions in the record  ppjoin+ algorithm

Experiments

Experiments

Experiments

Thinking Further optimization on performances 1. Index for similarity functions (e.g., cosine) 2. Better pruning techniques 3. Optimize for the specific similarity/distance function

Thinking 已有方法对于 token 的处理 基于 inverted-list 方法 TF ， IDF IR 中常见加权的方式 w i,j= tf*idf 直觉：既然 token 的权重对于算法的效率有影响，那么有 没有更好的方式处理 token 的排序呢？是否对结果有影响 呢？ 思考：对于 token 排序的过程中，对于某些词，是否可以 屏蔽掉，对于某些词，是否定义其权重。

continue 报告中所介绍的算法，都是基于 SET 的。如图

Related Work Approximate: LSH: A. Gionis, P. Indyk, and R. Motwani. Similarity search in high dimensions via hashing. In VLDB, Shingling: A. Z. Broder. On the resemblence and containment of documents. In SEQS, Exact: Index-based: S. Sarawagi and A. Kirpal. Efficient set joins on similarity predicates. In SIGMOD, Prefix-based: S. Chaudhuri, V. Ganti, and R. Kaushik. A primitive operator for similarity joins in data cleaning. In ICDE, All-Pairs: R. J. Bayardo, Y. Ma, and R. Srikant. Scaling up all pairs similarity search. In WWW, PPjoin,PPjoin+ Chuan Xiao, Wei Wang, Xuemin Lin, Jeffrey Xu Yu. Efficient Similarity Joins for Near Duplicate Detection. WWW 2008 Pigeon-hole principle based: PartEnum: A. Arasu, V. Ganti, and R. Kaushik. Efficient exact set-similarity joins. In VLDB, 2006.

References [SEQS97] A. Z. Broder. On the resemblance and containment of documents. In SEQS [MIR] R. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrival. Addison Wesley, 1 st edition, May [VLDB99] LSH: A. Gionis, P. Indyk, and R. Motwani. Similarity search in high dimensions via hashing. In VLDB, [SIGMOD04] S. Sarawagi and A. Kirpal. Efficient set joins on similarity predicates. In SIGMOD, [ICDE06] S. Chaudhuri, V. Ganti, and R. Kaushik. A primitive operator for similarity joins in data cleaning. In ICDE, [VLDB06] PartEnum: A. Arasu, V. Ganti, and R. Kaushik. Efficient exact set-similarity joins. In VLDB, [WWW07] All-Pairs: R. J. Bayardo, Y. Ma, and R. Srikant. Scaling up all pairs similarity search. In WWW, [WWW 2008] Efficient Similarity Joins for Near Duplicate Detection Chuan Xiao, Wei Wang, Xuemin Lin, Jeffrey Xu Yu.. WWW 2008 [VLDB 2008]. Ed-Join: An Efficient Algorithm for Similarity Joins with Edit Distance Constraints. VLDB [ICDE 2009]. Top-k Set Similarity Joins Chuan Xiao, Wei Wang, Xuemin Lin, Haichuan Shang