Efficient Approximate Search on String Collections Part I

Efficient Approximate Search on String Collections Part I
Marios Hadjieleftheriou Chen Li

DBLP Author Search

Try their names (good luck!)
Case Western AT&T--Research UCSD Yannis Papakonstantinou Meral Ozsoyoglu Marios Hadjieleftheriou

Better system?

People Search at UC Irvine

Web Search Errors in queries Errors in data
Bring query and meaningful results closer together Actual queries gathered by Google 7

Data Cleaning R S informix microsoft … infromix … mcrosoft

Problem Formulation Find strings similar to a given string: dist(Q,D) <= δ Example: find strings similar to “hadjeleftheriou” Performance is important! 10 ms: 100 queries per second (QPS) 5 ms: 200 QPS

Outline Motivation Preliminaries Trie-based approach
Gram-based algorithms Sketch-based algorithms Compression Selectivity estimation Transformations/Synonyms Conclusion Part I Part II

Next… Preliminaries

Similarity Functions Similar to: Examples: a domain-specific function
returns a similarity value between two strings Examples: Edit distance Hamming distance Jaccard similarity Soundex TF/IDF, BM25, DICE See [KSS06] for an excellent survey

Edit Distance A widely used metric to define string similarity
Ed(s1,s2) = minimum # of operations (insertion, deletion, substitution) to change s1 to s2 Example: s1: Tom Hanks s2: Ton Hank ed(s1,s2) = 2 13 13

Gram-based algorithms
Next… Gram-based algorithms List-merging algorithms [LLL08] Variable-length grams (VGRAM) [LWY07,YWL08]

“q-grams” of strings u n i v e r s a l 2-grams

Edit operation’s effect on grams
Fixed length: q u n i v e r s a l k operations could affect k * q grams If ed(s1,s2) <= k, then their # of common grams >= (|s1|- q + 1) – k * q 16

q-gram inverted lists at ch ck ic ri st ta ti tu uc id strings 1 2 3 4
1 2-grams at ch ck ic ri st ta ti tu uc id strings 1 2 3 4 rich stick stich stuck static

Searching using inverted lists
Query: “shtick”, ED(shtick, ?)≤1 sh ht ti ic ck ti ic ck # of common grams >= 3 at ch ck ic ri st ta ti tu uc 4 2 3 1 id strings 1 2 3 4 rich stick stich stuck static 2-grams

Find elements whose occurrences ≥ T
T-occurrence Problem Merge Ascending order Find elements whose occurrences ≥ T

Example T = 4 1 3 5 10 13 10 13 15 5 7 13 13 15 Result: 13

List-Merging Algorithms
HeapMerger MergeOpt [SK04] [LLL08, BK02] ScanCount MergeSkip DivideSkip

Count # of occurrences of each element using a heap
Heap-based Algorithm Push to heap …… Min-heap Count # of occurrences of each element using a heap

MergeOpt Algorithm [SK04]
Binary search Long Lists: T-1 Short Lists

Example of MergeOpt Count threshold T≥ 4 Long Lists: 3 Short Lists: 2
1 3 5 10 13 10 13 15 5 7 13 13 15 Long Lists: 3 Short Lists: 2 Count threshold T≥ 4

ScanCount Count threshold T≥ 4 1 2 3 … 1 1 3 5 10 13 10 13 15 5 7 13
String ids # of occurrences Increment by 1 … 1 1 3 5 10 13 10 13 15 5 7 13 13 15 1 13 4 Result! 14 15 2 Count threshold T≥ 4 25

List-Merging Algorithms
HeapMerger MergeOpt [SK04] [LLL08, BK02] ScanCount MergeSkip DivideSkip

MergeSkip algorithm [BK02, LLL08]
Pop T-1 …… Min-heap Jump Greater or equals T-1

Example of MergeSkip Count threshold T≥ 4 minHeap Jump 1 5 10 13 15 1
7 13 15 13 13 Jump 17 17 15 15 Count threshold T≥ 4

DivideSkip Algorithm [LLL08]
Binary search MergeSkip Long Lists Short Lists

How many lists are treated as long lists?

Length Filtering s: t: Length: 10 By length only! Ed(s,t) ≤ 2

Positional Filtering Ed(s,t) ≤ 2 s a b (ab,1) t a b (ab,12)

A filter tree Combine filters with list-merging algorithms [LLL08]

Variable-length grams (VGRAM) [LWY07,YWL08]
Next… Variable-length grams (VGRAM) [LWY07,YWL08]

2-grams -> 3-grams? sht hti tic ick tic ick
Query: “shtick”, ED(shtick, ?)≤1 sht hti tic ick tic ick # of common grams >= 1 ati ich ick ric sta sti stu tat tic tuc uck 4 2 1 3 id strings 1 2 3 4 rich stick stich stuck static id strings 1 2 3 4 rich stick stich stuck static id strings 1 2 3 4 rich stick stich stuck static 3-grams

Observation 1: dilemma of choosing “q”
Increasing “q” causing: Longer grams  Shorter lists Smaller # of common grams of similar strings 4 2 3 1 2-grams at ch ck ic ri st ta ti tu uc id strings 1 2 3 4 rich stick stich stuck static

Observation 2: skew distributions of gram frequencies
DBLP: 276,699 article titles Popular 5-grams: ation (>114K times), tions, ystem, catio

VGRAM: Main idea Grams with variable lengths (between qmin and qmax)
zebra ze(123) corrasion co(5213), cor(859), corr(171) Advantages Reduce index size  Reducing running time  Adoptable by many algorithms 

Challenges Generating variable-length grams?
Constructing a high-quality gram dictionary? Relationship between string similarity and their gram-set similarity? Adopting VGRAM in existing algorithms?

Challenge 1: String  Variable-length grams?
Fixed-length 2-grams u n i v e r s a l Variable-length grams ni ivr sal uni vers [2,4]-gram dictionary u n i v e r s a l

Representing gram dictionary as a trie
ni ivr sal uni vers

Step 2: Constructing a gram dictionary
qmin=2 qmax=4 Frequency-based [LYW07] Cost-based [YLW08]

Challenge 3: Edit operation’s effect on grams
Fixed length: q u n i v e r s a l k operations could affect k * q grams

Deletion affects variable-length grams
Not affected Not affected Affected i-qmax+1 i i+qmax- 1 Deletion

With 2 edit operations, at most 4 grams can be affected
Main idea For a string, for each position, compute the number of grams that could be destroyed by an operation at this position Compute number of grams possibly destroyed by k operations Store these numbers (for all data strings) as part of the index Vector of s = <2,4,6,8,9> With 2 edit operations, at most 4 grams can be affected Use this number to do count filtering

Summary of VGRAM index

Challenge 4: adopting VGRAM
Easily adoptable by many algorithms Basic interfaces: String s  grams String s1, s2 such that ed(s1,s2) <= k  min # of their common grams

Lower bound on # of common grams
Fixed length (q) u n i v e r s a l If ed(s1,s2) <= k, then their # of common grams >=: (|s1|- q + 1) – k * q Variable lengths: # of grams of s1 – NAG(s1,k)

Example: algorithm using inverted lists
Query: “shtick”, ED(shtick, ?)≤1 sh ht tick tick 2-grams 2-4 grams 2 4 1 3 … ck ic ich tic tick 1 2 4 3 … ck ic ti Lower bound = 3 id strings 1 2 3 4 rich stick stich stuck static id strings 1 2 3 4 rich stick stich stuck static id strings 1 2 3 4 rich stick stich stuck static Lower bound = 1

End of part I Motivation Preliminaries Trie-based approach
Gram-based algorithms Sketch-based algorithms Compression Selectivity estimation Transformations/Synonyms Conclusion Part I Part II

Efficient Approximate Search on String Collections Part I

Similar presentations

Presentation on theme: "Efficient Approximate Search on String Collections Part I"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Efficient Approximate Search on String Collections Part I

Similar presentations

Presentation on theme: "Efficient Approximate Search on String Collections Part I"— Presentation transcript:

Similar presentations

About project

Feedback