1. Text Search and Fuzzy Matching
Presented by Andre Dovgal, Sunaptic Solutions
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

2. Focus of the Presentation
Text Search in Big Databases
Data Cleansing in ETL
Word Matching
Usage of Different Matching Algorithms
Real-world data is "dirty" because of misspellings, truncations, missing or inserted tokens, null fields, unexpected abbreviations, and other irregularities.
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

3. Scenarios Scenario 1 Scenario 2 (ETL) Scenario 3 User Interface
“Dirty” Data
Other Systems
Other Systems
“clean” request for search
“dirty” request for search
“Dirty” Data
“Clean” Data
“Clean” Data
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

4. Text Search Challenges
Improving Search Speed
Searching for a substring in a string regardless of the substring nature.
Improving Relevance of Results
Searching for words of a human language.
Domain dependence.
See examples:
- Rabin-Karp algorithm
- Knuth-Morris-Pratt algorithm
- Boyer-Moore algorithm
- more …
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

5. Word Matching Approaches
Exact Matching
Partial Matching (Pattern Matching)
Grammatical Algorithms: Stemming Matching and Synonym Matching (Semantics)
Phonetic Matching
Fuzzy Matching
Exact, partial matching. Matches text strings, regardless of language, grammar, domain, sentences, etc.
Grammatical algorithms. Stemming matching uses grammar, morphology, matches words rather than just text strings. Domain does not matter.
Synonym matching works with semantics. Domain dependent.
Phonetic matching works with words. Most of the algorithms were designed for specific domains, i.e. names.
Fuzzy matching based on the probability of typographic errors. What else?
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

6. Exact Matching No additional challenge except speed.
Domain does not really matter.
Example: search in a file using notepad program.
Example (SQL): SELECT field FROM table WHERE field = ‘string’.
MS SQL Server: Proper indexing improves speed.
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

7. Partial Matching Domain does not really matter.
Example: wildcards, search patterns.
Example (SQL): SELECT field FROM table WHERE field LIKE ‘string%’.
MS SQL Server: Proper indexing improves speed.
Example (SQL): SELECT field FROM table WHERE field BETWEEN ‘string1’ AND ‘string2’.
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

8. Full-text Search in MS SQL Server
Needs MS Search Service (for SQL Server 2000)
Included in MS SQL Server 2005 as SQL Server Full Text Search Service
CONTAINS Predicate
Unlike LIKE, CONTAINS matches words.
Can search for a word inflectionally generated from another (stemming matching).
Can search for a word near another word.
SQL Server discards noise words from the search criteria.
FREETEXT Predicate
A word or phrase close to the search word or phrase.
Needs Additional Space on Disk
-- diagram from the slide #9 + description
EXAMPLES
Prefix search:
SELECT * FROM Production.ProductDescription WHERE CONTAINS(Description, ' "bik*" ')
Inflectional search:
SELECT * FROM Production.ProductDescription WHERE CONTAINS(Description, 'FORMSOF(INFLECTIONAL, "ride")')
Search for a word near another word:
SELECT * FROM Production.ProductDescription WHERE CONTAINS(Description, 'bike NEAR performance')
Size: 1 MB for Production.ProductDescription table. Nvarchar(400)* 762 records < 300K the table itself.
Size: 6.14 MB for FuzzyTest 2 varchars (50) * 280,000 records < 30,000,000 B – not bad
SELECT ProductName FROM Products WHERE CONTAINS(ProductName, ' "choc*" ')
SELECT ProductName FROM Products WHERE CONTAINS(ProductName, ' FORMSOF (INFLECTIONAL, dry) ')
SELECT ProductName FROM Products WHERE CONTAINS(ProductName, 'spread NEAR Boysenberry')
SELECT CategoryName FROM Categories WHERE FREETEXT (Description, 'sweetest candy bread and dry meat' )
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

9. Full-text Search Architecture in MS SQL 2000
MS Search Service is separate from MS SQL Server. Full-text catalog files are separate from the DB. Need a separate backup procedure. In MS SQL Server 2005 – are included in the backup.
Needs to create indexes first. To create a full-text index on a table, the table must have a single, unique not null column.
Indexes are sets of tokens indicating the position of the words in column.
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

10. Full-text Search Architecture in MS SQL 2005
CREATE/ALTER FULLTEXT INDEX – in T-SQL for MS SQL Server 2005
Accent sensitivity
Thesaurus. Only synonyms. The thesaurus files are empty, you must add the word yourself. XML files.
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

11. Grammatical Algorithms
Stemming Match
We already saw SQL Server Full Text search.
Google example: “cutting and paste”.
Why needs dictionary: to determine the stem.
MS Search Service provides only inflectional, not derivational, word generation.
Synonym Match
Most Grammatical Algorithms are Based on Dictionaries
Quasi Stemming Match
Can be developed without a main dictionary (using quasi–endings tree).
Relatively low relevance.
MS Search Service provides only inflectional, not derivational, word generation. Therefore, only words in the same family (noun, verb, adjective, adverb, and so forth) are generated. For example, gerunds are not generated from nouns. Stemming "swim" generates swim, swam, swum, since these are all verbs — but not nouns such as swimmer and swimmers.
Grammatical Algorithms – need Dictionaries and Grammar Rules.
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

12. Phonetic Matching
Phonetic Matching Algorithms (or Phonetic Encoding, or “Sounds Alike” Algorithms)
Language Dependent
Domain Dependent
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

13. Phonetic Matching Algorithms
The original SoundEx Algorithm
Has been used in US census since late 1890s.
Was patented by Margaret O'Dell and Robert C. Russell in 1918.
Improvements: Phonix (1988), Editex (phonetic distance measuring, circa 2000), etc.
Metaphone and Double Metaphone Algorithms
Author: Lawrence Phillips and 2000.
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

14. SoundEx Algorithm
1. Capitalize all letters in the word and drop all punctuation marks. Pad the word with rightmost blanks as needed during each procedure step.
2. Retain the first letter of the word.
3. Change all occurrence of the following letters to '0' (zero):   'A', E', 'I', 'O', 'U', 'H', 'W', 'Y'.
4. Change letters from the following sets into the digit given:
1 = 'B', 'F', 'P', 'V'
2 = 'C', 'G', 'J', 'K', 'Q', 'S', 'X', 'Z'
3 = 'D','T'
4 = 'L'
5 = 'M','N'
6 = 'R'
5. Remove all pairs of digits which occur beside each other from the string that resulted after step 4.
6. Remove all zeros from the string that results from step 5 (placed there in step 3).
7. Pad the string that resulted from step (6) with trailing zeros and return only the first four positions, which will be of the form <uppercase letter> <digit> <digit> <digit>.
select soundex(‘smyth')
select soundex('smoothie')
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

15. More About SoundEx Example (SQL) DIFFERENCE
Oracle SOUNDEX – Slightly Different from SQL Server SOUNDEX
Seems That Major DBMSs (SQL Server, Oracle, DB2) Don’t Have a Better Phonetic Matching
Enhancements
Replace DG with G etc.
Phonix algorithm.
select FirstName, LastName from Person.Contact where soundex(LastName) = soundex('pauluk')
select FirstName, LastName from Person.Contact where soundex(LastName) = soundex('damis')
select FirstName, LastName from Person.Contact where soundex(LastName) = soundex('smoothie')
select FirstName, LastName from FuzzySearchTest where soundex(LastName) = soundex('pauluk')
select FirstName, LastName from FuzzySearchTest where soundex(LastName) = soundex('smoothie')
select FirstName, LastName, difference (LastName, 'smoothie') from FuzzySearchTest where soundex(LastName) = soundex('smoothie')
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

16. SoundEx Limitations
SoundEx is only usable in applications that can tolerate high false positives (when words that don't match the sound of the inquiry are returned) and high false negatives (when words that match the sound of the inquiry are NOT returned).
In many instances, unreliable interfaces are used as a foundation, upon which a reliable layer may be built. Interfaces that build a reliable layer, based on context, over a SoundEx foundation may also be possible.
SQL: word can’t start with a space.
Mistake in first letter results in 100% mismatch.
SoundEx acts as a bridge between the fuzzy and inexact process of human vocal interaction, and the concise true/false processes at the foundation of computer communication. As such, SoundEx is an inherently unreliable interface. For this reason, SoundEx is only usable in applications that can tolerate high false positives (when words that don't match the sound of the inquiry are returned) and high false negatives (when words that match the sound of the inquiry are NOT returned).
Only 33% of the matches that would be returned by Soundex would be correct. Even more significant was the finding that 25% of correct matches would fail to be discovered by Soundex. (Alan Stanier, September 1990, Computers in Genealogy, Vol. 3, No. 7)
Only 36.37% of Soundex returns were correct, while more than 60% of correct names were never returned by Soundex. (A.J. Lait and B. Randell, 1996)
This limitation is true even of the best SoundEx improvement techniques available. As long as you accept and honor this limitation, SoundEx and its derivatives can be a very useful tool in helping to improve the quality and usefulness of databases.
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

17. Metaphone and Double Metaphone
An algorithm to code English words phonetically by reducing them to 16 consonant sounds.
Double Metaphone
An algorithm to code English words (and foreign words often heard in the United States) phonetically by reducing them to 12 consonant sounds.
Author: Lawrence Phillips, 1990 and 2000
Metaphone Description and Demo:
SQL Example
select FirstName, LastName from Person.Contact where dbo.DoubleMetaPhone(LastName) = dbo.DoubleMetaPhone('damis')
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

18. Double Metaphone Advantages and Limitations
Free, Efficient, and Easy to Use
Provides Better Results Compare to SOUNDEX
Returns Two Possible Matches
Works Best with Proper Names
May Fail to Match Misspelled Words
Much Slower than SOUNDEX
Though it works as a general-purpose phonetic search algorithm, Double Metaphone was designed for, and works best with, searching lists of proper names rather than large fields of generic text.
Double Metaphone provides minimal ranking ability, apart from the three match levels described elsewhere in the series. This limits the ability to tune search results.
Being a phonetic matching (vs. fuzzy matching like q-grams and edit distances) algorithm, Double Metaphone may fail to match misspelled words when the misspelling substantively alters the phonetic structure of a word.
Even bearing these limitations in mind, Double Metaphone is free, efficient, easy to use, and adaptable to a number of scenarios. Ultimately, only the designer of a particular system can decide if Double Metaphone is appropriate to his/her particular problem space.
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

19. Fuzzy Matching What is Fuzzy Matching?
Fuzzy query in Index Server are simple prefix matching, like dog* returns dogmatic and doghouse, + stem matching.
Originally Meant “Not Exact Matching”
Web Search Engines
Edit Distance Based Algorithms
Simple: Hamming distance algorithms.
Most popular: Levenshtein distance algorithm.
Q-Gram Based Algorithms
Both Types of Algorithms Are Language and Domain Independent
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

20. Levenshtein Distance Developed in 1965
LD is a Measure of the Similarity Between Two Strings
It is the smallest number of insertions, deletions, and substitutions required to change one string into another.
Language and Domain Independent
Demo
More demos:
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

21. Q-Grams
Q-Grams Are Obtained by Sliding a Window of Size Q over the Characters of a Given String
Example
2-grams of “john smith” are $j jo oh hn n_ _s sm mi it th h#
IDEA: If Strings Match, They Have Many Common Q-Grams
Example: “john smith” and jonh smith” have 9 common q-grams.
Language and Domain Independent
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

22. Designed for data cleanup.
“Fuzzy” SSIS
Fuzzy Lookup enables to match input records with clean, standardized records in a reference table.
Fuzzy Grouping enables to identify groups of records in a table where each record in the group potentially corresponds to the same real-world entity.
Designed for data cleanup.
Based on Q-Grams and Levenshtein Distance (?).
Fuzzy search databases can be amassed that compile common misspellings (or variants) of specific words which can then be substituted during the cleansing process.
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

23. Design a Simple SSIS Fuzzy Lookup Package
Setting Up
String Data Types (DT_STR and DT_WSTR)
ETI (Error-Tolerant Index), Tokens, Delimiters
Tokens are not Q-Grams
Similarity Threshold
Number of Matches
The lower Similarity, the more likely to find a match, the search could take longer.
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

24. Can Fuzzy Lookup Be Accessed From C# Code?
NOT YET
Develop your own implementation of the algorithm.
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.

25. Conclusions Language and Domain Knowledge is Important
No Implementations? – Develop Yourself!
Questions?
April 20, 2017
© Copyright 2004 Sunaptic Solutions. All rights reserved.