Distance functions and IE – 5 William W. Cohen CALD

Announcements Current statistics: –days with unscheduled student talks: 5 –students with unscheduled student talks: 3 –Projects are due: 4/28 (last day of class) –Additional requirement: draft (for comments) no later than 4/21

String distance metrics so far... Term-based (e.g. TF/IDF as in WHIRL) –Distance depends on set of words contained in both s and t – so sensitive to spelling errors. –Usually weight words to account for “importance” –Fast comparison: O(n log n) for |s|+|t|=n Edit-distance metrics –Distance is shortest sequence of edit commands that transform s to t. –No notion of word importance –More expensive: O(n 2 ) Other metrics –Jaro metric & variants –Monge-Elkan’s recursive string matching –etc? Which metrics work best, for which problems?

Results - Overall

Combining Information Extraction and Similarity Computations Krauthammer et al

Background Common task in proteomics/genomics: –look for (soft) matches to a query sequence in a large “database” of sequences. –want to find subsequences (genes) that are highly similar (and hence probably related) –want to ignore “accidental” matches –possible technique is Smith-Waterman (local alignment) want char-char “reward” for alignment to reflect confidence that the alignment is not due to chance

Background Common task in proteomics/genomics: –look for (soft) matches to a query sequence in a large “database” of sequences. –want to find subsequences (genes) that are highly similar (and hence probably related) –want to ignore “accidental” matches –possible technique is Smith-Waterman (local alignment) want char-char “reward” for alignment to reflect confidence that the alignment is not due to chance

Smith-Waterman distance c o h e n d o r f m c c o h n s k i dist=5

In general “peaks” in the matrix scores indicate highly similar substrings.

Background Common task in proteomics/genomics: –look for (soft) matches to a query sequence in a large “database” of sequences. –possible technique is Smith-Waterman (local alignment) want char-char “reward” for alignment to reflect confidence that the alignment is not due to chance based on substitutability theory/stats for amino acids –doesn’t scale well BLAST and FASTA: fast approximate S-W

BLAST/FASTA ideas Find all char n-grams (“words”) in the query string. FASTA: –Use inverted indices to find out where these words appear in the DB sequence –Use S-W only near DB sections that contain some of these words

BLAST/FASTA ideas Find all char n-grams (“words”) in the query string. BLAST: –Generate variations of these words by looking for changes that would lead to strong similarities –Discard “low IDF” words (where accidental matches are likely) –Use expanded set of n-grams to focus search

query string words and expansions

BLAST/FASTA ideas Find all char n-grams (“words”) in the query string. BLAST: –Generate variations of these words by looking for changes that would lead to strong similarities –Discard “low IDF” words (where accidental matches are likely) –Use expanded set of n-grams to focus search The BLAST program: –Widely used, –Fast implementation, –Supports asking multiple queries against a database at once... –Can one use it find soft matches of protein names (from a dictionary) in text?

Basic idea: Protein database Query strings Proposed alignment (query->database) Query algorithm: BLAST Biomedical paper Protein name dictionary Extracted protein name (dict. entry->text) IE system: dictionaries+BLAST (optimized for this problem)

1) Mapping text to DNA sequences (Q: what sort of char similarity is this?)

2) Optimizing blast Split protein-name database into several parts (for short, medium-length, long protein names) –Scoring depends on length of matched string Require space chars before and after “short” protein names. Manually search (grid search?) for better settings for certain key parameters for each protein-name subdatabase –With what data? Evaluate on one review article, 1162 protein names –inter-annotator agreement not great (70-85%)

2) Optimizing blast

Results

Overall: precision 71.1%, recall 78.8% (optimized)

IE with Dictionaries Cohen & Sarawagi

Finding names you know about Problem: given dictionary of names, find them in text –Important task beyond (biology, link analysis,...) –Exact match is unlikely to work perfectly, due to nicknames (Will Cohen), abbreviations (William C), misspellings (Willaim Chen), polysemous words (June, Bill), etc –In informal text it sometimes works very poorly –Problem is similar to record linkage (aka data cleaning, de-duping, merge-purge,...) problem of finding duplicate database records in heterogeneous databases.

Finding names you know about Problem: given dictionary of names, find them in text –Exact match is unlikely to work well for informal text. –Problem is similar to record linkage –Hard to combine state of the art similarity metrics (as used in record linkage) with state of the art NER system due to representational mismatch: Opening up the box, modern NER systems don’t really know anything about names....

IE as Sequential Word Classification Yesterday Pedro Domingos spoke this example sentence. Person name: Pedro Domingos A trained IE system models the relative probability of labeled sequences of words. To classify, find the most likely state sequence for the given words: Any words said to be generated by the designated “person name” state extract as a person name: person name location name background

IE as Sequential Word Classification Modern IE systems use a rich representation for words, and clever probabilistic models of how labels interact in a sequence, but do not explicitly represent the names extracted. w t-1 w t O t w t+1 O t +1 O t - 1 identity of word ends in “-ski” is capitalized is part of a noun phrase is in a list of city names is under node X in WordNet is in bold font is indented is in hyperlink anchor last person name was female next two words are “and Associates” … … part of noun phrase is “Wisniewski” ends in “-ski”

Semi-Markov models for IE Train on sequences of labeled segments, not labeled words. S=(start,end,label) Build probability model of segment sequences, not word sequences Define features f of segments (Approximately) optimize feature weights on training data f(S) = words x t...x u, length, previous words, case information,..., distance to known name maximize: with Sunita Sarawagi, IIT Bombay

Details: Semi-Markov model

Conditional Semi-Markov models CMM: CSMM:

A training algorithm for CSMM’s (1) Review: Collins’ perceptron training algorithm Correct tags Viterbi tags

A training algorithm for CSMM’s (2) Variant of Collins’ perceptron training algorithm: voted perceptron learner for T TRANS like Viterbi

A training algorithm for CSMM’s (3) Variant of Collins’ perceptron training algorithm: voted perceptron learner for T TRANS like Viterbi

A training algorithm for CSMM’s (3) Variant of Collins’ perceptron training algorithm: voted perceptron learner for T SEGTRANS like Viterbi

Sample CSMM features

Experimental results Baseline algorithms: –HMM-VP/1: tags are “in entity”, “other” –HMM-VP/4: tags are “begin entity”, “end entity”, “continue entity”, “unique”, “other” –SMM-VP: all features f(w) have versions for “f(w) true for some w in segment that is first (last, any) word of segment” –dictionaries: like Borthwick HMM-VP/1: f D (w)=“word w is in D” HMM-VP/4: f D,begin (w)=“word w begins entity in D”, etc, etc Dictionary lookup

Datasets used Used small training sets (10% of available) in experiments.

Results

Results: varying history

Results: changing the dictionary

Results: vs CRF