Pair HMMs and edit distance Ristad & Yianilos

Special meeting Wed 4/14 What: Evolving and Self-Managing Data Integration Systems Who: AnHai Doan, Univ. of Illinois at Urbana-Champaign When: Wednesday, April 14, 11am (food at 10:30am) Where: Sennott Square Building, room 5317

Special meeting 4/28 (last class) First International Joint Conference on Information Extraction, Information Integration, and Sequential Learning 10:30-11:50 am, Wean Hall 4601 All project proposals have been accepted as paper abstracts, and you’re all invited to present for 10min (including questions)

Pair HMMs – Ristad & Yianolis HMM review –notation –inference (forward algorithm) –learning (forward-backward & EM) Pair HMMs –notation –generating edit strings –distance metrics (stochastic, viterbi) –inference (forward) –learning (forward-backward & EM) Results from R&Y paper –K-NN with trained distance, hidden prototypes –problem: phoneme strings => words Advanced Pair HMMs –adding state (eg for affine gap models) –Smith Waterman? –CRF training? last week today

HMM Notation

HMM Example 1 2 Pr(1->2) Pr(2->1) Pr(2->2)Pr(1->1) Pr(1->x) d0.3 h0.5 b0.2 Pr(2->x) a0.3 e0.5 o0.2 Sample output: x T =heehahaha, s T =

HMM Inference t=1t=2...t=T l=1... l=2... l=K... Key point: Pr(s i =l) depends only on Pr(l’->l) and s i-1 x1x1 x2x2 x3x3 xTxT

HMM Inference t=1t=2...t=T l=1... l=2... l=K... Key point: Pr(s i =l) depends only on Pr(l’->l) and s i-1 so you can propogate probabilities forward... x1x1 x2x2 x3x3 xTxT

HMM Inference – Forward Algorithm l=1.... l= l=K.... x1x1 x2x2 x3x3 xTxT

HMM Learning - EM Expectation maximization: –Find expectations, i.e. Pr(s i =l) for i=1,...,T forward algorithm + epsilon hidden variables are states s at times t=1,...,t=T –Maximize probability of parameters given expectations: replace #(l’->l)/#(l’) with weighted version of counts replace #(l’->x)/#(l’) with weighted version

HMM Inference t=1t=2...t=T l=1... l=2... l=K... Forward algorithm: computes probabilities α(l,t) based on information in first t letters of string, ignores “downstream” information x1x1 x2x2 x3x3 xTxT

HMM Inference l=1... l=2... l=K x1x1 x2x2 x3x3 xTxT

HMM Learning - EM Expectation maximization: –Find expectations, i.e. Pr(s i =l) for i=1,...,T forward backward algorithm hidden variables are states s at times t=1,...,t=T –Maximize probability of parameters given expectations: replace #(l’->l)/#(l’) with weighted version of counts replace #(l’->x)/#(l’) with weighted version

Pair HMM Notation

Pair HMM Example 1 ePr(e)

Pair HMM Example 1 ePr(e) Sample run: z T =,,, Strings x,y produced by z T : x=heehee, y=teehe Notice that x,y is also produced by z 4 +, and many other edit strings

Distances based on pair HMMs

Pair HMM Inference Dynamic programming is possible: fill out matrix left-to-right, top-down

Pair HMM Inference t=1t=2...t=T v=1... v=2... v=K... x1x1 x2x2 x3x3 xTxT

Pair HMM Inference t=1t=2...t=T v=1... v=2... v=K...

Pair HMM Inference t=1t=2...t=T v=1... v=2... v=K... One difference: after i emissions of pair HMM, we do not know the column position i=1 i=3 i=1 i=2 i=3

Pair HMM Inference: Forward-Backward t=1t=2...t=T v=1... v=2... v=K...

Multiple states 1 ePr(e) ePr(e) ePr(e)

...v=K... v=2...v=1 t=T...t=2 t=1 l=2 An extension: multiple states...v=K... v=2...v=1 t=T...t=2 t=1 l=1 conceptually, add a “state” dimension to the model EM methods generalize easily to this setting

Back to R&Y paper... They consider “coarse” and “detailed” models, as well as mixtures of both. Coarse model is like a back-off model – merge edit operations into equivalence classes (e.g. based on equivalence classes for chars). Test by learning distance for K-NN with an additional latent variable

K-NN with latent prototypes test example y (a string of phonemes) possible prototypes x (known word pronounciation ) x1x1 x2x2 x3x3 xmxm words from dictionary y w1w1 w2w2 wKwK learned phonetic distance

K-NN with latent prototypes x1x1 x2x2 x3x3 xmxm y w1w1 w2w2 wKwK learned phonetic distance Method needs (x,y) pairs to train a distance – to handle this, an additional level of E/M is used to pick the “latent prototype” to pair with each y

Hidden prototype K-nn

Experiments E1: on-line pronounciation dictionary E2: subset of E1 with corpus words E3: dictionary from training corpus E4: dictionary from training + test corpus (!) E5: E1 + E3

Experiments

Special meeting Wed 4/14 What: Evolving and Self-Managing Data Integration Systems Who: AnHai Doan, Univ. of Illinois at Urbana-Champaign When: Wednesday, April 14, 11am (food at 10:30am) Where: Sennott Square Building, room 5317

Special meeting 4/28 (last class) First International Joint Conference on Information Extraction, Information Integration, and Sequential Learning 10:30-11:50 am, Wean Hall 4601 All project proposals have been accepted as paper abstracts, and you’re all invited to present for 10min (including questions)