1. Pair HMMs and edit distance Ristad & Yianilos

2. Special meeting Wed 4/14 What: Evolving and Self-Managing Data Integration Systems Who: AnHai Doan, Univ. of Illinois at Urbana-Champaign When: Wednesday, April 14, 2004 @ 11am (food at 10:30am) Where: Sennott Square Building, room 5317

3. Special meeting 4/28 (last class) First International Joint Conference on Information Extraction, Information Integration, and Sequential Learning 10:30-11:50 am, Wean Hall 4601 All project proposals have been accepted as paper abstracts, and you’re all invited to present for 10min (including questions)

4. Pair HMMs – Ristad & Yianolis HMM review –notation –inference (forward algorithm) –learning (forward-backward & EM) Pair HMMs –notation –generating edit strings –distance metrics (stochastic, viterbi) –inference (forward) –learning (forward-backward & EM) Results from R&Y paper –K-NN with trained distance, hidden prototypes –problem: phoneme strings => words Advanced Pair HMMs –adding state (eg for affine gap models) –Smith Waterman? –CRF training? last week today

5. HMM Notation

6. HMM Example 1 2 Pr(1->2) Pr(2->1) Pr(2->2)Pr(1->1) Pr(1->x) d0.3 h0.5 b0.2 Pr(2->x) a0.3 e0.5 o0.2 Sample output: x T =heehahaha, s T =122121212

7. HMM Inference t=1t=2...t=T l=1... l=2... l=K... Key point: Pr(s i =l) depends only on Pr(l’->l) and s i-1 x1x1 x2x2 x3x3 xTxT

8. HMM Inference t=1t=2...t=T l=1... l=2... l=K... Key point: Pr(s i =l) depends only on Pr(l’->l) and s i-1 so you can propogate probabilities forward... x1x1 x2x2 x3x3 xTxT

9. HMM Inference – Forward Algorithm l=1.... l=2....... l=K.... x1x1 x2x2 x3x3 xTxT

10. HMM Learning - EM Expectation maximization: –Find expectations, i.e. Pr(s i =l) for i=1,...,T forward algorithm + epsilon hidden variables are states s at times t=1,...,t=T –Maximize probability of parameters given expectations: replace #(l’->l)/#(l’) with weighted version of counts replace #(l’->x)/#(l’) with weighted version

11. HMM Inference t=1t=2...t=T l=1... l=2... l=K... Forward algorithm: computes probabilities α(l,t) based on information in first t letters of string, ignores “downstream” information x1x1 x2x2 x3x3 xTxT

12. HMM Inference l=1... l=2... l=K x1x1 x2x2 x3x3 xTxT

13. HMM Learning - EM Expectation maximization: –Find expectations, i.e. Pr(s i =l) for i=1,...,T forward backward algorithm hidden variables are states s at times t=1,...,t=T –Maximize probability of parameters given expectations: replace #(l’->l)/#(l’) with weighted version of counts replace #(l’->x)/#(l’) with weighted version

14. Pair HMM Notation

15. Pair HMM Example 1 ePr(e) 0.10 0.10 0.10 0.05 0.05 0.01.....

16. Pair HMM Example 1 ePr(e) 0.10 0.10 0.10 0.05 0.05 0.01..... Sample run: z T =,,, Strings x,y produced by z T : x=heehee, y=teehe Notice that x,y is also produced by z 4 +, and many other edit strings

17. Distances based on pair HMMs

18. Pair HMM Inference Dynamic programming is possible: fill out matrix left-to-right, top-down

19. Pair HMM Inference t=1t=2...t=T v=1... v=2... v=K... x1x1 x2x2 x3x3 xTxT

20. Pair HMM Inference t=1t=2...t=T v=1... v=2... v=K...

21. Pair HMM Inference t=1t=2...t=T v=1... v=2... v=K... One difference: after i emissions of pair HMM, we do not know the column position i=1 i=3 i=1 i=2 i=3

22. Pair HMM Inference: Forward-Backward t=1t=2...t=T v=1... v=2... v=K...

23. Multiple states 1 ePr(e) 0.10 0.10 0.10 0.05 0.05 0.01..... 3 ePr(e) 0.10 0.10 0.10 0.05 2 ePr(e) 0.10 0.10 0.10 0.05

24. ...v=K... v=2...v=1 t=T...t=2 t=1 l=2 An extension: multiple states...v=K... v=2...v=1 t=T...t=2 t=1 l=1 conceptually, add a “state” dimension to the model EM methods generalize easily to this setting

25. Back to R&Y paper... They consider “coarse” and “detailed” models, as well as mixtures of both. Coarse model is like a back-off model – merge edit operations into equivalence classes (e.g. based on equivalence classes for chars). Test by learning distance for K-NN with an additional latent variable

26. K-NN with latent prototypes test example y (a string of phonemes) possible prototypes x (known word pronounciation ) x1x1 x2x2 x3x3 xmxm words from dictionary y w1w1 w2w2 wKwK learned phonetic distance

27. K-NN with latent prototypes x1x1 x2x2 x3x3 xmxm y w1w1 w2w2 wKwK learned phonetic distance Method needs (x,y) pairs to train a distance – to handle this, an additional level of E/M is used to pick the “latent prototype” to pair with each y

28. Hidden prototype K-nn

29. Experiments E1: on-line pronounciation dictionary E2: subset of E1 with corpus words E3: dictionary from training corpus E4: dictionary from training + test corpus (!) E5: E1 + E3

30. Experiments

32. Special meeting Wed 4/14 What: Evolving and Self-Managing Data Integration Systems Who: AnHai Doan, Univ. of Illinois at Urbana-Champaign When: Wednesday, April 14, 2004 @ 11am (food at 10:30am) Where: Sennott Square Building, room 5317

33. Special meeting 4/28 (last class) First International Joint Conference on Information Extraction, Information Integration, and Sequential Learning 10:30-11:50 am, Wean Hall 4601 All project proposals have been accepted as paper abstracts, and you’re all invited to present for 10min (including questions)