Lecture 7 HMMs – the 3 Problems Forward Algorithm CSCE 771 Natural Language Processing Lecture 7 HMMs – the 3 Problems Forward Algorithm Topics Overview Readings: Chapter 6 February 6, 2013

Overview Last Time Today Tagging Markov Chains Hidden Markov Models NLTK book – chapter 5 tagging Today Viterbi dynamic programming calculation Noam Chomsky on You Tube Revisited smoothing Dealing with zeroes Laplace Good-Turing

Katz Backoff

Back to Tagging Brown Tagset - In 1967, Kucera and Francis published their classic work Computational Analysis of Present-Day American English – tags added later ~1979 500 texts each roughly 2000 words Zipf’s Law – “the frequency of the n-th most frequent word is roughly proportional to 1/n” Newer larger corpora ~ 100 million words Corpus of Contemporary American English, the British National Corpus or the International Corpus of English http://en.wikipedia.org/wiki/Brown_Corpus

Figure 5.4 pronoun in Celex Counts from COBUILD 16-million word corpus

Figure 5.6 Penn Treebank Tagset

Figure 5.7

Figure 5.7 continued

Figure 5.8

Figure 5.10

5.5.4 Extending HMM to Trigrams Find best tag sequence Bayes rule Markov assumption Extended for Trigrams

Chapter 6 - HMMs formalism revisited

Markov – Output Independence Markov Assumption Output Independence: (Eq 6.7)

Figure 6.2 initial probabilities

Figure 6.3 Example Markov chain Probability of a sequence

Figure 6.4 Probability zero links (Bakis model for temporal problems)

HMMs – The Three Problems

Likelihood Computation – The Forward Algorithm Computing Likelihood: Given an HMM λ = (A, B) and an observation sequence O = o1, o2, … ot, determine the likelihood P(O | λ)

Figure 6.5 B – observational Probabilities for 3 1 3 ice creams

Figure 6.6 transitions for 3 1 3 ice creams

Likelihood computation

Likelihood Probability – P(Q | λ)  

Fig 6.7 forward computation Example

Notations for the Forward Algorithm αt-1 (i) = previous forward probability from step t-1 for state I aij = the transition probability from state qi to qj bj(ot) = the observational likelihood = P(ot | qj) Note output independence means the Observational likelihood bj(ot) = P(ot | qj ) does not depend on the previous states or previous observations

Figure 6.8 Forward computation α1(j)

Figure 6.9 Forward Algorithm

Figure 6.10 Viterbi for Problem 2 Decoding – finding tag sequence that gives max

Figure 6.11 Viterbi again

Figure 6.12 Viterbi Example

Figure 6.13 Upcoming Attractions Next time learning the Model (A,B)