Lecture 7 HMMs – the 3 Problems Forward Algorithm CSCE 771 Natural Language Processing Lecture 7 HMMs – the 3 Problems Forward Algorithm Topics Overview Readings: Chapter 6 February 6, 2013

Overview Last Time Today Tagging Markov Chains Hidden Markov Models NLTK book – chapter 5 tagging Today Viterbi dynamic programming calculation Noam Chomsky on You Tube Revisited smoothing Dealing with zeroes Laplace Good-Turing

Katz Backoff

Back to Tagging Brown Tagset - In 1967, Kucera and Francis published their classic work Computational Analysis of Present-Day American English – tags added later ~1979 500 texts each roughly 2000 words Zipf’s Law – “the frequency of the n-th most frequent word is roughly proportional to 1/n” Newer larger corpora ~ 100 million words Corpus of Contemporary American English, the British National Corpus or the International Corpus of English http://en.wikipedia.org/wiki/Brown_Corpus

Figure 5.4 pronoun in Celex Counts from COBUILD 16-million word corpus

Figure 5.6 Penn Treebank Tagset

Figure 5.7

Figure 5.7 continued

Figure 5.8

Figure 5.10

5.5.4 Extending HMM to Trigrams Find best tag sequence Bayes rule Markov assumption Extended for Trigrams

Chapter 6 - HMMs formalism revisited

Markov – Output Independence Markov Assumption Output Independence: (Eq 6.7)

Figure 6.2 initial probabilities

Figure 6.3 Example Markov chain Probability of a sequence

Figure 6.4 Probability zero links (Bakis model for temporal problems)

HMMs – The Three Problems

Likelihood Computation – The Forward Algorithm Computing Likelihood: Given an HMM λ = (A, B) and an observation sequence O = o1, o2, … ot, determine the likelihood P(O | λ)

Figure 6.5 B – observational Probabilities for 3 1 3 ice creams

Figure 6.6 transitions for 3 1 3 ice creams

Likelihood computation

Figure 6.7 forward computation

Figure 6.8

Figure 6.9 Forward Algorithm

Figure 6.10

Figure 6.11

Figure 6.12

Figure 6.13

Figure 6.14