Part-of-Speech Tagging Using Hidden Markov Models Alankar kotwal Sharad mirani 12d070010 12d070003 Hi, I’m Alankar, this is Sharad. Our project is about a crucial part of Natural Language Processing, called POS Tagging. EE 638: Estimation and Identification

Teach a computer to recognize the role of words in a sentence So this is the problem in general.

‘Roles of words’? Why do we care? Text ? Understanding Text ‘Why do we care?’ is a natural question to ask. The entire field of Natural Language Processing is built with the hope that computes can, one day, ‘understand’ text content as well as humans can. However, there is an inherent gap between words, their meanings an the concept they convey in a sentence. Clearly, all three are important in understanding the sentence. Our job here is to deal with the highest level of all, at a concept level. Each word expresses a concept: a noun is an entity, an adjective is a quality. We see how parts of speech come in naturally while trying to understand the ‘concept’ carried by words in a sentence.

Why is this non-trivial? Are you understanding this? She was very understanding. Verb (Participle)! Adjective! One might say, this could easily be achieved! Build a set of rules that dictate what word is what part of speech. But there exist words that can act as multiple parts of speech with changing context. Even the word ‘dogs’ can be a verb. Some words like ‘still’ can represent as many as 7 distinct parts of speech. Now let us take the example on the screen. What is the difference between ‘understanding’ used in both sentences? It is a verb in the first sentence and an adjective in the second. Note that given the ‘context’ of the word, it is possible to predict the POS definitively! This turns out to be true in general. In fact, this motivates why Hidden Markov Models are such a good fit for the problem. Context Context

The Markov Model: Transitions Noun Pronoun Start Verb End Thus, in general, a small sentence model would look like this. For instance, an instance ‘She was very understanding’ would have been pronoun, verb, adj, adj. Article Adjective

The Markov Model: Emissions Noun He, She, It Is, Was, Presenting Estimation, Identification Pronoun Start Verb End Each state would emit words, we model the distribution of words it emits. Unknown words? Define word __UNK__ which has equal probability of emission for all states. Beautiful, Interesting Article A, An, The Adjective

Sentence Representation Words POS Input Numbers HashMap Computers cannot handle words. How do we represent sentences? HashMaps.

Our dataset: WSJ Excerpts Confidence/NN in/IN the/DT pound/NN is/VBZ widely/RB expected/VBN to/TO take/VB another/DT sharp/JJ dive/NN ./. Word/POS We picked out a dataset of sentences taken from the Wall Street Journal. You see an example on the screen.

Estimating Transition Probability Training Estimating Transition Probability Increment the “next POS” count for each pair of POS Estimating Emission Probability Increment the “POS type” for each word How do we train?

Hash the input test cases Testing Hash the input test cases Run Viterbi algorithm How do we test?

Our results A set is a collection. => {ART, NN, VBZ, ART, NN, .} Did you set my alarm? => {VBD, PRP, VB, PRP$, NN, .} When bank financing for the buy-out collapsed last week, so did UAL's stock. => {WRB, NN, NN, IN, DT, NN, VBD, JJ, NN, ,, RB, VBD, NNP, POS, NN, .} This is my awesome* project for Estimation*. => {DT, VBZ, PRP$, JJ, NN, IN, NN, .} A few results demonstrating the capability of the HMM to model context. The stars denote words not used in the train set. Train set with 8637 sentences, test set with 1952 sentences, 45 POS => Error rate 5.17%

Where do we fail? Unknown words in some cases => Not much we can do without ‘meaning’ Confusing usage: How do you tell if set is present or past? => Global context? Difficulty disambiguating POS from just previous word => Trigram HMMs! So here are our failure cases. Compare these to what a human would do in the same situation!

Thank You It has been a real pleasure exploring NLP from a statistical perspective! Thank you!