CS460/IT632 Natural Language Processing/Language Technology for the Web Lecture 2 (06/01/06) Prof. Pushpak Bhattacharyya IIT Bombay Part of Speech (PoS) Tagging

Prof. Pushpak Bhattacharyya, IIT Bombay Tagging or Annotation Purpose is Disambiguation A word can have a number of labels The problem is to give unique label. PoS tagging makes use of the “local context”, whereas Sense tagging needs “long distance dependency” and hence difficult too. PoS tagging is needed in mainly parsing and also in other applications. 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Prof. Pushpak Bhattacharyya, IIT Bombay Approaches Rule Based approach Statistical approach we will mainly focus on the statistical approach 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Prof. Pushpak Bhattacharyya, IIT Bombay Types of Tagging Tasks PoS Named entity Sense Parse tree 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Prof. Pushpak Bhattacharyya, IIT Bombay PoS Tagging Example “The Orange ducks clean the bills.” Assign tags to each word from the lexicon; multiple possibilities exist 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Prof. Pushpak Bhattacharyya, IIT Bombay Lexicon dictionary The: DT (Determiner) Orange: NN (Noun) JJ (Adjective) Duck: NN VB ( Basic verb) Clean: NN VB Bill: JJ, VB, NN are called as Syntactic entities or PoS tags 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

PoS tagging as a sequence labelling task Task is to assign the correct PoS tag sequence to the words. It can be: Unigram: Consider one word while deciding the sequence. Multigram: Consider multiple words. 16 (=1*2*2*2*1*2) possible sequences for the “Duck” example. It is a classification problem: classify each word’s tag correctly into the right category. 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Prof. Pushpak Bhattacharyya, IIT Bombay Challenges Lexical ambiguity: Multiple choices Morphology analysis: Find the root word Tokenization: Find word boundaries In Thai language there is no blank space Non trivial (example: capturing boundaries when the word is continued to the next line with a “-”) 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Prof. Pushpak Bhattacharyya, IIT Bombay Named Entity tagging Example 1: “Mohan went to school in Kolkata” Tagged as: “Mohan_Person went to School_Place in Kolkata_Place”. Example 2: “Kolkata bore the brunt of 1947 riots when 1947 children died at Kolkata. “Kolkata_? bore the brunt of 1947_year riots when 1947_num children died at Kolkata_Place. 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Prof. Pushpak Bhattacharyya, IIT Bombay Sense tagging Detecting the meaning. Our example tagged as: The Orange_{colour} ducks_{bird} clean the bills_{body_part} Sense tagging has been done by means of hypernymy. Semantic relations like hypernymy are stored in the lexical resource called “WordNet”. 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Prof. Pushpak Bhattacharyya, IIT Bombay Parse Tree tagging Example parse tree: 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Parse Tree tagging (contd.) Given a grammar, one can construct the parse tree. Annotation will produce following structure: [ [The_DT [Orange_JJ Ducks_NN]NP]NP [clean_VB[the_VB [bills_NN]NP]NP]VP]S This structure is called the Penn Treebank form From the Treebank form, one can arrive at a grammar through learning. 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Statistical Formulation of the PoS tagging problem Input: W1,W2,...Wn words C1,C2,....Cm Lexical tags reposition (DT,JJ, NN et. al.) Output: “Best” PoS tag sequence Ci1, Ci2, Ci3....Cin for the given words. Best means: P(Ci1, Ci2, Ci3....Cin|W1,W2,...Wn) is the maximum of all possible C-sequence. 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Statistical Formation of PoS tagging problem Example: P(DT JJ NN| The Orange duck) > P(DT NN VB| The Orange duck) is required Why?: Because given the phrase “The orange duck”, there is overwhelming evidence in the corpus that “DT JJ NN” is the right tag sequence. 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Mathematical machinery 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Prof. Pushpak Bhattacharyya, IIT Bombay Bayes Theorem P(A|B) = (P(A).P(B|A)) / P(B) Where, P(A): Prior probability P(A|B): Posterior probability P(B|A): likelihood Why apply Bayes theorem: This is the Generative Vs Discriminative model question. 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Prof. Pushpak Bhattacharyya, IIT Bombay Apply Bayes theorem P(Ci1, Ci2, Ci3....Cin|W1,W2,...Wn) = P(C|W) = where, C = <Ci1, Ci2, Ci3....Cin> W = <W1,W2,...Wn> P(C). P(W|C) P(W) 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Prof. Pushpak Bhattacharyya, IIT Bombay Best tag sequence C* = <Ci1, Ci2, Ci3....Cin>* , where * signifies best C- sequence = argmax(P(C|W)) As denominator is common in all the tag sequences Therefore, C* = argmax(P(C).P(W|C)) 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Prof. Pushpak Bhattacharyya, IIT Bombay Processing the1st part P(C) = P(Ci1, Ci2, Ci3....Cin) = P(Ci1).P(Ci2|Ci1).P(Ci3|Ci1. Ci2)..P(Cin|Ci1Ci2.. Cin-1) (on applying chain rule of probability) Ex: P(DT JJ NN) = P(DT).P(JJ|DT).P(NN|DT JJ) 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Prof. Pushpak Bhattacharyya, IIT Bombay Markov assumption Tag depends only on a window, not on everything that the “chain law” of probability demands. Kth order Markov assumption considers only previous K tags. Typical values of K = 3 for English, and (it seems) 5 for Hindi. 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Prof. Pushpak Bhattacharyya, IIT Bombay Apply assumption With K=2, our problem will be: P(C) = P(Ci|Ci-1), i: 1..n C0: sentence beginning marker. 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay

Exercise given in the lecture Contrast PoS tagging with Sense tagging. Find an example to show the difference. 06/01/06 Prof. Pushpak Bhattacharyya, IIT Bombay