인공지능 연구실 정 성 원 Part-of-Speech Tagging

2 The beginning The task of labeling (or tagging) each word in a sentence with its appropriate part of speech. –The representative put chairs on the table AT NN VBD NNS IN AT NN Using Brown/Penn tag sets A problem of limited scope –Instead of constructing a complete parse –Fix the syntactic categories of the word in a sentence Tagging is a limited but useful application. –Information extraction –Question and answering –Shallow parsing

3 The Information Sources in Tagging Syntagmatic: look at the tags assigned to nearby words; some combinations are highly likely while others are highly unlikely or impossible –ex) a new play –AT JJ NN –AT JJ VBP Lexical : look at the word itself. (90% accuracy just by picking the most likely tag for each word) –Verb is more likely to be a noun than a verb

4 Notation w i the word at position i in the corpus t i the tag of w i w i,i+m the words occurring at positions i through i+m t i,i+m the tags t i … t i+m for w i … w i+m w l the l th word in the lexicon t j the j th tag in the tag set C(w l ) the number of occurrences of w l in the training set C(t j )the number of occurrences of t j in the training set C(t j,t k )the number of occurrences of t j followed by t k C(w l,t j )the number of occurrences of w l that are tagged as t j Tnumber of tags in tag set Wnumber of words in the lexicon nsentence length

5 The Probabilistic Model (I) The sequence of tags in a text as Markov chain. –A word’s tag only depends on the previous tag (Limited horizon) –Dependency does not change over time (Time invariance) compact notation : Limited Horizon Property

6 The Probabilistic Model (II) Maximum likelihood estimate tag following

7 The Probabilistic Model (III) (We define P(t 1 |t 0 )=1.0 to simplify our notation) The final equation

8 The Probabilistic Model (III) Algorithm for training a Visible Markov Model Tagger Syntagmatic Probabilities: for all tags t j do for all tags t k do P(t k | t j )=C(t j, t k )/C(t j ) end Lexical Probabilities: for all tags t j do for all words w l do P(w l | t j )=C(w l, t j )/C(t j ) end

9 The Probabilistic Model (IV) Second tag First tagATBEZINNNVBPERIOD AT BEZ IN NN VB PERIOD

10 The Probabilistic Model (V) ATBEZINNNVBPERIOD bear is move on president progress the

11 The Viterbi algorithm comment : Given: a sentence of length n comment : Initialization δ 1 (PERIOD) = 1.0 δ 1 (t) = 0.0 for t ≠ PERIOD comment : Induction for i := 1 to n step 1 do for all tags tj do δ i+1 (t j ) := max 1<=k<=T [δ i (t k )*P(w i+1 |t j )*P(t j |t k )] ψ i+1 (t j ) := argmax 1<=k<=T [δ i (t k )*P(w i+1 |t j )*P(t j |t k )] end comment : Termination and path-readout X n+1 = argmax 1<=j<=T δ n+1 (j) for j := n to 1 step – 1 do X j = ψ j+1 (X j+1 ) end P(X 1, …, X n ) = max 1<=j<=T δ n+1 (t j )

12 Variations (I) Unknown words –Unknown words are a major problem for taggers –The simplest model for unknown words Assume that they can be of any part of speech –Use morphological information Past tense form : words ending in –ed –Capitalized

13 Variations (II) Trigram taggers –The basic Markov Model tagger = bigram tagger –two tag memory –disambiguate more cases Interpolation and variable memory –trigram tagger may make worse pridictions than a bigram tagger –linear interpolation Variable Memory Markov Model

14 Variations (III) Smoothing Reversibility –Markov model decodes from left to right = decodes from right to left K l is the number of possible parts of speech of w l

15 Variations (IV) Maximum Likelihood: Sequence vs. tag by tag –Viterbi Alogorithm : maximize P(t 1,n |w 1,n ) –Consider : maximize P(t i |w 1,n ) for all i which amounts to summing over different tag sequance –ex) Time flies like a arrow. a. NN VBZ RB AT NN.P(.) = 0.01 b. NN NNS VB AT NN.P(.) = 0.01 c. NN NNS RB AT NN.P(.) = d. NN VBZ VB AT NN.P(.) = 0 –one error does not affect the tagging of other words

16 Applying HMMs to POS tagging(I) If we have no training data, we can use a HMM to learn the regularities of tag sequences. HMM consists of the following elements –a set of states ( = tags ) –an output alphabet ( words or classes of words ) –initial state probabilities –state transition probabilities –symbol emission probabilities

17 Applying HMMs to POS tagging(II) Jelinek’s method –b j.l : probability that word (or word class) l is emitted by tag j

18 Applying HMMs to POS tagging(III) Kupiec’s method |L| is the number of indices in L

19 Transformation-Based Learning of Tags Markov assumption are too crude → transformation-based tagging Exploit a wider range An order of magnitude fewer decisions Two key components –a specification of which ‘error-correcting’ transformations are admissible –The learning algorithm

20 Transformation(I) A triggering environment A rewrite rule –Form t 1 →t 2 : replace t 1 by t 2

21 Transformation(II) environments can be conditioned –combination of words and tags Morphology-triggered transformation –ex) Replace NN by NNS if the unknown word’s suffix is -s Source tagTarget TagTrigging environment NNVBprevious tag is TO VBPVBone of the previous three tags is MD JJRRBRnext tag is JJ VBPVBone of the previous two words is n’t

22 The learning algorithm C 0 := corpus with each word tagged with its most frequent tag for k:=0 step 1 do ν:=the transformation u i that minimizes E(u i (C k )) if (E(C k )-E(ν(C k ))) < Є then break fi C k+1 := ν(C k ) τ k+1 := ν end Output sequence: τ 1, …, τ k

23 Relation to other models Decision trees –similarity with Transformation-based learning a series of relableing –difference with Transformation-based learning split at each node in a decision tree different sequence of transformation for each node Probabilistic models in general

24 Automata Transformation-based tagging has a rule component, it also has a quantitative component. Once learning is complete, transformation-based tagging is purely symbolic Transformation-based tagger can be converted into another symbolic object Roche and Schobes(1995) : finite state transducer Advantage : speed

25 Other Method, Other Languages Other approaches to tagging –In chapter 16 Languages other than English –In many other languages, word order is much freer –The rich inflections of a word contribute more information about part of speech

26 Tagging accuracy 95%~97% when calculated over all words Considerable factors –The amount of training data available –The tag set –The difference between training set and test set –Unknown words a ‘dump’ tagger –Always chooses a word’s most frequent tag –Accuracy of about 90% EngCG

27 Applications of tagging Benefit from syntactically disambiguated text Partial Parsing –Finding none phrases of sentence Information Extraction –Finding value for the predefined slots of a template –Finding good indexing term in information retrieval Question Answering –Returning an appropriate noun such as a location, a person, or a date