Part-of-speech tagging A simple but useful form of linguistic analysis Christopher Manning

Parts of Speech Perhaps starting with Aristotle in the West (384–322 BCE), there was the idea of having parts of speech a.k.a lexical categories, word classes, “tags”, POS It comes from Dionysius Thrax of Alexandria (c. 100 BCE) the idea that is still with us that there are 8 parts of speech in Greek But actually his 8 aren’t exactly the ones we are taught today Thrax : noun, verb, article, adverb, preposition, conjunction, participle, pronoun School English grammar: noun, verb, adjective, adverb, preposition, conjunction, pronoun, interjection 2

Why POS is important? Parts-of-speech are useful because of the large amount of information they give about a word and its neighbors: For instance: Knowing whether a word is a noun or a verb tells us a lot about likely neighboring words (nouns are preceded by determiners and adjectives, verbs by nouns) and about the syntactic structure around the word (nouns are generally part of noun phrases), which makes part-of-speech tagging an important component of syntactic parsing(ch12) Parts of speech are useful features for finding named entities like people or organizations in text and other information extraction tasks Parts of speech are useful in summarization for improving the selection of nouns or other important words from a document

Open class (lexical) words Nouns Verbs Adjectives old older oldest Proper Common Main Adverbs slowly IBM Italy cat / cats snow see registered Numbers … more 122,312 one Closed class (functional) auxiliary Determiners the some can had Prepositions to with Conjunctions and or Particles off up … more Pronouns he its Interjections Ow Eh

Open vs. Closed classes Parts-of-speech can be divided into two broad supercategories: Closed class: are those with relatively fixed membership , such as: Prepositions: on, under, over, near, by, at, from, to, with . Determiners: a, an, the pronouns: she, who, I, Conjunctions: and, but, or, as, if, when . Auxiliary verbs: can, may, should, are particles: up, down, on, off, in, out, at, by. Numerals: one, two, three, first, second, third. Why “closed”? Open class: new nouns and verbs like iPhone or to fax are continually being created or borrowed. Nouns, main Verbs, Adjectives, Adverbs. Why “open”? 5

Penn Treebank POS (45 tags now used instead of 8)

POS Tagging The POS tagging problem is to determine the POS tag for a particular instance of a word because words are ambiguous have more than one possible POS. Example of word has more than one POS: “back” The back door = JJ “adjective” On my back = NN “noun” Win the voters back = RB “adverb” Promised to back the bill = VB “verb base”

Deciding on the correct part of speech can be difficult even for people Mrs/NNP Shaefer/NNP never/RB got/VBD around/RP to/TO joining/VBG All/DT we/PRP gotta/VBN do/VB is/VBZ go/VB around/IN the/DT corner/NN Chateau/NNP Petrus/NNP costs/VBZ around/RB 250/CD

How hard is the tagging problem? And how common is tag ambiguity? Fig. 9.2 shows the answer for the Brown and WSJ corpora tagged using the 45-tag Penn tree bank

POS tagging performance How many tags are correct? Is called (Tag accuracy) Baseline is already 90% (Tag accuracy) Baseline is algorithm for part of speech tagging POS which its goal to reduce that ambiguous words which is Tagging every word with its most frequent tag and Tagging unknown words as nouns Baseline is partly easy (why?) because Many words are unambiguous (85%-86%) You get points for them (the, a, etc.) and for punctuation marks! Because it easy to disambiguate. For example: “a” can be a determiner or the letter a (perhaps as part of an acronym or an initial). But the determiner sense of a is much more likely About 97% (Tag accuracy )currently by HMM, MEMM and other models.

Part-of-speech tagging A simple but useful form of linguistic analysis Christopher Manning

Part-of-speech tagging revisited A simple but useful form of linguistic analysis Christopher Manning

Sources of information What are the main sources of information for POS tagging? Knowledge of neighboring words Bill saw that man yesterday NNP NN DT NN NN VB VB(D) IN VB NN Knowledge of word probabilities (Knowledge of most frequent tag to word) man is rarely used as a verb…. These were the features of HMM taggers model

More and Better Features  Feature-based tagger Can do surprisingly well just looking at a word by itself: Word the: the  DT Lowercased word Importantly: importantly  RB Prefixes unfathomable: un-  JJ Suffixes Importantly: -ly  RB Capitalization Meridian: CAP  NNP Word shapes 35-year: d-x  JJ Then build a maxent (or whatever) model to predict tag Maxent P(t|w): 93.7% overall / 82.6% unknown Doing better

Overview: POS Tagging Accuracies Rough accuracies: Most freq tag: ~90% / ~50% Trigram HMM: ~95% / ~55% Maxent P(t|w): 93.7% / 82.6% TnT (HMM++): 96.2% / 86.0% MEMM tagger: 96.9% / 86.9% Bidirectional dependencies: 97.2% / 90.0% Upper bound: ~98% (human agreement) Most errors on unknown words

How to improve supervised results? Build better features! We could fix this with a feature that looked at the next word We could fix this by linking capitalized words to their lowercase versions RB PRP VBD IN RB IN PRP VBD . They left as soon as he arrived . JJ NNP NNS VBD VBN . Intrinsic flaws remained undetected .

Tagging Without Sequence Information Baseline Three Words t0 t0 w0 w-1 w0 w1 Model Features Token Unknown Sentence Baseline 56,805 93.69% 82.61% 26.74% 3Words 239,767 96.57% 86.78% 48.27% Using words only in a straight classifier works as well as a basic (HMM or discriminative) sequence model!!