1. CSC 594 Topics in AI – Natural Language Processing
Spring 2016/17
2. Linguistic Essentials
(Some slides adapted from Ralph Grishman at NYU, Joyce Choi at Michigan State and Andrew McCallum, UMass Amherst)

2. Levels of Language Analysis
Phonology
study of sound systems of languages
Morphology
study of structure of words: the structure of words in a language, including patterns of inflections and derivations
Syntax
study of organization of words in sentences: the ordering of and relationship between the words in phrases and sentences
Semantics
study of meaning in language: the study of how meaning in language is created
Pragmatics
study of language in use: the branch of linguistics that studies language use rather than language structure
Discourse
study of language, especially the type of language used in a particular context or subject
World/Common-sense Knowledge

3. Parts of Speech
There are eight major parts of speech for words in the English language: noun, pronoun, verb, adjective, adverb, preposition, conjunction and interjection.
The part of speech indicates how the word functions in meaning as well as grammatically within a sentence.
Noun: people, animals, concepts, things (e.g. “birds”)
Pronoun: a word used in place of a noun (e.g. “it”, “they”, “I”, “she”)
Verb: express action in the sentence (e.g. “sing”)
Adjective: describe properties of nouns (e.g. “yellow”)
Adverb: modifies or describes a verb, an adjective, or another adverb (e.g. “extremely”, “slowly”)
Preposition: a word placed before a noun/pronoun to form a phrase modifying another word/phrase (e.g. “in”, “for”, “without”)
Conjunction: a word that connects/conjoins another sentence or phrase (e.g. “and”, “or”, “but”, “so”, “Since”)
Interjection: a word that expresses spontaneous feeling (e.g. “uh”).

4. Nouns can form plural and/or possessive countable nouns vs. mass nouns
cat  cats, cat’s
countable nouns vs. mass nouns
singular countable nouns must appear with a determiner:
Cats sleep.
* Cat sleeps.
The cat sleeps.
Ralph Grishman at NYU

5. Nouns can form plural and/or possessive countable nouns vs. mass nouns
cat  cats, cat’s
countable nouns vs. mass nouns
singular countable nouns must appear with a determiner:
Cats sleep.
* Cat sleeps. (* indicates this is not a grammatical sentence)
The cat sleeps.
Ralph Grishman at NYU

6. Nouns can form plural and/or possessive countable nouns vs. mass nouns
cat  cats, cat’s
countable nouns vs. mass nouns
singular countable nouns must appear with a determiner (mainly articles (“a”, “the”) and possessive pronouns (“my”, “his”)):
Cats sleep.
* Cat sleeps.
The cat sleeps.
Ralph Grishman at NYU

7. Verbs Most verbs can appear in “They must _____ (it).”
Verbs can occur in different (inflected) forms:
base or infinitive ("be", "eat", "sleep")
present tense ("is", "am", "are";  "eats", "eat"; "sleeps", "sleep")
past tense ("was", "were";  "ate"; "slept")
present participle ("being", "eating"; "sleeping")
past participle ("been", "eaten"; "slept")
Ralph Grishman at NYU

8. Adjectives Adjectives can appear in comparative or superlative forms:
happy  happier, happiest
and with an intensifier:
happy  very happy
Ralph Grishman at NYU

9. Adjectives vs. nouns
we will not consider a word an adjective just because it appears as a modifier to the left of a noun:
“the brick wall”
most nouns can appear in this position
Ralph Grishman at NYU

10. Adverbs Can move within sentence: He ate the brownie quickly.
He quickly ate the brownie.
Quickly, he ate the brownie.
Ralph Grishman at NYU

11. Personal Pronouns
personal pronouns occur in nominative (“I”, “he”) and accusative (“me”, “him”)
last remaining evidence of case in English
Ralph Grishman at NYU

12. Quiz!
Identify and tag all words which are the basic eight parts-of-speech in the following sentences.
“The student put books on the table.”
“We may also collect information you voluntarily add to your profile, such as your mobile phone number and mobile service provider.”

13. Source: Joyce Choi, CSE 842, Michigan State University
Morphology
The study of how words are composed of morphemes (the smallest meaning-bearing units of a language)
Two broad classes of morphemes:
Stems: “main” morpheme of the word, supplying meaning
Affixes: Bits and pieces that combine with stems to modify their meanings and grammatical functions (prefixes, suffixes, circumfixes, infixes)
Unlike
Trying
Multiple affixes
Unreadable
Source: Joyce Choi, CSE 842, Michigan State University

14. Source: Joyce Choi, CSE 842, Michigan State University
Ways to Form Words
Inflection: new forms of the same word (usually in the same class)
Tense, number, mood, voice marking in verbs
Number, gender marking in nominals
Comparison of adjectives
Derivation: yield different words in different class
Deverbal nominals
Denominal adjectives and verbs
Compounding: new words out of two or more other words
Noun-noun compounding (e.g., doghouse)
Cliticization: combine a word with a clitic (which acts syntactically like a word but in a reduced form, e.g., I’ve)
Source: Joyce Choi, CSE 842, Michigan State University

15. English Inflectional Morphology
Word stem combines with grammatical morpheme
Usually produces word of same class
Usually serves a grammatical role that the stem could not (e.g. agreement)
like -> likes or liked
bird -> birds
Nouns have a simple inflectional morphology: markers for plural and markers for possessives
Verbs are slightly more complex:
Source: Joyce Choi, CSE 842, Michigan State University

16. Source: Joyce Choi, CSE 842, Michigan State University
Nominal Inflection
Nominal morphology
Plural forms
s or es
Irregular forms, e.g., Goose/Geese, Mouse/Mice
Possessives
children’s
Source: Joyce Choi, CSE 842, Michigan State University

17. Source: Joyce Choi, CSE 842, Michigan State University
Verbal Inflection
Main verbs (walk, like) are relatively regular
-s, ing, ed
And productive: ed, instant-messaged, faxed
But eat/ate/eaten, catch/caught/caught
Primary (be, have, do) and modal verbs (can, will, must) are often irregular and not productive
Be: am/is/are/were/was/been/being
Irregular verbs few (~250) but frequently occurring English verbal inflection is much simpler than e.g. Latin
Source: Joyce Choi, CSE 842, Michigan State University

18. Source: Joyce Choi, CSE 842, Michigan State University

19. English Derivational Morphology
Word stem combines with grammatical morpheme
Usually produces word of different class
More complicated than inflectional
Example: nominalization
-ize verbs -> -ation nouns
generalize, realize -> generalization, realization
Example: verbs, nouns -> adjectives
embrace, pity-. embraceable, pitiable
care, wit -> careless, witless
Source: Joyce Choi, CSE 842, Michigan State University

20. Source: Joyce Choi, CSE 842, Michigan State University
Example: adjective -> adverb
happy -> happily
More complicated to model than inflection
Less productive: *science-less, *concern-less, *go-able, *sleep-able
Meanings of derived terms harder to predict by rule
Source: Joyce Choi, CSE 842, Michigan State University

21. Morphological Analysis Tools
E.g. Porter Stemmer
A simple approach: just hack off the end of the word! Does NOT convert a word to its base form!!!
Frequently used in Information Retrieval, but results are pretty ugly!
Original *****************************
Rudolph Agnew , 55 years old and former chairman of
Consolidated Gold Fields PLC , was named a nonexecutive director of
this British industrial conglomerate . A form of asbestos once used to
make Kent cigarette filters has caused a high percentage of cancer
deaths among a group of workers exposed to it more than 30 years ago ,
Results *******************************
Rudolph Agnew , 55 year old and former chairman of
Consolid Gold Field PLC , wa name a nonexecut director of
thi British industri conglomer . A form of asbesto onc use to
make Kent cigarett filter ha caus a high percentag of cancer
death among a group of worker expos to it more than 30 year ago ,
Source: Marti Hearst, i256, at UC Berkeley

22. Stemming vs. Lemmatization
The purpose of both stemming and lemmatization is to reduce morphological variation.
Stemming reduces word-forms to (pseudo)stems, whereas lemmatization reduces the word-forms to linguistically valid lemmas (morphological stems).
Stemming: car, cars, car's, cars' => car
Lemmatizing: am, are, is => be ; drive, drives, drove, driven => drive
In a way, lemmatization deals only with inflectional variance, whereas stemming may also deal with derivational variance;

23. Is Stemming/Lemmatization Useful?
Both help reduce the size of vocabulary.
However problems…
Stemming can conflate semantically different words
E.g. “Gallery” and “gall” may both be stemmed to “gall”
Also truncated stems can be intelligible to users
Lemmatization is better, but it only deals with inflectional variance (e.g. “go”, “went”, “gone” => “go”, but not “attend”/verb, “attendance”/noun)
Despite the problems, stemming is done often in Information Retrieval (IR) and Text Mining.

24. Quiz!
The following pairs of words are stemmed to the same form by the Porter stemmer. Which pairs, would you agree, should NOT be conflated? Give your reasoning.
abandon / abandonment
marketing / markets
university / universe
volume / volumes
FYI: Porter Stemmer Online (
Introduction to Information Retrieval, C. Manning, P. Raghavan and H. Schutze, 2008

25. Source: Jurafsky & Martin “Speech and Language Processing”
POS Tagging
The process of assigning a part-of-speech or lexical class marker to each word in a sentence (and all sentences in a collection).
Input: the lead paint is unsafe
Output: the/Det lead/N paint/N is/V unsafe/Adj
Source: Jurafsky & Martin “Speech and Language Processing”

26. Why is POS Tagging Useful?
First step of a vast number of practical tasks
Helps in stemming/lemmatization
Parsing
Need to know if a word is an N or V before you can parse
Parsers can build trees directly on the POS tags instead of maintaining a lexicon
Information Extraction
Finding names, relations, etc.
Machine Translation
Selecting words of specific Parts of Speech (e.g. nouns) in pre-processing documents (for IR etc.)
Source: Jurafsky & Martin “Speech and Language Processing”

27. POS Tagging Choosing a Tagset
To do POS tagging, we need to choose a standard set of tags to work with
Could pick very coarse tagsets
N, V, Adj, Adv.
More commonly used set is finer grained, the “Penn TreeBank tagset”, 45 tags
PRP$, WRB, WP$, VBG
Even more fine-grained tagsets exist
Source: Jurafsky & Martin “Speech and Language Processing”

28. Penn TreeBank POS Tagset

29. Difficulties with POS Tagging
Words often have more than one POS – ambiguity:
The back door = JJ
On my back = NN
Win the voters back = RB
Promised to back the bill = VB
The POS tagging problem is to determine the POS tag for a particular instance of a word.
Another example of Part-of-speech ambiguities
NNP NNS NNS NNS CD NN VBZ VBZ VBZ VB
“Fed raises interest rates 0.5 % in effort to control inflation”
Source: Jurafsky & Martin “Speech and Language Processing”, Andrew McCallum, UMass Amherst

30. POS Tagging Techniques
Rule-based
Hand-coded rules
Probabilistic/Stochastic
Sequence (n-gram) models; machine learning
HMM (Hidden Markov Model)
MEMMs (Maximum Entropy Markov Models)
Transformation-based
Rules + n-gram machine learning
Brill tagger
Source: Jurafsky & Martin “Speech and Language Processing”, Andrew McCallum, UMass Amherst

31. Source: Andrew McCallum, UMass Amherst
Current Performance
Input: the lead paint is unsafe
Output: the/Det lead/N paint/N is/V unsafe/Adj
Using state-of-the-art automated method, how many tags are correct?
About 97% currently
But baseline is already 90%
Baseline is performance of simplest possible method: Tag every word with its most frequent tag, and Tag unknown words as nouns
Source: Andrew McCallum, UMass Amherst

32. Quiz!
Find one tagging error in each of the following sentences that are tagged with the Penn treebank tagset.
I/PRP need/VBP a/DT flight/NN from/IN Atlanta/NN.
Can/VBP you/PRP list/VB the/DT nonstop/JJ afternoon/NN flights/NNS ?
Tag each word in the following sentence with the Penn Treebank tagset.
“We may also collect information you voluntarily add to your profile, such as your mobile phone number and mobile service provider.”