1 CSC 594 Topics in AI – Text Mining and Analytics Fall 2015/16 2. Linguistic Essentials and Text Mining Preliminaries

The Description of Language Language = Words and Rules  Dictionary (vocabulary) + Grammar Dictionary –set of words defined in the language –open (dynamic) Grammar –set of rules which describe what is allowable in a language Classic/empirical Grammars –definitions and rules are mainly supported by examples –no (or almost no) formal description tools Explicit/formal Grammar (CFG, Dependency Grammars etc.) –formal description –can be programmed & tested on data (texts) 2

3 Levels of Language Analysis 1.Phonology study of sound systems of languages 2.Morphology study of structure of words: the structure of words in a language, including patterns of inflections and derivations 3.Syntax study of organization of words in sentences: the ordering of and relationship between the words in phrases and sentences 4.Semantics study of meaning in language: the study of how meaning in language is created 5.Pragmatics study of language in use: the branch of linguistics that studies language use rather than language structure 6.Discourse study of language, especially the type of language used in a particular context or subject 7.World Knowledge

Parts of Speech There are eight basic parts of speech for words in the English language: noun, pronoun, verb, adjective, adverb, preposition, conjunction, and interjection. The part of speech indicates how the word functions in meaning as well as grammatically within a sentence. 1.Noun: people, animals, concepts, things (e.g. “birds”) 2.Pronoun: a word used in place of a noun (e.g. “it”, “they”, “I”, “she”) 3.Verb: express action in the sentence (e.g. “sing”) 4.Adjective: describe properties of nouns (e.g. “yellow”) 5.Adverb: modifies or describes a verb, an adjective, or another adverb (e.g. “extremely”, “slowly”) 6.Preposition: a word placed before a noun/pronoun to form a phrase modifying another word/phrase (e.g. “in”, “for”, “without”) 4

Quiz! Identify all words which are the basic eight parts-of- speech in the following sentences. –The student put books on the table.” –“We may also collect information you voluntarily add to your profile, such as your mobile phone number and mobile service provider.” 5

6 Morphology The study of how words are composed of morphemes (the smallest meaning-bearing units of a language) Two broad classes of morphemes: –Stems: “main” morpheme of the word, supplying meaning –Affixes: Bits and pieces that combine with stems to modify their meanings and grammatical functions (prefixes, suffixes, circumfixes, infixes) Unlike Trying Multiple affixes – Unreadable Source: Joyce Choi, CSE 842, Michigan State University

7 Ways to Form Words Inflection: new forms of the same word (usually in the same class) –Tense, number, mood, voice marking in verbs –Number, gender marking in nominals –Comparison of adjectives Derivation: yield different words in different class –Deverbal nominals –Denominal adjectives and verbs Compounding: new words out of two or more other words –Noun-noun compounding (e.g., doghouse) Cliticization: combine a word with a clitic (which acts syntactically like a word but in a reduced form, e.g., I’ve) Source: Joyce Choi, CSE 842, Michigan State University

8 English Inflectional Morphology Word stem combines with grammatical morpheme –Usually produces word of same class –Usually serves a grammatical role that the stem could not (e.g. agreement) like -> likes or liked bird -> birds Nouns have a simple inflectional morphology: markers for plural and markers for possessives Verbs are slightly more complex: Source: Joyce Choi, CSE 842, Michigan State University

9 Nominal Inflection Nominal morphology –Plural forms s or es Irregular forms, e.g., Goose/Geese, Mouse/Mice –Possessives children’s Source: Joyce Choi, CSE 842, Michigan State University

10 Verbal Inflection Main verbs (walk, like) are relatively regular –-s, ing, ed –And productive: ed, instant-messaged, faxed –But eat/ate/eaten, catch/caught/caught Primary (be, have, do) and modal verbs (can, will, must) are often irregular and not productive –Be: am/is/are/were/was/been/being Irregular verbs few (~250) but frequently occurring English verbal inflection is much simpler than e.g. Latin Source: Joyce Choi, CSE 842, Michigan State University

11 Source: Joyce Choi, CSE 842, Michigan State University

12 English Derivational Morphology Word stem combines with grammatical morpheme –Usually produces word of different class –More complicated than inflectional Example: nominalization – -ize verbs -> -ation nouns – generalize, realize -> generalization, realization Example: verbs, nouns -> adjectives –embrace, pity-. embraceable, pitiable –care, wit -> careless, witless Source: Joyce Choi, CSE 842, Michigan State University

13 Example: adjective -> adverb –happy -> happily More complicated to model than inflection –Less productive: *science-less, *concern-less, *go-able, *sleep- able –Meanings of derived terms harder to predict by rule Source: Joyce Choi, CSE 842, Michigan State University

14 Morphological Analysis Tools E.g. Porter Stemmer –A simple approach: just hack off the end of the word! Does NOT convert a word to its base form!!! –Frequently used in Information Retrieval, but results are pretty ugly! Source: Marti Hearst, i256, at UC Berkeley Original ***************************** Rudolph Agnew, 55 years old and former chairman of Consolidated Gold Fields PLC, was named a nonexecutive director of this British industrial conglomerate. A form of asbestos once used to make Kent cigarette filters has caused a high percentage of cancer deaths among a group of workers exposed to it more than 30 years ago, Results ******************************* Rudolph Agnew, 55 year old and former chairman of Consolid Gold Field PLC, wa name a nonexecut director of thi British industri conglomer. A form of asbesto onc use to make Kent cigarett filter ha caus a high percentag of cancer death among a group of worker expos to it more than 30 year ago,

Stemming vs. Lemmatization The purpose of both stemming and lemmatization is to reduce morphological variation. Stemming reduces word-forms to (pseudo)stems, whereas lemmatization reduces the word-forms to linguistically valid lemmas (morphological stems). –Stemming: car, cars, car's, cars' => car –Lemmatizing: am, are, is => be ; drive, drives, drove, driven => drive In a way, lemmatization deals only with inflectional variance, whereas stemming may also deal with derivational variance; 15

Is Stemming/Lemmatization Useful? Both help reduce the size of vocabulary. However problems… –Stemming can conflate semantically different words E.g. “Gallery” and “gall” may both be stemmed to “gall” –Also truncated stems can be intelligible to users –Lemmatization is better, but it only deals with inflectional variance (e.g. “go”, “went”, “gone” => “go”, but not “attend”/verb, “attendance”/noun) Despite the problems, stemming is done often in Information Retrieval (IR) and Text Mining. 16

Quiz! The following pairs of words are stemmed to the same form by the Porter stemmer. Which pairs, would you agree, should NOT be conflated? Give your reasoning. –abandon / abandonment –marketing / markets –university / universe –volume / volumes FYI: Porter Stemmer Online ( ) Introduction to Information Retrieval, C. Manning, P. Raghavan and H. Schutze,

18 POS Tagging The process of assigning a part-of-speech or lexical class marker to each word in a sentence (and all sentences in a collection). Input: the lead paint is unsafe Output: the/Det lead/N paint/N is/V unsafe/Adj Source: Jurafsky & Martin “Speech and Language Processing”

19 Why is POS Tagging Useful? First step of a vast number of practical tasks Helps in stemming/lemmatization Parsing –Need to know if a word is an N or V before you can parse –Parsers can build trees directly on the POS tags instead of maintaining a lexicon Information Extraction –Finding names, relations, etc. Machine Translation Selecting words of specific Parts of Speech (e.g. nouns) in pre-processing documents (for IR etc.) Source: Jurafsky & Martin “Speech and Language Processing”

20 POS Tagging Choosing a Tagset To do POS tagging, we need to choose a standard set of tags to work with Could pick very coarse tagsets –N, V, Adj, Adv. More commonly used set is finer grained, the “Penn TreeBank tagset”, 45 tags –PRP$, WRB, WP$, VBG Even more fine-grained tagsets exist Source: Jurafsky & Martin “Speech and Language Processing”

21 Penn TreeBank POS Tagset

22 Difficulties with POS Tagging Words often have more than one POS – ambiguity: –The back door = JJ –On my back = NN –Win the voters back = RB –Promised to back the bill = VB The POS tagging problem is to determine the POS tag for a particular instance of a word. Another example of Part-of-speech ambiguities NNP NNS NNS NNS CD NN VBZ VBZ VBZ VB “Fed raises interest rates 0.5 % in effort to control inflation” Source: Jurafsky & Martin “Speech and Language Processing”, Andrew McCallum, UMass Amherst

23 POS Tagging Techniques Source: Jurafsky & Martin “Speech and Language Processing”, Andrew McCallum, UMass Amherst 1.Rule-based Hand-coded rules 2.Probabilistic/Stochastic Sequence (n-gram) models; machine learning  HMM (Hidden Markov Model)  MEMMs (Maximum Entropy Markov Models) 3.Transformation-based Rules + n-gram machine learning  Brill tagger

24 Current Performance Input: the lead paint is unsafe Output: the/Det lead/N paint/N is/V unsafe/Adj Using state-of-the-art automated method, how many tags are correct? –About 97% currently –But baseline is already 90% Baseline is performance of simplest possible method: Tag every word with its most frequent tag, and Tag unknown words as nouns Source: Andrew McCallum, UMass Amherst

Quiz! 1.Find one tagging error in each of the following sentences that are tagged with the Penn treebank tagset. a.I/PRP need/VBP a/DT flight/NN from/IN Atlanta/NN. b.Can/VBP you/PRP list/VB the/DT nonstop/JJ afternoon/NN flights/NNS ? 2.Tag each word in the following sentence with the Penn Treebank tagset. –“We may also collect information you voluntarily add to your profile, such as your mobile phone number and mobile service provider.” 25

26 Named Entity Recognition (NER) Named Entities (NEs) are proper names in texts, i.e. the names of persons, organizations, locations, times and quantities NE Recognition (NER) is a sub-task of Information Extraction (IE) NER is to process a text and identify named entities –e.g. “U.N. official Ekeus heads for Baghdad.” NER is also an important task for texts in specific domains such as biomedical texts Source: J. Choi, CSE842, MSU; Marti Hearst, i256, at UC Berkeley

27 Common Entity Types NE TypeExamples ORGANIZATIONGeorgia-Pacific Corp., WHO PERSONEddy Bonte, President Obama LOCATIONMurray River, Mount Everest DATEJune, TIMEtwo fifty a m, 1:30 p.m. MONEY175 million Canadian Dollars, GBP PERCENTtwenty pct, % FACILITYWashington Monument, Stonehenge GPE (geo political entity)South East Asia, Midlothian

28 Difficulties with NER Names are too numerous to include in dictionaries Variations –e.g. “John Smith”, “Mr Smith”, “John” Changing constantly –new names invent unknown words Ambiguities –Same name refers to different entities, e.g. JFK – the former president JFK – his son JFK – airport in NY Multi-word entities – difficult to find boundaries –“DePaul University” –“Cecil H. Green Library and Escondido Village Conference Service Center”

29 Landscape of IE/NER Techniques Any of these models can be used to capture words, formatting or both. Lexicons Alabama Alaska … Wisconsin Wyoming Abraham Lincoln was born in Kentucky. member? Classify Pre-segmented Candidates Abraham Lincoln was born in Kentucky. Classifier which class? Sliding Window Abraham Lincoln was born in Kentucky. Classifier which class? Try alternate window sizes: Boundary Models Abraham Lincoln was born in Kentucky. Classifier which class? BEGINENDBEGINEND BEGIN Context Free Grammars Abraham Lincoln was born in Kentucky. NNPVPNPVNNP NP PP VP S Most likely parse? Finite State Machines Abraham Lincoln was born in Kentucky. Most likely state sequence? Source: Marti Hearst, i256, at UC Berkeley

30 NER State of the Art Performance Named entity recognition –Person, Location, Organization, … –F1 score (similar to accuracy) in high 80’s or low- to mid-90’s However, performance depends on the entity types [Wikipedia] At least two hierarchies of named entity types have been proposed in the literature. BBN categories [1], proposed in 2002, is used for Question Answering and consists of 29 types and 64 subtypes. Sekine's extended hierarchy [2], proposed in 2002, is made of 200 subtypes.hierarchiesBBN[1]Question Answering[2] Also, various domains use different entity types (e.g. concepts in biomedical texts)

Is NER Useful? Yes, especially when the text uses domain-specific vocabulary (e.g. legal, medical). 31

UIC - CS 594 Bing Liu Stop words Many of the most frequently used words in English are worthless in IR and text mining – these words are called stop words. –the, of, and, to, …. –Typically about 400 to 500 such words –For an application, an additional domain specific stop words list may be constructed Why do we want to remove stop words? –Reduce indexing (or data) file size stopwords accounts 20-30% of total word counts. –Improve efficiency stop words are not useful for searching or text mining stop words always have a large number of hits Example Stopword ListsStopword Lists

Difficulties with Stopwords Though stop words usually refer to the most common words in a language, there is no single universal list of stop words used by all processing of natural language tools, and indeed not all tools even use such a list. (Wikipedia) 33