Text Classification Eric Doi Harvey Mudd College November 20th, 2008

Kinds of Classification  Language Hello. My name is Eric. Hola. Mi nombre es Eric. こんにちは。 私の名前はエリックである. 你好。 我叫 Eric 。

Kinds of Classification  Type “approaches based on n-grams obtain generalization by concatenating”* To: Subject: McCain and Obama use it too You have received this message because you opted in to receives Sund Design pecial offers via . Login to your member account to edit your subscription. Click here to unsubscribe. ACAAGATGCCATTGTCCCCCGGCCTCCTG *(Bengio)

Difficulties  Dictionary? Generalization?  Over 500,000 words in English language (and over one million if counting scientific words)  Typos/OCR errors  Loan words We practice ballet at the café. Nous pratiquons le ballet au café.

Approaches  Unique letter combinations LanguageString English “ery” French “eux” Gaelic “mh” Italian “cchi“ Dunning, Statistical Identification of Language

Approaches  “Unique” letter combinations LanguageString English “ery” French “milieux” Gaelic “farmhand” Italian “zucchini“ Dunning, Statistical Identification of Language

Approaches  “Unique” letter combinations LanguageString English “ery” French “milieux” Gaelic “farmhand” Italian “zucchini“  Requires hand-coding; what about other languages (6000+)? Dunning, Statistical Identification of Language

Approaches  Try to minimize: Hand-coded knowledge Training data Input data (isolating phrases?)  Dunning, “Statistical Identification of Language.”  Bengio, “A Neural Probabilistic Language Model.” 2003.

Statistical Approach: N-Grams  N-grams are sequences of n elements Professor Keller is not a goth. Word-level bigrams: Char-level trigrams:

Statistical Approach: N-Grams  N-grams are sequences of n elements Professor Keller is not a goth. Word-level bigrams: (Professor, Keller) Char-level trigrams:

Statistical Approach: N-Grams  N-grams are sequences of n elements Professor Keller is not a goth. Word-level bigrams: (Keller, is) Char-level trigrams:

Statistical Approach: N-Grams  N-grams are sequences of n elements Professor Keller is not a goth. Word-level bigrams: (is, not) Char-level trigrams:

Statistical Approach: N-Grams  N-grams are sequences of n elements Professor Keller is not a goth. Word-level bigrams: (not, a) Char-level trigrams:

Statistical Approach: N-Grams  N-grams are sequences of n elements Professor Keller is not a goth. Word-level bigrams: (a, goth) Char-level trigrams:

Statistical Approach: N-Grams  N-grams are sequences of n elements Professor Keller is not a goth. Word-level bigrams: Char-level trigrams:

Statistical Approach: N-Grams  N-grams are sequences of n elements Professor Keller is not a goth. Word-level bigrams: Char-level trigrams: (P, r, o)

Statistical Approach: N-Grams  N-grams are sequences of n elements Professor Keller is not a goth. Word-level bigrams: Char-level trigrams: (r, o, f)

Statistical Approach: N-Grams  N-grams are sequences of n elements Professor Keller is not a goth. Word-level bigrams: Char-level trigrams: (o, f, e)

Statistical Approach: N-Grams  N-grams are sequences of n elements Professor Keller is not a goth. Word-level bigrams: Char-level trigrams: (f, e, s)

Statistical Approach: N-Grams  N-grams are sequences of n elements Professor Keller is not a goth. Word-level bigrams: Char-level trigrams:

Statistical Approach: N-Grams  Mined from 1,024,908,267,229 words  Sample 4-grams serve as the infrastructure 500 serve as the initial 5331 serve as the injector 56

Statistical Approach: N-Grams  Informs some notion of probability Normalize frequencies P(serve as the initial) > P(serve as the injector) Classification P(English | serve as the initial) > P(Spanish | serve as the initial) P(Spam | serve as the injector) < P(!Spam | serve as the injector)

Statistical Approach: N-Grams  But what about P(serve as the ink)? = 0? P(serve as the ink) = P(vxvw aooa *%^$) = 0? How about P(sevre as the initial)?

Statistical Approach: N-Grams  How do we smooth out sparse data?  Additive smoothing  Interpolation  Good-Turing estimate  Backoff  Witten-Bell smoothing  Absolute discounting  Kneser-Key smoothing MacCartney

Statistical Approach: N-Grams  Additive smoothing  Interpolation- consider smaller n-grams as well, e.g. (serve as the), (serve)  Backoff- use interpolation only if necessary MacCartney

Statistical Approach: Results  Dunning: Compared parallel translated texts in English and Spanish 20 char input, 50K training: 92% accurate 500 char input, 50K training: 99.9%  Modified for comparing DNA sequences of Humans, E-Coli, and Yeast

Neural Network Approach Bengio et. al, “A Neural Probabilistic Language Model.” 2003:  N-gram does handle sparse data well  However, there are problems: Narrow consideration of context (~1–2 words) Does not consider semantic/grammatical similarity: “A cat is walking in the bedroom” “A dog was running in a room”

Neural Network Approach  The general idea: 1. Associate with each word in the vocabulary (e.g. size 17,000) a feature vector (30–100 features) 2. Express the joint probability function of word sequences in terms of feature vectors 3. Learn simultaneously the word feature vectors and the parameters of the probability function

References  Dunning, “Statistical Identification of Language.”  Bengio, “A Neural Probabilistic Language Model.”  MacCartney, “NLP Lunch Tutorial: Smoothing.” 2005.