Introduction to Classification Shallow Processing Techniques for NLP Ling570 November 9, 2011

Roadmap Classification problems: Definition Solutions Case studies Based on slides by F. Xia

Example: Text Classification Task: Given an article, predict its category Categories:

Example: Text Classification Task: Given an article, predict its category Categories: Sports, entertainment, news, weather,.. Spam/not spam

Example: Text Classification Task: Given an article, predict its category Categories: Sports, entertainment, news, weather,.. Spam/not spam What kind of information is useful for this task?

Classification Task Task: C is a finite set of labels (aka categories, classes) Given x, determine its category y in C

Classification Task Task: C is a finite set of labels (aka categories, classes) Given x, determine its category y in C Instance: (x,y) x: thing to be labeled/classified y: label/class

Classification Task Task: C is a finite set of labels (aka categories, classes) Given x, determine its category y in C Instance: (x,y) x: thing to be labeled/classified y: label/class Data: set of instances labeled data: y is known unlabeled data: y is unknown

Classification Task Task: C is a finite set of labels (aka categories, classes) Given x, determine its category y in C Instance: (x,y) x: thing to be labeled/classified y: label/class Data: set of instances labeled data: y is known unlabeled data: y is unknown Training data, test data

Text Classification Examples Spam filtering Call routing Sentiment classification Positive/Negative Score: 1 to 5

POS Tagging Task: Given a sentence, predict tag of each word Is this a classification problem?

POS Tagging Task: Given a sentence, predict tag of each word Is this a classification problem? Categories: N, V, Adj,… What information is useful?

POS Tagging Task: Given a sentence, predict tag of each word Is this a classification problem? Categories: N, V, Adj,… What information is useful? How do POS tagging, text classification differ?

POS Tagging Task: Given a sentence, predict tag of each word Is this a classification problem? Categories: N, V, Adj,… What information is useful? How do POS tagging, text classification differ? Sequence labeling problem

Word Segmentation Task: Given a string, break into words Categories:

Word Segmentation Task: Given a string, break into words Categories: B(reak), NB (no break) B(eginning), I(nside), E(nd) e.g. c1 c2 || c3 c4 c5

Word Segmentation Task: Given a string, break into words Categories: B(reak), NB (no break) B(eginning), I(nside), E(nd) e.g. c1 c2 || c3 c4 c5 c1/NB c2/B c3/NB c4/NB c5/B c1/B c2/E c3/B c4/I c5/E What type of task?

Word Segmentation Task: Given a string, break into words Categories: B(reak), NB (no break) B(eginning), I(nside), E(nd) e.g. c1 c2 || c3 c4 c5 c1/NB c2/B c3/NB c4/NB c5/B c1/B c2/E c3/B c4/I c5/E What type of task? Also sequence labeling

Solving a Classification Problem

Two Stages Training: Learner: training data  classifier

Two Stages Training: Learner: training data  classifier Testing: Decoder: test data + classifier  classification output

Two Stages Training: Learner: training data  classifier Testing: Decoder: test data + classifier  classification output Also Preprocessing Postprocessing Evaluation

Representing Input Potentially infinite values to represent

Representing Input Potentially infinite values to represent Represent input as feature vector x=

Representing Input Potentially infinite values to represent Represent input as feature vector x= What are good features?

Example I Spam Tagging Classes: Spam/Not Spam Input: messages

Doc1 Western Union Money Transfer One Bishops Square Akpakpa E1 6AO, Cotonou Benin Republic Website: info/selectCountry.asP Phone: Attention Beneficiary, This to inform you that the federal ministry of finance Benin Republic has started releasing scam victim compensation fund mandated by United Nation Organization through our office. I am contacting you because our agent have sent you the first payment of $5,000 for your compensation funds total amount of $ USD (Five hundred thousand united state dollar) We need your urgent response so that we shall release your payment information to you. You can call our office hot line for urgent attention( )

Doc2 Hello! my dear. How are you today and your family? I hope all is good, kindly pay Attention and understand my aim of communicating you today through this Letter, My names is Saif al-Islam al-Gaddafi the Son of former Libyan President. i was born on 1972 in Tripoli Libya,By Gaddafi’s second wive. I want you to help me clear this fund in your name which i deposited in Europe please i would like this money to be transferred into your account before they find it. the amount is ,000 million GBP British Pounds sterling through a

Doc3 from: Apply for loan at 3% interest Rate..Contact us for details.

Doc4 from: REMINDER: If you have not received a PIN number to vote in the elections and have not already contacted us, please contact either Drago Radev or Priscilla Rasmussen right away. Everyone who has not received a pin but who has contacted us already will get a new pin over the weekend. Anyone who still wants to join for 2011 needs to do this by Monday (November 7th) in order to be eligible to vote. And, if you do have your PIN number and have not voted yet, remember every vote

What are good features?

Possible Features Words!

Possible Features Words! Feature for each word

Possible Features Words! Feature for each word Binary: presence/absence Integer: occurrence count Particular word types: money/sex/: [Vv].*gr.*

Possible Features Words! Feature for each word Binary: presence/absence Integer: occurrence count Particular word types: money/sex/: [Vv].*gr.* Errors: Spelling, grammar

Possible Features Words! Feature for each word Binary: presence/absence Integer: occurrence count Particular word types: money/sex/: [Vv].*gr.* Errors: Spelling, grammar Images

Possible Features Words! Feature for each word Binary: presence/absence Integer: occurrence count Particular word types: money/sex/: [Vv].*gr.* Errors: Spelling, grammar Images Header info

Representing Input: Attribute-Value Matrix f 1 Currency f 2 Country …f m Date Label x 1 = Doc1 x 2 =Doc2.. x n =Doc4

Representing Input: Attribute-Value Matrix f 1 Currency f 2 Country …f m Date Label x 1 = Doc1Spam x 2 =Doc2Spam.. x n =Doc4NotSpam

Representing Input: Attribute-Value Matrix f 1 Currency f 2 Country …f m Date Label x 1 = Doc Spam x 2 =Doc2Spam.. x n =Doc4NotSpam

Representing Input: Attribute-Value Matrix f 1 Currency f 2 Country …f m Date Label x 1 = Doc Spam x 2 =Doc Spam.. x n =Doc4NotSpam

Representing Input: Attribute-Value Matrix f 1 Currency f 2 Country …f m Date Label x 1 = Doc Spam x 2 =Doc Spam.. x n =Doc40002NotSpam

Classifier Result of training on input data With or without class labels

Classifier Result of training on input data With or without class labels Formal perspective: f(x) =y: x is input; y in C

Classifier Result of training on input data With or without class labels Formal perspective: f(x) =y: x is input; y in C More generally: f(x)={(c i,score i )}, where x is input, c i in C, score i is score for category assignment

Testing Input: Test data: e.g. AVM Classifier Output:

Testing Input: Test data: e.g. AVM Classifier Output: Decision matrix Can assign highest scoring class to each input

Testing Input: Test data: e.g. AVM Classifier Output: Decision matrix Can assign highest scoring class to each input x1x1 x2x2 x3x3 …. c1c … c2c … c3c … c 4 … …

Testing Input: Test data: e.g. AVM Classifier Output: Decision matrix Can assign highest scoring class to each input x1x1 x2x2 x3x3 …. c1c … c2c … c3c … c 4 … … x1x1 x2x2 x3x3 c4c4 c2c2 c3c3

Evaluation Confusion matrix: Precision: TP/(TP+FP) Gold System TP FP - FN TN

Evaluation Confusion matrix: Precision: TP/(TP+FP) Recall: TP/(TP+FN) Gold System TP FP - FN TN

Evaluation Confusion matrix: Precision: TP/(TP+FP) Recall: TP/(TP+FN) F-score: 2PR/(P+R) Gold System TP FP - FN TN

Evaluation Confusion matrix: Precision: TP/(TP+FP) Recall: TP/(TP+FN) F-score: 2PR/(P+R) Accuracy = (TP+TN)/(TP+TN+FP+TN) Gold System TP FP - FN TN

Evaluation Confusion matrix: Precision: TP/(TP+FP) Recall: TP/(TP+FN) F-score: 2PR/(P+R) Accuracy = (TP+TN)/(TP+TN+FP+TN) Why F-score? Accuracy? Gold System TP FP - FN TN

Evaluation Example Confusion matrix: Precision: 1/(1+4)=1/5 Recall: TP/(TP+FN)=1/6 F-score: 2PR/(P+R)=2*1/5*1/6/(1/5+1/6)=2/11 Accuracy = 91% Gold System

Evaluation Example Confusion matrix: Precision: Gold System

Evaluation Example Confusion matrix: Precision: 1/(1+4)=1/5 Recall: TP/(TP+FN) Gold System

Evaluation Example Confusion matrix: Precision: 1/(1+4)=1/5 Recall: TP/(TP+FN)=1/6 F-score: 2PR/(P+R)= Gold System

Evaluation Example Confusion matrix: Precision: 1/(1+4)=1/5 Recall: TP/(TP+FN)=1/6 F-score: 2PR/(P+R)=2*1/5*1/6/(1/5+1/6)=2/11 Accuracy = 91% Gold System

Classification Problem Steps Input processing: Split data into training/dev/test

Classification Problem Steps Input processing: Split data into training/dev/test Convert data into an Attribute-Value Matrix Identify candidate features Perform feature selection Create AVM representation

Classification Problem Steps Input processing: Split data into training/dev/test Convert data into an Attribute-Value Matrix Identify candidate features Perform feature selection Create AVM representation Training

Classification Problem Steps Input processing: Split data into training/dev/test Convert data into an Attribute-Value Matrix Identify candidate features Perform feature selection Create AVM representation Training Testing Evaluation

Classification Algorithms Will be covered in detail in 572 Nearest Neighbor Naïve Bayes Decision Trees Neural Networks Maximum Entropy

Feature Design & Representation What sorts of information do we want to encode?

Feature Design & Representation What sorts of information do we want to encode? words, frequencies, ngrams, morphology, sentence length, etc Issue

Feature Design & Representation What sorts of information do we want to encode? words, frequencies, ngrams, morphology, sentence length, etc Issue: Learning algorithms work on numbers Many work only on binary values (0/1)

Feature Design & Representation What sorts of information do we want to encode? words, frequencies, ngrams, morphology, sentence length, etc Issue: Learning algorithms work on numbers Many work only on binary values (0/1) Others work on any real-valued input How can we represent different information Numerically? Binary?

Representation Words/tags/ngrams/etc

Representation Words/tags/ngrams/etc One feature per item:

Representation Words/tags/ngrams/etc One feature per item: Binary: presence/absence Real: counts Binarizing numeric features:

Representation Words/tags/ngrams/etc One feature per item: Binary: presence/absence Real: counts Binarizing numeric features: Single threshold Multiple thresholds Binning: 1 binary feature/bin

Feature Template Example: Prevword (or w -1 )

Feature Template Example: Prevword (or w -1 ) Template corresponds to many features e.g. time flies like an arrow

Feature Template Example: Prevword (or w -1 ) Template corresponds to many features e.g. time flies like an arrow w -1 = w -1 =time w -1 =flies w -1 =like w -1 =an…

Feature Template Example: Prevword (or w -1 ) Template corresponds to many features e.g. time flies like an arrow w -1 = w -1 =time w -1 =flies w -1 =like w -1 =an… Shorthand for: w -1 = 0 or w -1 =time 1

AVM Example Time flies like an arrow Note: this is a compact form of the true sparse vector w -1 =w 0 or 1, for w in |V| w -1 w0w0 w -1 w 0 w +1 label x1 x2 x3

AVM Example Time flies like an arrow Note: this is a compact form of the true sparse vector w -1 =w 0 or 1, for w in |V| w -1 w0w0 w -1 w 0 w +1 label x1 Time fliesN x2 x3

AVM Example Time flies like an arrow Note: this is a compact form of the true sparse vector w -1 =w 0 or 1, for w in |V| w -1 w0w0 w -1 w 0 w +1 label x1 Time fliesN x2TimefliesTime flieslikeV x3

AVM Example Time flies like an arrow w -1 w0w0 w -1 w 0 w +1 label x1 Time fliesN x2TimefliesTime flieslikeV x3flieslikeflies likeanP

AVM Example Time flies like an arrow Note: this is a compact form of the true sparse vector w -1 =w 0 or 1, for w in |V| w -1 w0w0 w -1 w 0 w +1 label x1 Time fliesN x2TimefliesTime flieslikeV x3flieslikeflies likeanP

Example: NER Named Entity tagging: John visited New York last Friday  [person John] visited [location New York] [time last Friday] As a classification problem

Example: NER Named Entity tagging: John visited New York last Friday  [person John] visited [location New York] [time last Friday] As a classification problem John/PER-B visited/O New/LOC-B York/LOC-I last/TIME-B Friday/TIME-I

Example: NER Named Entity tagging: John visited New York last Friday  [person John] visited [location New York] [time last Friday] As a classification problem John/PER-B visited/O New/LOC-B York/LOC-I last/TIME-B Friday/TIME-I Input?

Example: NER Named Entity tagging: John visited New York last Friday  [person John] visited [location New York] [time last Friday] As a classification problem John/PER-B visited/O New/LOC-B York/LOC-I last/TIME-B Friday/TIME-I Input? Categories?

Example: Coreference Queen Elizabeth set about transforming her husband, King George VI, into a viable monarch. Logue, a renowned speech therapist, was summoned to help the King overcome his speech impediment...

Example: Coreference Queen Elizabeth set about transforming her husband, King George VI, into a viable monarch. Logue, a renowned speech therapist, was summoned to help the King overcome his speech impediment... Can be viewed as a classification problem

Example: Coreference Queen Elizabeth set about transforming her husband, King George VI, into a viable monarch. Logue, a renowned speech therapist, was summoned to help the King overcome his speech impediment... Can be viewed as a classification problem What are the inputs?

Example: Coreference Queen Elizabeth set about transforming her husband, King George VI, into a viable monarch. Logue, a renowned speech therapist, was summoned to help the King overcome his speech impediment... Can be viewed as a classification problem What are the inputs? What are the categories?

Example: Coreference Queen Elizabeth set about transforming her husband, King George VI, into a viable monarch. Logue, a renowned speech therapist, was summoned to help the King overcome his speech impediment... Can be viewed as a classification problem What are the inputs? What are the categories? What features would be useful?

HW#7 Viterbi! Implement Viterbi algorithm Use a standard, precomputed, presmoothed model Made available after HW#6 handed in Testing & Evaluation Convert output format to enable comparison w/gold Compare to gold-standard and produce a score