Presentation is loading. Please wait.

Presentation is loading. Please wait.

A Comparison of Event Models for Naïve Bayes Text Classification Andrew McCallumKamal Nigam Just Research and Carnegie Mellon University.

Published byConrad Powers Modified over 9 years ago

Similar presentations

Presentation on theme: "A Comparison of Event Models for Naïve Bayes Text Classification Andrew McCallumKamal Nigam Just Research and Carnegie Mellon University."— Presentation transcript:

1 A Comparison of Event Models for Naïve Bayes Text Classification Andrew McCallumKamal Nigam Just Research and Carnegie Mellon University

2 Naïve Bayes for Text Commonly and successfully used Two implementations with different formulations: multi-variate Bernoulli & Multinomial

3 Documents are binary word vectors Generative model: biased coin flips Used by: –[Robertson & Sparck-Jones 76] –[Lewis 92] –[Kalt & Croft 96] –[Larkey & Croft 96] –[Sahami 96] –[Koller & Sahami 97] cable corn damp drawer dropped Multi-variate Bernoulli activity

4 Multinomial Documents have word occurrence counts Generative model: urn drawing Used by: –[Guthrie & Walker 94] –[Lewis & Gale 94] –[Kalt & Croft 96] –[Li & Yamanishi 97] –[Mitchell 97] –[McCallum et al. 98] –[Nigam et al. 98] corn activity dropped maize in while

5 Classifier Representation Multi-variate BernoulliMultinomial

6 Building Classifiers Multi-variate Bernoulli Corn tastes good. Multinomial Eat corn; drink corn. Maize is pretty.

7 Document Representation Corn prices rose today while corn futures dropped in surprising trading activity. Corn... Multinomial Multi-variate Bernoulli

8 Classification with Bayes’ Rule Multi-variate Bernoulli -Binary over all words Multinomial -Evidence from occurring words

9 Feature Selection Select words with highest average mutual information with class: Multi-variate Bernoulli: over document events. Multinomial: over word events. Corn tastes good. Eat corn; drink corn. Maize is pretty. Corntastes good. Eatcorn; drinkcorn. Oil is expensive. Candy corn is sweet. Candy Oil is expensive. cornis sweet.

10 Experiments 5 domains; between 2-100 classes Vary vocabulary using mutual information Compare Multi-variate Bernoulli and Multinomial

11 Industry Sector Hierarchy 71 classes; 6500 documents

12 Yahoo ‘Science’ Hierarchy 95 classes; 13500 documents

13 Twenty Newsgroups 20 classes; 20000 documents

14 WebKB Homepages 4 classes; 4200 documents

15 Reuters-21578 Categories interestmoney-fxship Binary classification; 12900 documents

16 Discussion Multinomial better for tasks needing large vocabularies Vocabulary size matters Multinomial accounts for amount of evidence (number of words) Limited dependencies easier with Multi-variate Bernoulli Take care when adding non-text features to Multinomial

17 Naïve Bayes Classification Simplistic independence assumption Surprisingly good results in many domains

18 corn maize activity

Download ppt "A Comparison of Event Models for Naïve Bayes Text Classification Andrew McCallumKamal Nigam Just Research and Carnegie Mellon University."

Similar presentations

Classification Classification Examples

Classification Classification Examples
Naïve Bayes Advanced Statistical Methods in NLP Ling572 January 19, 2012 1.

Naïve Bayes Advanced Statistical Methods in NLP Ling572 January 19,
Information Retrieval Models: Probabilistic Models

Information Retrieval Models: Probabilistic Models
Merging Taxonomies. Assertion Creation and maintenance of large ontologies will require the capability to merge taxonomies This problem is similar to.

Merging Taxonomies. Assertion Creation and maintenance of large ontologies will require the capability to merge taxonomies This problem is similar to.
Optimizing Text Classification Mark Trenorden Supervisor: Geoff Webb.

Optimizing Text Classification Mark Trenorden Supervisor: Geoff Webb.
Distributional Clustering of Words for Text Classification Authors: L.Douglas Baker Andrew Kachites McCallum Presenter: Yihong Ding.

Distributional Clustering of Words for Text Classification Authors: L.Douglas Baker Andrew Kachites McCallum Presenter: Yihong Ding.
Logistic Regression Rong Jin. Logistic Regression Model  In Gaussian generative model:  Generalize the ratio to a linear model Parameters: w and c.

Logistic Regression Rong Jin. Logistic Regression Model  In Gaussian generative model:  Generalize the ratio to a linear model Parameters: w and c.
An Overview of Text Mining Rebecca Hwa 4/25/2002 References M. Hearst, “Untangling Text Data Mining,” in the Proceedings of the 37 th Annual Meeting of.

An Overview of Text Mining Rebecca Hwa 4/25/2002 References M. Hearst, “Untangling Text Data Mining,” in the Proceedings of the 37 th Annual Meeting of.
Distributional Clustering of Words for Text Classification L. Douglas Baker Andrew Kachites McCallum SIGIR’98.

Distributional Clustering of Words for Text Classification L. Douglas Baker Andrew Kachites McCallum SIGIR’98.
ECOC for Text Classification Hybrids of EM & Co-Training (with Kamal Nigam) Learning to build a monolingual corpus from the web (with Rosie Jones) Effect.

ECOC for Text Classification Hybrids of EM & Co-Training (with Kamal Nigam) Learning to build a monolingual corpus from the web (with Rosie Jones) Effect.
Text Classification from Labeled and Unlabeled Documents using EM Kamal Nigam Andrew K. McCallum Sebastian Thrun Tom Mitchell Machine Learning (2000) Presented.

Text Classification from Labeled and Unlabeled Documents using EM Kamal Nigam Andrew K. McCallum Sebastian Thrun Tom Mitchell Machine Learning (2000) Presented.
Distributional Clustering of Words for Text Classification Presentation by: Thomas Walsh (Rutgers University) L.Douglas Baker (Carnegie Mellon University)

Distributional Clustering of Words for Text Classification Presentation by: Thomas Walsh (Rutgers University) L.Douglas Baker (Carnegie Mellon University)
Using Error-Correcting Codes For Text Classification Rayid Ghani Center for Automated Learning & Discovery, Carnegie Mellon University.

Using Error-Correcting Codes For Text Classification Rayid Ghani Center for Automated Learning & Discovery, Carnegie Mellon University.
Co-training LING 572 Fei Xia 02/21/06. Overview Proposed by Blum and Mitchell (1998) Important work: –(Nigam and Ghani, 2000) –(Goldman and Zhou, 2000)

Co-training LING 572 Fei Xia 02/21/06. Overview Proposed by Blum and Mitchell (1998) Important work: –(Nigam and Ghani, 2000) –(Goldman and Zhou, 2000)
Using Error-Correcting Codes For Text Classification Rayid Ghani This presentation can be accessed at

Using Error-Correcting Codes For Text Classification Rayid Ghani This presentation can be accessed at
HypertextHypertext Categorization Rayid Ghani IR Seminar - 10/3/00.

HypertextHypertext Categorization Rayid Ghani IR Seminar - 10/3/00.
Multi-view Exploratory Learning for AKBC Problems Bhavana Dalvi and William W. Cohen School Of Computer Science, Carnegie Mellon University Motivation.

Multi-view Exploratory Learning for AKBC Problems Bhavana Dalvi and William W. Cohen School Of Computer Science, Carnegie Mellon University Motivation.
Naïve Bayes Classification Debapriyo Majumdar Data Mining – Fall 2014 Indian Statistical Institute Kolkata August 14, 2014.

Naïve Bayes Classification Debapriyo Majumdar Data Mining – Fall 2014 Indian Statistical Institute Kolkata August 14, 2014.
CHAPTER 4: Parametric Methods. Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 Parametric Estimation X = {

CHAPTER 4: Parametric Methods. Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 Parametric Estimation X = {
CHAPTER 4: Parametric Methods. Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 Parametric Estimation Given.

CHAPTER 4: Parametric Methods. Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 Parametric Estimation Given.

Similar presentations

Ads by Google