Presentation is loading. Please wait.

Presentation is loading. Please wait.

Using Error-Correcting Codes For Text Classification Rayid Ghani This presentation can be accessed at

Published by Modified over 9 years ago

Similar presentations

Presentation on theme: "Using Error-Correcting Codes For Text Classification Rayid Ghani This presentation can be accessed at"— Presentation transcript:

1 Using Error-Correcting Codes For Text Classification Rayid Ghani rayid@cs.cmu.edu This presentation can be accessed at http://www.cs.cmu.edu/~rayid/talks/

2 Outline Introduction to ECOC Intuition & Motivation Some Questions? Experimental Results Semi-Theoretical Model Types of Codes Drawbacks Conclusions

3 Introduction Decompose a multiclass classification problem into multiple binary problems One-Per-Class Approach (moderately expensive) All-Pairs (very expensive) Distributed Output Code (efficient but what about performance?) Error-Correcting Output Codes (?)

4

5 Is it a good idea? Larger margin for error since errors can now be “corrected” One-per-class is a code with minimum hamming distance (HD) = 2 Distributed codes have low HD The individual binary problems can be harder than before Useless unless number of classes > 5

6 Training ECOC Given m distinct classes Create an m x n binary matrix M. Each class is assigned ONE row of M. Each column of the matrix divides the classes into TWO groups. Train the Base classifiers to learn the n binary problems.

7 Testing ECOC To test a new instance Apply each of the n classifiers to the new instance Combine the predictions to obtain a binary string(codeword) for the new point Classify to the class with the nearest codeword (usually hamming distance is used as the distance measure)

8 ECOC - Picture 0 0 1 1 0 1 0 1 0 0 0 1 1 1 0 0 1 0 0 1 ABCDABCD A D C B f 1 f 2 f 3 f 4 f 5

9 ECOC - Picture 0 0 1 1 0 1 0 1 0 0 0 1 1 1 0 0 1 0 0 1 ABCDABCD A D C B f 1 f 2 f 3 f 4 f 5

10 ECOC - Picture 0 0 1 1 0 1 0 1 0 0 0 1 1 1 0 0 1 0 0 1 ABCDABCD A D C B f 1 f 2 f 3 f 4 f 5

11 ECOC - Picture 0 0 1 1 0 1 0 1 0 0 0 1 1 1 0 0 1 0 0 1 ABCDABCD A D C B f 1 f 2 f 3 f 4 f 5 X 1 1 1 1 0

12 Questions? How well does it work? How long should the code be? Do we need a lot of training data? What kind of codes can we use? Are there intelligent ways of creating the code?

13 Previous Work Combine with Boosting – ADABOOST.OC (Schapire, 1997), (Guruswami & Sahai, 1999) Local Learners (Ricci & Aha, 1997) Text Classification (Berger, 1999)

14 Experimental Setup Generate the code BCH Codes Choose a Base Learner Naive Bayes Classifier as used in text classification tasks (McCallum & Nigam 1998) Naive Bayes Classifier

15 Dataset Industry Sector Dataset Consists of company web pages classified into 105 economic sectors Standard stoplist No Stemming Skip all MIME headers and HTML tags Experimental approach similar to McCallum et al. (1998) for comparison purposes.

16 Results Classification Accuracies on five random 50-50 train-test splits of the Industry Sector dataset with a vocabulary size of 10000. ECOC is 88% accurate!

17 Results Industry Sector Data Set Naïve Bayes Shrinkage 1 ME 2 ME/ w Prior 3 ECOC 63-bit 66.1%76%79%81.1%88.5% ECOC reduces the error of the Naïve Bayes Classifier by 66% 1.(McCallum et al. 1998) 2,3. (Nigam et al. 1999)

18 The Longer the Better! Table 2: Average Classification Accuracy on 5 random 50-50 train-test splits of the Industry Sector dataset with a vocabulary size of 10000 words selected using Information Gain. Longer codes mean larger codeword separation The minimum hamming distance of a code C is the smallest distance between any pair of distance codewords in C If minimum hamming distance is h, then the code can correct  (h-1)/2 errors

19 Size Matters?

20 Size does NOT matter!

21 Semi-Theoretical Model Model ECOC by a Binomial Distribution B(n,p) n = length of the code p = probability of each bit being classified incorrectly

22 Semi-Theoretical Model Model ECOC by a Binomial Distribution B(n,p) n = length of the code p = probability of each bit being classified incorrectly # of BitsH min E max P ave Accuracy 1552.85.59 1552.89.80 1552.91.84 31115.85.67 31115.89.91 31115.91.94 633115.89.99

23 Semi-Theoretical Model Model ECOC by a Binomial Distribution B(n,p) n = length of the code p = probability of each bit being classified incorrectly # of BitsH min E max P ave Accuracy 1552.85.59 1552.89.80 1552.91.84 31115.85.67 31115.89.91 31115.91.94 633115.89.99

24

25 Types of Codes Data-Independent Data-Dependent Algebraic Random Hand-Constructed Adaptive

26 What is a Good Code? Row Separation Column Separation (Independence of errors for each binary classifier) Efficiency (for long codes)

27 Choosing Codes RandomAlgebraic Row SepOn Average For long codes Guaranteed Col SepOn Average For long codes Can be Guaranteed EfficiencyNoYes

28 Experimental Results CodeMin Row HD Max Row HD Min Col HD Max Col HD Error Rate 15-Bit BCH 515496420.6% 19-Bit Hybrid 518156922.3% 15-bit Random 2 (1.5) 13426024.1%

29 Interesting Observations NBC does not give good probabilitiy estimates- using ECOC results in better estimates.

30 Drawbacks Can be computationally expensive Random Codes throw away the real- world nature of the data by picking random partitions to create artificial binary problems

31 Future Work Combine ECOC with Co-Training Automatically construct optimal / adaptive codes

32 Conclusion Improves Classification Accuracy considerably! Can be used when training data is sparse Algebraic codes perform better than random codes for a given code lenth Hand-constructed codes are not the answer

Download ppt "Using Error-Correcting Codes For Text Classification Rayid Ghani This presentation can be accessed at"

Similar presentations

Study on Ensemble Learning By Feng Zhou. Content Introduction A Statistical View of M3 Network Future Works.

Study on Ensemble Learning By Feng Zhou. Content Introduction A Statistical View of M3 Network Future Works.
Mustafa Cayci INFS 795 An Evaluation on Feature Selection for Text Clustering.

Mustafa Cayci INFS 795 An Evaluation on Feature Selection for Text Clustering.
On-line learning and Boosting

On-line learning and Boosting
CPSC 502, Lecture 15Slide 1 Introduction to Artificial Intelligence (AI) Computer Science cpsc502, Lecture 15 Nov, 1, 2011 Slide credit: C. Conati, S.

CPSC 502, Lecture 15Slide 1 Introduction to Artificial Intelligence (AI) Computer Science cpsc502, Lecture 15 Nov, 1, 2011 Slide credit: C. Conati, S.
Linear Classifiers (perceptrons)

Linear Classifiers (perceptrons)
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.
Data Mining Classification: Alternative Techniques

Data Mining Classification: Alternative Techniques
Ensemble Methods An ensemble method constructs a set of base classifiers from the training data Ensemble or Classifier Combination Predict class label.

Ensemble Methods An ensemble method constructs a set of base classifiers from the training data Ensemble or Classifier Combination Predict class label.
Boosting Approach to ML

Boosting Approach to ML
2D1431 Machine Learning Boosting.

2D1431 Machine Learning Boosting.
Predictive Automatic Relevance Determination by Expectation Propagation Yuan (Alan) Qi Thomas P. Minka Rosalind W. Picard Zoubin Ghahramani.

Predictive Automatic Relevance Determination by Expectation Propagation Yuan (Alan) Qi Thomas P. Minka Rosalind W. Picard Zoubin Ghahramani.
Reducing Multiclass to Binary LING572 Fei Xia Week 9: 03/04/08.

Reducing Multiclass to Binary LING572 Fei Xia Week 9: 03/04/08.
ECOC for Text Classification Hybrids of EM & Co-Training (with Kamal Nigam) Learning to build a monolingual corpus from the web (with Rosie Jones) Effect.

ECOC for Text Classification Hybrids of EM & Co-Training (with Kamal Nigam) Learning to build a monolingual corpus from the web (with Rosie Jones) Effect.
Using Error-Correcting Codes For Text Classification Rayid Ghani Center for Automated Learning & Discovery, Carnegie Mellon University.

Using Error-Correcting Codes For Text Classification Rayid Ghani Center for Automated Learning & Discovery, Carnegie Mellon University.
Combining Labeled and Unlabeled Data for Multiclass Text Categorization Rayid Ghani Accenture Technology Labs.

Combining Labeled and Unlabeled Data for Multiclass Text Categorization Rayid Ghani Accenture Technology Labs.
Efficient Text Categorization with a Large Number of Categories Rayid Ghani KDD Project Proposal.

Efficient Text Categorization with a Large Number of Categories Rayid Ghani KDD Project Proposal.
Co-training LING 572 Fei Xia 02/21/06. Overview Proposed by Blum and Mitchell (1998) Important work: –(Nigam and Ghani, 2000) –(Goldman and Zhou, 2000)

Co-training LING 572 Fei Xia 02/21/06. Overview Proposed by Blum and Mitchell (1998) Important work: –(Nigam and Ghani, 2000) –(Goldman and Zhou, 2000)
1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006 5. Bayesian Learning 5.1 Introduction –Bayesian learning algorithms calculate explicit probabilities.

1er. Escuela Red ProTIC - Tandil, de Abril, Bayesian Learning 5.1 Introduction –Bayesian learning algorithms calculate explicit probabilities.
Boosting Main idea: train classifiers (e.g. decision trees) in a sequence. a new classifier should focus on those cases which were incorrectly classified.

Boosting Main idea: train classifiers (e.g. decision trees) in a sequence. a new classifier should focus on those cases which were incorrectly classified.
Using Error-Correcting Codes for Efficient Text Categorization with a Large Number of Categories Rayid Ghani Center for Automated Learning & Discovery.

Using Error-Correcting Codes for Efficient Text Categorization with a Large Number of Categories Rayid Ghani Center for Automated Learning & Discovery.

Similar presentations

Ads by Google