Using Error-Correcting Codes for Efficient Text Categorization with a Large Number of Categories Rayid Ghani Center for Automated Learning & Discovery Carnegie Mellon University

 Learning from Sequences of fMRI Brain Images (with Tom Mitchell)  Learning to automatically build language-specific corpora from the web (with Rosie Jones & Dunja Mladenic)  Effect of Smoothing on Naive Bayes for Text Classification (with Tong IBM Research)  Hypertext Categorization using links and extracted information (with Sean Slattery & Yiming Yang)  Hybrids of EM & Co-Training for semi-supervised learning (with Kamal Nigam)  Error-Correcting Output Codes for Text Classification Some Recent Work

Text Categorization Numerous Applications Search Engines/Portals Customer Service Routing …. Domains: Topics Genres Languages $$$ Making

Problems  Practical applications such as web portal deal with a large number of categories  A lot of labeled examples are needed for training the system

How do people deal with a large number of classes?  Use fast multiclass algorithms (Naïve Bayes) Builds one model per class  Use Binary classification algorithms (SVMs) and break an n class problems into n binary problems  What happens with a 1000 class problem?  Can we do better?

ECOC to the Rescue!  An n-class problem can be solved by solving log 2 n binary problems  More efficient than one-per-class  Does it actually perform better?

What is ECOC?  Solve multiclass problems by decomposing them into multiple binary problems ( Dietterich & Bakiri 1995 )  Use a learner to learn the binary problems

Training ECOC ABCDABCD f 1 f 2 f 3 f 4 X 1 1 Testing ECOC

ECOC - Picture ABCDABCD A D C B f 1 f 2 f 3 f 4

ECOC - Picture ABCDABCD A D C B f 1 f 2 f 3 f 4

ECOC - Picture ABCDABCD A D C B f 1 f 2 f 3 f 4

ECOC - Picture ABCDABCD A D C B f 1 f 2 f 3 f 4 X 1 1

ECOC works but…  Increased code length = Increased Accuracy  Increased code length = Increased Computational Cost

Classification Performance EfficiencyEfficiency Naïve Bayes ECOC GOAL (as used in Berger 99)

Choosing the codewords  Random? [Berger 1999, James 1999] Asymptotically good (the longer the better) Computational Cost is very high  Use Coding Theory for Good Error-Correcting Codes? [Dietterich & Bakiri 1995] Guaranteed properties for a fixed-length code

Experimental Setup  Generate the code BCH Codes  Choose a Base Learner Naive Bayes Classifier as used in text classification tasks (McCallum & Nigam 1998) Naive Bayes Classifier

Text Classification with Naïve Bayes  “Bag of Words” document representation  Estimate parameters of generative model:  Naïve Bayes classification:

Industry Sector Dataset Consists of company web pages classified into 105 economic sectors [McCallum et al. 1998, Ghani 2000]

Results Industry Sector Data Set Naïve Bayes Shrinkage 1 MaxEnt 2 MaxEnt/ w Prior 3 ECOC 63-bit 66.1%76%79%81.1%88.5% ECOC reduces the error of the Naïve Bayes Classifier by 66% with no increase in computational cost 1.(McCallum et al. 1998) 2,3. (Nigam et al. 1999)

ECOC for better Precision

Classification Performance EfficiencyEfficiency NB ECOC GOAL New Goal (as used in Berger 99)

Solutions  Design codewords that minimize cost and maximize “performance”  Investigate the assignment of codewords to classes  Learn the decoding function  Incorporate unlabeled data into ECOC

Size Matters?

What happens with sparse data?

Use unlabeled data with a large number of classes  How? Use EM Mixed Results  Think Again! Use Co-Training Disastrous Results  Think one more time

How to use unlabeled data?  Current learning algorithms using unlabeled data (EM, Co-Training) don’t work well with a large number of categories  ECOC works great with a large number of classes but there is no framework for using unlabeled data

ECOC + CoTraining = ECoTrain  ECOC decomposes multiclass problems into binary problems  Co-Training works great with binary problems  ECOC + Co-Train = Learn each binary problem in ECOC with Co-Training

ECOC+CoTrain - Results Algorithm300L+ 0U Per Class 50L + 250U Per Class 5L + 295U Per Class Naïve BayesUses No Unlabeled Data ECOC 15bit EMUses Unlabeled Data - 105Class Problem Co-Train ECoTrain (ECOC + Co- Training) Uses Unlabeled Data

What Next?  Use improved version of co-training (gradient descent) Less prone to random fluctuations Uses all unlabeled data at every iteration  Use Co-EM (Nigam & Ghani 2000) - hybrid of EM and Co-Training

Potential Drawbacks  Random Codes throw away the real-world nature of the data by picking random partitions to create artificial binary problems

Summary  Use ECOC for efficient text classification with a large number of categories  Increase Accuracy & Efficiency  Use Unlabeled data by combining ECOC and Co-Training  Generalize to domain-independent classification tasks involving a large number of categories

Semi-Theoretical Model Model ECOC by a Binomial Distribution B(n,p) n = length of the code p = probability of each bit being classified incorrectly

Semi-Theoretical Model Model ECOC by a Binomial Distribution B(n,p) n = length of the code p = probability of each bit being classified incorrectly # of BitsH min E max P ave Accuracy

Semi-Theoretical Model Model ECOC by a Binomial Distribution B(n,p) n = length of the code p = probability of each bit being classified incorrectly # of BitsH min E max P ave Accuracy

The Longer the Better! Table 2: Average Classification Accuracy on 5 random train-test splits of the Industry Sector dataset with a vocabulary size of words selected using Information Gain.  Longer codes mean larger codeword separation  The minimum hamming distance of a code C is the smallest distance between any pair of distance codewords in C  If minimum hamming distance is h, then the code can correct  (h-1)/2  errors

Data-Independent Data-Dependent Algebraic Random Hand-Constructed Adaptive Types of Codes

ECOC + CoTraining = ECoTrain  ECOC decomposes multiclass problems into binary problems  Co-Training works great with binary problems  ECOC + Co-Train = Learn each binary problem in ECOC with Co-Training  Preliminary Results: Not so great! (very sensitive to initial labeled documents)

What is a Good Code?  Row Separation  Column Separation (Independence of errors for each binary classifier)  Efficiency (for long codes)

Choosing Codes RandomAlgebraic Row SepOn Average For long codes Guaranteed Col SepOn Average For long codes Can be Guaranteed EfficiencyNoYes

Experimental Results CodeMin Row HD Max Row HD Min Col HD Max Col HD Error Rate 15-Bit BCH % 19-Bit Hybrid % 15-bit Random 2 (1.5) %

Interesting Observations  NBC does not give good probabilitiy estimates- using ECOC results in better estimates.

Testing ECOC  To test a new instance Apply each of the n classifiers to the new instance Combine the predictions to obtain a binary string(codeword) for the new point Classify to the class with the nearest codeword (usually hamming distance is used as the distance measure)

The Decoding Step  Standard: Map to the nearest codeword according to hamming distance  Can we do better?

The Real Question?  Tradeoff between “learnability” of binary problems and the error-correcting power of the code

Codeword assignment  Standard Procedure: Assign codewords to classes randomly  Can we do better?

Goal of Current Research  Improve classification performance without increasing cost Design short codes that perform well Develop algorithms that increase performance without affecting code length

Previous Results  Performance increases with length of code  Gives the same percentage increase in performance over NB regardless of training set size  BCH Codes > Random Codes > Hand- constructed Codes

Others have shown that ECOC  Works great with arbitrary long codes  Longer codes = More Error-Correcting Power = Better Performance  Longer codes = More Computational Cost

ECOC to the Rescue!  An n-class problem can be solved by solving log 2 n problems  More efficient than one-per-class  Does it actually perform better?

Previous Results Industry Sector Data Set Naïve Bayes Shrinkage 1 ME 2 ME/ w Prior 3 ECOC 63-bit 66.1%76%79%81.1%88.5% ECOC reduces the error of the Naïve Bayes Classifier by 66% with no increase in computational cost 1.(McCallum et al. 1998) 2,3. (Nigam et al. 1999) (Ghani 2000)

Design codewords  Maximize Performance (Accuracy, Precision, Recall, F1?)  Minimize length of codes  Search in the space of codewords through gradient descent G=Error + Code_Length

Codeword Assignment  Generate the confusion matrix and use that to assign the most confusable classes the codewords that are farthest apart  Pros Focusing on confusable classes more can help  Cons Individual binary problems can be very hard

The Decoding Step  Weight the individual classifiers according to their training accuracies and do weighted majority decoding.  Pose the decoding as a separate learning problem and use regression/Neural Network