Unsupervised Transfer Classification Application to Text Categorization Tianbao Yang, Rong Jin, Anil Jain, Yang Zhou, Wei Tong Michigan State University

Overview  Introduction  Related Work  Unsupervised Transfer Classification  Problem Definition  Approach & Analysis  Experiments  Conclusions

Introduction  Classification:  supervised learning  semi-supervised learning  What if No label information is available?  impossible but not with some additional information supervised semi-supervised unsupervised classification

Introduction  Unsupervised transfer classification (UTC)  a collection of training examples and their assignments to auxiliary classes  to build a classification model for a target class …. auxiliary class 1auxiliary class K target class No Labeled training examples prior conditional probabilities

Introduction: Motivated Examples Image Annotation sky 1 sun water grass ? ? ? ? Social Tagging phoneverizonapple 1 google ? ? ? ? How to predict an annotation word/social tag that does not appear in the training data ? ?// / ///? auxiliary classes target classes

Related Work  Transfer Learning  transfer knowledge from source domain to target domain  similarity: transfer label information for auxiliary classes to target class  difference: assume NO label information for target class  Multi-Label Learning, Maximum Entropy Model

Unsupervised Transfer Classification Data for auxiliary class target class target class label target classification model Goal Prior probabilityconditional probabilities Class Information Examples Auxiliary Classes assignments to auxiliary classes

Maximum Entropy Model (MaxEnt) Favor uniform distribution Feature statistics computed from conditional model Feature statistics computed from training data : the jth feature function

Generalized MaxEnt With a large probability Equality constraints Inequality constraints

Generalized MaxEnt

is unknown for target class How to extend generalized MaxEnt to unsupervised transfer classification ?

 Estimating feature statistics of target class from those of the auxiliary classes Unsupervised Transfer Classification ~ ~

 Build up Relation between Auxiliary Classes and Target Class Independence Assumption

Unsupervised Transfer Classification  Estimating feature statistics for the target class by regression Feature Statistics for Auxiliary Classes Feature Statistics for Target Class Class Information

Unsupervised Transfer Classification  Dual problem : function of U; definition can be found in paper

Consistency Result With a large probability The optimal dual solution using the label information for the target class The dual solution obtained by the proposed approach

Experiments  Text categorization  Data sets: multi-labeled data  Protocol: leave one-class out as the target class  Metric: AUC (Area under ROC curve)

Experiments: Baselines  cModel  train a classifier for each auxiliary class  linearly combine them for the target class  cLabel  predict the assignment of the target class for training examples by linearly combining the labels of auxiliary classes  train a classifier using the predicted labels for target class  GME-avg  use generalized maxent model  compute the feature statistics for the target class by linearly combining those for the auxiliary classes  Proposed Approach: GME-Reg

Experiment (I)  Estimate class information from training data

 Compare to the classifier of the target class learned by supervised learning Experiment (I)

Experiment (II)  Obtain class information from external sources  Datasets: bibtex and delicious  bibsonomy  bibtexwww.bibsonomy.org/tags  ACM DL  bibtexwww.portal.acm.org  d eli.cio.us  deliciouswww.delicious.com/tag

Experiment (II)  Comparison with Supervised Classification ~1200

Conclusions  A new problem: unsupervised transfer classification  A statistical framework for unsupervised transfer classification  based on generalized maximum entropy  robust estimate feature statistics for target class  provable performance by consistency analysis  Future Work  relax independence assumption  better estimation of feature statistics for target class

Thanks Questions ?