1. Partially Supervised Classification of Text Documents
Authors: Bing Liu Wee Sun Lee
Philip S. Yu Xiaoli Li
Presented by: Swetha Nandyala
Hi Everyone, I am Swetha and Today I would be talking about Partially Supervised Classification of Text Documents. The authors of this paper are Bing Liu, Wee Sun Lee, Philips S. Yu, Xiaoli Li.
CIS 525: Neural Computation

2. CIS 525: Neural Computation
Overview
Introduction
Theoretical Foundation
Background Methodology
NB-C
EM-Algorithm
Proposed Strategy
Evaluation Measures & Experiments
Conclusion
In this talk I will present an Introduction, certain theoretical Foundation Background Methodologyon NB-C and EM-Algorithm proposed strategy, Evaluation measures & Experiments and Conclusion.
CIS 525: Neural Computation

3. CIS 525: Neural Computation
Text Categorization
… the activity of labeling natural language texts with thematic categories from a pre-defined set [Sebastiani, 2002]
Text Categorization is a task of automatically assigning to a text document d from a given domain D, a category label c selected among a predefined set of category labels C. D
Categorization
System … c1 c2
….. cj ck
……...
Let look at Text Categorization. TC is the activity of labeling natural language texts with thematic categories from a pre-defined set i.e.. Say we have document domain D and a set of category labels C TC assigns a category label to each of the document in D. Then learning algorithms are applied on this new set of labeled documents to produce classifiers. This is nothing but standard supervised learning problem C
CIS 525: Neural Computation

4. Text Categorization(Contd.)
Standard Supervised Learning Problem
Bottleneck: Need for very large number of labeled training documents to build accurate classifier
Goal: To identify a particular class of documents from a set of mixed unlabeled documents
Standard classifications inapplicable
Partially Supervised Classification is used
But as we saw one bottle-neck of Supervised Learning is the need for training documents to be labeled (sometimes manually)…. So the main goal of this paper was to propose a strategy that could identify a particular class of documents from a set of unlabeled documents given a small set of positive documents. Since we have to classify unlabeled text documents the traditional classification problems are inapplicable and so partially supervised classification is used.
CIS 525: Neural Computation

5. Theoretical foundations
AIM: To show PSC is a constrained optimization problem
fixed distribution D over X x Y, where Y = {0,1}
X, Y: sets of possible documents, classes
Two sets of documents
labeled as positive P of size n1 drawn from X for DX|Y=1
Unlabeled U of size n2 drawn from X for DX independently
GOAL: Find the positive documents in U
If we have a fixed distribution D over X x Y where X is set of possible documents and Y is classes here{0,1} as we are concerned of two classes positive and negative.
Now that we know what partially supervised classification is let us see
CIS 525: Neural Computation

6. Theoretical foundations
learning algorithm: selects a function f  F: X  {0, 1}(a class of functions) to classify unlabeled documents
probability of error: Pr[f(X)  Y] is sum of “false positive” and “false negative” cases
rewritten as
Pr[f(X)  Y] = Pr[f(X) = 1  Y=0]+Pr[f(X) = 0  Y=1]
After transforming
Pr[f(X)  Y] = Pr[f(X) = 1] - Pr[Y = 1]
+ 2Pr[f(X) = 0|Y = 1]Pr[Y = 1]
why learning is possible in the partially supervised case
The paper theoretically shows that PSC is a constrained optimization problem
PrD[A]: A  X x Y chosen randomly according to D
T: a finite sample being a subset of our dataset
PrT[A]: A  T  X x Y chosen randomly
CIS 525: Neural Computation

7. Theoretical foundations (contd..)
Pr[f(X)  Y] = Pr[f(X) = 1] - Pr[Y = 1] + 2Pr[f(X) = 0|Y = 1]Pr[Y = 1]
Note that Pr[Y = 1] is constant
approximation:
keeping Pr[f(X) = 0|Y = 1] small
error  Pr[f(X) = 1] - Pr[Y = 1]
 Pr[f(X) = 1] – const
i.e. minimizing Pr[f(X) = 1]  minimizing error
 minimizing PrU[f(X) = 1]) & keeping PrP[f(X) = 1])  r
NOTHING BUT CONSTRAINT OPTIMIZATION PROBLEM  Learning Possible
Note that Pr[Y = 1] is constant no change in criteria
Recall= (relevant retrieved) / (all relevant)
for large enough sets P(ositive) and U(nlabeled)
CIS 525: Neural Computation

8. Naïve Bayesian Text Classification
D be set of training documents
C = {c1, c2, ...,c|C|}: predif. classes, here: c1, c2
For diD, Pr[cj|di]: posterior probs are calculated
in NB model: class with the highest Pr[cj|di] is assigned to the document
Say D is a set of training documents, each document considered an ordered list of words wt
V = <w1, w2, ..., w|V|>: vocabulary used
wd i,k  V is a word in position k of doc. Di
C = {c1, c2, ...,c|C|}: predif. Classes
Posterior probabilities are calculated and the class with highest Pr[cj|di] is assigned to the document
CIS 525: Neural Computation

9. CIS 525: Neural Computation
The EM-Algorithm
Iterative algorithm for maximum likelihood estimation in problems with incomplete data
Two step method
Expectation Step
Fills in missing data
Maximization Step
Estimate parameters after the missing data is filled
As we have learnt earlier EM is an Iterative algorithm for maximum likelihood estimation in problems with incomplete data. It has 2 steps the Expectation step and Maximization step. In Expectation step it fills missing data and in maximization step estimates parameters
CIS 525: Neural Computation

10. CIS 525: Neural Computation
Proposed Strategy
Step 1: Re-initialization
Iterative-EM: by applying EM-algorithm over P and U
Identifying a set of reliable negative documents from the unlabeled set, by introducing spies
Step 2: Building and selecting a classifier
Spy-EM: building a set of classifiers iteratively
selecting a good classifier from the set of classifiers constructed above
After building an initial classifier using naïve bayes and EM, the documents that are most likely to be negative in the mixed set are identified using some spies
The EM Algorithm generates sequence of solutions that increase the likelihood function, but the classification error need not be necessarily improving
CIS 525: Neural Computation

11. CIS 525: Neural Computation
Iterative EM with NB-C
Assign each document in P(ositive) to class label c1 and in U(nlabeled) to class c2
Pr[c1/ di] = 1 & Pr[c2/ di] = 0 for each di in P
Pr[c2/ dj] = 1 & Pr[c1/ dj] = 0 for each dj in U
After initial labeling, a NB-C is built and used to classify documents in U
revise posterior probabilities for documents in U
After revising, a NB-C with new posterior probs. is built
Iterative process goes on until EM converges
Setback: strongly biased towards positive documents
In IEM, each document in P is assigned to class c1 and each document in U to c2 i.e. Pr[c1/ di] = 1 & Pr[c2/ di] = 0 for each di in P
Pr[c2/ dj] = 1 & Pr[c1/ dj] = 0 for each dj in U. Using this initial labeling Naïve bayes classifier is built to classify documents in U. In particular NB-c is used to compute the posterior probs of documents in U which are now assigned as new probabilistic class labels. After all the probs are revised a new classifier NB-C is built on new posterior probs of Unlabeled and positive documents. This iterative process goes until the EM-converges. The final probabilistic class labels can be used in classification. One set back is that it is strongly biased towards positive documents. So a technique call spy technique is proposed to deal this problem and balance both the positive and negative documents.
CIS 525: Neural Computation

12. Step1: Re-Initialization
Sample a certain % of positive examples say “S” and put them into unlabeled set to act as “spies”
I-EM algorithm is utilized but the U(nlabeled) set now has some spy documents
After EM completes, the probabilistic labels of spies are used to decide which documents are most likely negative(LN)
threshold t used for decision making:
if Pr[c1|dj] < t: denoted as L(ikely)N(egative)
if for dj  S Pr[c1|dj] > t: denoted as U(nlabeled)
Here we sample a certain % of positive examples and put them into unlabeled set to act as “spies". The Unlabeled now has spy documents. The IEM is used. After the EM complete, the probabilistic labels of spies are used in deciding which documents belong to the likely negative set. To make this decision a threshold t is considered. Documents with if Pr[c1|dj] < t belong to LN and documents in U and not in P with if Pr[c1|dj] > t belong to U
CIS 525: Neural Computation

13. CIS 525: Neural Computation
positives
Step-1 effect
negatives
BEFORE AFTER
LN (likely
negative)
U(un-labeled)
U un-labeled
spies
some spies
P(positive)
P(positive)
Lets now see the results of step1. Before applying spy technique we have a set of positives (labeled) and an Unlabeled set which has positives negatives initially and we add some spies from P to U. After applying the spy technique, the spies help in putting most of the negatives in Likely negative set and most of the positives in Unlabeled set.
initial situation: U = P  N
no clue which are P and N
spies from P added to U
help of spies: most positives in U get into unlabeled set, while most negatives get into LN; purity of LN higher than that of U
CIS 525: Neural Computation

14. CIS 525: Neural Computation
Step-2: S-EM
Apply EM over P, LN and U
algorithm proceeds as follows:
put all spies S back to P (where they were before)
diP:  c1 (i.e. Pr[c1|di] = 1); (fixed thru iterations)
djLN:  c2 (i.e. Pr[c2|dj] = 1); (changing thru EM)
dkU: initially assigned no label (will be after EM(1))
run EM using P, LN and U until it converges
final classifier is produced when EM stops
Now that we have balanced the positive and negative documents and EM is employed over P,LN and U. The steps ate we put all the spies where they were initially, for all the documents in P we assign label c1 and LN we assign label c2. Initially unlabeled ones are assigned no labels. At the end of first iteration it will be assigned a probabilistic label and in subsequent iterations revised. The EM is iterated till convergence. This technique leaves us with a set of classifiers, producing a final classifier at the stop. This whole technique is called spy-technique.
CIS 525: Neural Computation

15. CIS 525: Neural Computation
Selecting Classifier
Pr[f(X)  Y] = Pr[f(X) = 1] - Pr[Y = 1] + 2Pr[f(X) = 0|Y = 1]Pr[Y = 1]
S-EM generates set of classifiers but classification is not necessarily improving
remedy: stop iterating of EM at some point
estimating the change of the probability error between iterations i and i+1
i = Pr[fi+1(X)  Y] - Pr[fi(X)  Y]
if i > 0 for the first time, then ith classifier produced is the final classifier
As we know EM is prone to local maxima trap i.e. if local maxima separates the 2 classes well EM works well. Otherwise (i.e. positives and negatives consist of many clusters each) the data is not separable. In such a case it may be better to stop at NB-C instead of iterating EM. To be more specific if we repeat the S-EM n times we produce n-classifiers from which we chose one. This selection is done after estimating the change of the probability error between iterations i and i+1. i.e. if i > 0 for the first time, then ith classifier produced is the final classifier
CIS 525: Neural Computation

16. CIS 525: Neural Computation
Evaluation measures
Accuracy (of a classifier) A = m/(m+i) , where m, i are numbers of correct and incorrect decisions, respectively
F-Score:
F = 2pr / (p+r) is a classification performance measure
Where
recall r = a/(a+c)
precision p = a/(a+b)
The F-value reflects the average effect of both precision and recall
Accuracy of a classifier is defined as ratio of number of correct decisions made to total decisions.
Recall is the ratio of number of true positives and total number of positives in test data while precision is number of true positives and all examples classified as positive by system
The F-value reflects the average effect of both precision and recall.
CIS 525: Neural Computation

17. CIS 525: Neural Computation
Experiments
30 datasets created from 2 large document corpora
objective:
recovering positive documents placed into mixed sets
for each experiment:
dividing full positive set into two subsets: P and R
P: positive set used in the algorithm with a% of the full positive set
R: set of remaining documents with b% have been put into U (not all in R put to U)
For experimentation 30 datasets were created from two large document corpora for e.g. 20 newsgroups subdivided into 4 groups
e.g. WebKB (CS depts.) subdivided into 7 categories.
The objective was to recover positive documents placed in mixed sets
For each experiment, the positive set is divided into two subsets P and R, P has a% of the full positive set and used in algorithm and R has b% of full positive set and part of R is put n mixed set.
parameters a and b have been varied to cover different scenarios
belief: in reality M is large and has a small proportion of positive documents
CIS 525: Neural Computation

18. CIS 525: Neural Computation
Experiments (contd…)
techniques used
NB-C: applied directly to P (c1) and U(c2) to built a classifier to classify data in set U
I-EM: applies EM to P and U as long as converges (no spy yet); final classifier to be applied to U to identify its positives
S-EM: spies used to re-initialize; I-EM to build the final classifier; threshold t used
3 techniques were applied. NB-C is applies directly to P and M to build a classifier to classify data
I-EM applies EM to P and M till convergence and final classifier is applied to M to identify its positives
S-EM reinitializes I-EM using spies to build the final classifier; threshold t used
CIS 525: Neural Computation

19. CIS 525: Neural Computation
Experiments (contd…)
S-EM outperforms NB and I-EM in F dramatically
S-EM outperforms NB and I-EM in A as well
comment: datasets skewed, so A is not a reliable measure of classifier’s performance
Here a and b are %s drawn form positive data to put in P and R, NB is naïve bayes, IEM8 is IEM after 8 iterations. I-EM did not seem to improve after that. And S-EM is spy-EM
If we look into the values clearly S-EM outperforms NB and I-EM in F and A
CIS 525: Neural Computation

20. CIS 525: Neural Computation
Experiments (contd…)
results show great effect of re-initialization with spies:
S-EM outperforms I-EMbest
re-initialization is not, however, the only factor of improvement:
S-EM outperforms S-EM4
conclusions: both Step-1 (reinitializing) and Step-2 (selecting the best model) are needed!
If we look into this table with various types of algorithms
IE-best is best results obtained from IM from all 8 iterations, I-EM8 is IEM after 8 iterations, S-EM4 is S-EM after 4th iteration and S-EM after selecting good model
CIS 525: Neural Computation

21. CIS 525: Neural Computation
Conclusion
Gives an overview of the theory on learning with positive and unlabeled examples
Describes a two-step strategy for learning which produces extremely accurate classifiers
Partially supervised classification is most helpful when initial model is insufficiently trained
The paper Gives an overview of the theory on learning with positive and unlabeled examples. Describes a two-step strategy for learning which produces extremely accurate classifiers and Partially supervised classification is most helpful when initial model is insufficiently trained
CIS 525: Neural Computation

22. CIS 525: Neural Computation
Questions?
CIS 525: Neural Computation