Vector Space Text Classification Adapted from Lectures by Raymond Mooney and Barbara Rosario If “XML Index and Search” is skipped summarize it as follows: Information Retrieval vs Information Extraction Use aggregation tree structure (DAG, in general due to Xlink) explicit in the document to “extract” more focused fragment than throwing up the entire document as is done in IR. (3) Research Issues: (i) Design of query language (User perspective on IR – Semantics of query that is intuitive) (ii) Implementation: indexing and search (need to index on fragments of XML file) We used Lucene Text Index and Search APIs (iii) Ranking : that uses TF-IDF plus labelled tree structure (*) Our work -> “Coherent Query Language for XML” Prasad L14VectorClassify

Text Classification Today: Later Introduction to Text Classification K Nearest Neighbors Decision boundaries Vector space classification using centroids Decision Trees (briefly) Later More text classification Support Vector Machines Text-specific issues in classification Vector space model based approaches (Centroid -> Rocchio) Prasad L14VectorClassify

Recap: Naïve Bayes classifiers Classify based on prior weight of class and conditional parameter for what each word says: Training is done by counting and dividing: Don’t forget to smooth Focus of NB approach is on the association of each word with a class. Multinomial approach preferred over Multivariate Bernoulli

Recall: Vector Space Representation Each document is a vector, one component for each term (= word). Normally normalize vector to unit length. High-dimensional vector space: Terms are axes 10,000+ dimensions, or even 100,000+ Docs are vectors in this space How can we do classification in this space? Prasad L14VectorClassify 14.1

Classification Using Vector Spaces As before, the training set is a set of documents, each labeled with its class (e.g., topic) In vector space classification, this set corresponds to a labeled set of points (or, equivalently, vectors) in the vector space Premise 1: Documents in the same class form a contiguous region of space Premise 2: Documents from different classes don’t overlap (much) We define surfaces to delineate classes in the space Prasad L14VectorClassify

Documents / Classes in a Vector Space Government Science Arts Prasad L14VectorClassify

Test Document of what class? Government Science Arts Prasad L14VectorClassify

Test Document = Government Is this similarity hypothesis true in general? Government Science Arts Our main topic today is how to find good separators

k Nearest Neighbor Classification Prasad L14VectorClassify

Aside: 2D/3D graphs can be misleading This example shows how distances in higher dimensions is not preserved when it is mapped to lower dimensions. Another example with sphere: imagine alternate 3D points x1, x3 and x5 on different hemispheres than x2 and x4.

k Nearest Neighbor Classification kNN = k Nearest Neighbor To classify document d into class c: Define k-neighborhood N as k nearest neighbors of d Count number of documents ic in N that belong to c Estimate P(c|d) as ic/k Choose as class arg maxc P(c|d) [ = majority class] Pick the class/label of the majority of the k-neighbors Prasad L14VectorClassify 14.3

Example: k=6 (6NN) P(science| )? Government Science Arts Prasad Govt. 5/6 Science 1/6 Science Arts Prasad L14VectorClassify

Nearest-Neighbor Learning Algorithm Learning is just storing the representations of the training examples in D. Does not explicitly compute a generalization, category prototypes, or class boundaries. Testing instance x: Compute similarity between x and all examples in D. Assign x the category of the most similar example in D. Also called: Case-based learning, Memory-based learning, Lazy learning Rationale of kNN: contiguity hypothesis Compared to other classification methods … learning step is trivial (In other algorithms the training set needs to be analyzed to learn class boundaries.) Prasad L14VectorClassify

kNN: Value of k Using only the closest example (k=1) to determine the category is subject to errors due to: A single atypical example. Noise in the category label of a single training example. More robust alternative is to find the k (> 1) most-similar examples and return the majority category of these k examples. Value of k is typically odd to avoid ties; 3 and 5 are most common. kNN is Close to Optimal (Cover and Hart 1967) Asymptotically, the error rate of 1-nearest-neighbor classification is less than twice the Bayes rate [error rate of classifier knowing model that generated data] In particular, asymptotic error rate is 0 if Bayes rate is 0. *****Theory that we skip**** Assume: query point coincides with a training point. Both query point and training point contribute error → 2 times Bayes rate Prasad L14VectorClassify

Probabilistic kNN 1NN, 3NN classification decision for star? 15

Exercise How is star classified by: (i) 1-NN (ii) 3-NN (iii) 9-NN (iv) 15-NN (v) Rocchio? 16

kNN decision boundaries Boundaries are, in principle, arbitrary surfaces – but usually polyhedra Government kNN can potetnially give rise to a non-linear classifier. Its behavior is dictated by local neighborhood as opposed to all points in a class : PROS: relatively more robust wrt multimodal distribution CONS: ignores overall strcuture of a class Marmelstein’s AFIT dissertation: learning piece-wise linear enclosures of classes for assessement by human-in-the-loop In higher dimensions it is very hard to see the overall shape of clustering of points based on raw data -> computationally very expensive Science Arts kNN gives locally defined decision boundaries between classes – far away points do not influence each classification decision (unlike in Naïve Bayes, Rocchio, etc.) Prasad L14VectorClassify

kNN is not a linear classifier Classification decision based on majority of k nearest neighbors. The decision boundaries between classes are piecewise linear . . . . . . but they are in general not linear classifiers that can be described as 18

Similarity Metrics Nearest neighbor method depends on a similarity (or distance) metric. Simplest for continuous m-dimensional instance space is Euclidean distance. Simplest for m-dimensional binary instance space is Hamming distance (number of feature values that differ). For text, cosine similarity of tf.idf weighted vectors is typically most effective. Prasad L14VectorClassify

Illustration of 3 Nearest Neighbor for Text Vector Space Prasad L14VectorClassify

Nearest Neighbor with Inverted Index Naively finding nearest neighbors requires a linear search through |D| documents in collection But determining k nearest neighbors is the same as determining the k best retrievals using the test document as a query to a database of training documents. Heuristics: Use standard vector space inverted index methods to find the k nearest neighbors. Testing Time: O(B|Vt|) where B is the average number of training documents in which a test-document word appears. Typically B << |D| Compute cosines with docs that match on at least one word = B docs x V number of words Prasad L14VectorClassify

kNN: Discussion No feature selection necessary Scales well with large number of classes Don’t need to train n classifiers for n classes No training necessary Actually: some data editing may be necessary. Classes can influence each other Small changes to one class can have ripple effect Scores can be hard to convert to probabilities May be more expensive at test time kNN does not work well if the dataset is not cleanly separable. Higher k values do not always guarantee better performance even though single value can cause trouble. In the extreme, consider concentric rings of points belonging to alternate classes. Classification errors may be high if k is high. Prasad L14VectorClassify

kNN vs. Naive Bayes : Bias/Variance tradeoff [Bias : Variance] :: Lack of [expressiveness : robustness] kNN has high variance and low bias. Infinite memory NB has low variance and high bias. Decision surface has to be linear (hyperplane – see later) Consider asking a botanist: Is an object a tree? Too much capacity/variance, low bias Botanist who memorizes Will always say “no” to new object (e.g., different # of leaves) Not enough capacity/variance, high bias Lazy botanist Says “yes” if the object is green You want the middle ground Bias/variance in terms of resulting classifier given randomly selected training set; why is it a tradeoff; when to choose low-bias method, when to choose low-variance method BIAS => Lack of expressive power VARIANCE => Sensitivity to the structure of data / Lack of robustness Linear classifier such as Rocchio : High bias (linear boundaries) but low variance (Good generalizer but may miss the mark for complex datasets) Non-linear classifier such as kNN: Low bias (flexible boundaries) but high variance (Can overfit noise and may miss generalizing data) (Example due to C. Burges)

Bias vs. variance: Choosing the correct model capacity Bias -> assumption? Variance -> actual data Linear vs Non-linear classifiers Linear classifiers are normally easier to learn and GENERALIZE well Non-linear classifiers are more powerful in that they can learn any shape of the boundary but in the extreme may OVERFIT the data All this also depends on how reflective is training set of the test set (or entire dataset). Prasad L14VectorClassify 14.6

Decision Boundaries Classification Prasad L14VectorClassify

Binary Classification : Linear Classifier Consider 2 class problems Deciding between two classes, perhaps, government and non-government [one-versus-rest classification] How do we define (and find) the separating surface? How do we test which region a test doc is in? Prasad L14VectorClassify

Separation by Hyperplanes A strong high-bias assumption is linear separability: in 2 dimensions, can separate classes by a line in higher dimensions, need hyperplanes Can find separating hyperplane by linear programming (or can iteratively fit solution via perceptron): separator can be expressed as ax + by = c Prasad

Linear programming / Perceptron Find a,b,c, such that ax + by  c for red points ax + by  c for green points. Prasad L14VectorClassify

x + y  1.5 for red points x + y  1.5 for green points. y Linearly separable AND 0 1 1 x y c 1 x Prasad L14VectorClassify

x + y  0.5 for red points x + y  0.5 for green points. y OR 0 1 1 Linearly separable x y c 1 x Prasad L14VectorClassify

The planar decision surface in data-space for the simple linear discriminant function: Y = wT x + w0 Take the component of x in the direction of w (divide dot product wT x by ||w||) At the separating surface, it is –w0/||w|| ==================================================== The y/|w| is the distance from the separating surface (component along the dir(w)). dir(w) = w/||w|| Distance from origin = component along w for any point p on the plane = dot product of unit vector along w and p = wT p /||w|| = -w0 /||w|| because the equation of plane is (wT x + w0 = 0) ====================================================== To characterize a hyperplane, draw a vector w perpendicular to it from the origin. The distance to the plane is –w0/||w|| along the unit vector dir(w). If vector x is on the plane, then its component along dir(w) is –w0/||w|| long (that is, w^T x will be –w0). If vector x is as shown, then its component along dir(w) > –w0/||w||. Notice that for (x1,x2) = (0,0), y < 0. Thus, y = w^T x + w0 simplifies to w0 < 0 which means the value of w0 is negative (or the distance from plane from the origin is positive.) If vector x is as shown, then its component along dir(w) > magnitude(w0/||w||) > 0 If vector x is below the red line, then its component along dir(w) < magnitude(w0/||w||) or < 0 ============================================================ Hyperplane dir = w/||w|| intercept = w0/||w|| Y-separation from plane = wT x /||w|| - (-w0/||w||) Prasad

y XOR Linearly inseparable 1 x 0 1 x y c 1 Prasad L14VectorClassify 1 Can be recast as three class lineraly separable problem? x 0 1 Prasad L14VectorClassify

Which Hyperplane? In general, lots of possible solutions for a,b,c. Prasad L14VectorClassify

Which Hyperplane? Lots of possible solutions for a,b,c. Some methods find a separating hyperplane, but not the optimal one [according to some criterion of expected goodness] E.g., perceptron Which points should influence optimality? All points Linear regression, Naïve Bayes Only “difficult points” close to decision boundary Support vector machines finding an optimal separating hyperplane

Linear Classifiers Many common text classifiers are linear classifiers Naïve Bayes Perceptron Rocchio Logistic regression Support vector machines (with linear kernel) Linear regression (Simple) perceptron neural networks Exercise: show Naive Bayes is linear in log space Prasad L14VectorClassify

Linear Classifiers Despite this similarity, noticeable performance differences For separable problems, there are infinite number of separating hyperplanes. Which one do you choose? What to do for non-separable problems? Different training methods pick different hyperplanes Classifiers more powerful than linear often don’t perform better. Why? Effectiveness: Generalization vs overfitting Efficiency: Prasad L14VectorClassify

Naive Bayes is a linear classifier Two-class Naive Bayes. We compute: Decide class C if the odds is greater than 1, i.e., if the log odds is greater than 0. So decision boundary is hyperplane:

A nonlinear problem A linear classifier like Naïve Bayes does badly on this task kNN will do very well (assuming enough training data)

High Dimensional Data Pictures like the one on the right are absolutely misleading! Documents are zero along almost all axes Most document pairs are very far apart (i.e., not strictly orthogonal, but only share very common words and a few scattered others) In classification terms: virtually all document sets are separable, for most classification This is partly why linear classifiers are quite successful in this domain Intuitions in higher dimensional space lacking Prasad L14VectorClassify

More Than Two Classes Any-of or multivalue classification Classes are independent of each other. A document can belong to 0, 1, or >1 classes. Decompose into n binary problems Quite common for documents One-of or multinomial or polytomous classification Classes are mutually exclusive. Each document belongs to exactly one class E.g., digit recognition is polytomous classification Digits are mutually exclusive Prasad 14.5

Set of Binary Classifiers: Any of Build a separator between each class and its complementary set (docs from all other classes). Given test doc, evaluate it for membership in each class. Apply decision criterion of classifiers independently Done Sometimes you could do better by considering dependencies between categories Prasad L14VectorClassify

Set of Binary Classifiers: One of Build a separator between each class and its complementary set (docs from all other classes). Given test doc, evaluate it for membership in each class. Assign document to class with: maximum score maximum confidence maximum probability Why different from multiclass/ any of classification? ? Prasad L14VectorClassify

Classification using Centroids Prasad L14VectorClassify

Relevance feedback: Basic idea The user issues a (short, simple) query. The search engine returns a set of documents. User marks some docs as relevant, some as nonrelevant. Search engine computes a new representation of the information need – should be better than the initial query. Search engine runs new query and returns new results. New results have (hopefully) better recall. 44

Rocchio illustrated 45

Using Rocchio for text classification Relevance feedback methods can be adapted for text categorization 2-class classification (Relevant vs. nonrelevant documents) Use standard TF/IDF weighted vectors to represent text documents For each category, compute a prototype vector by summing the vectors of the training documents in the category. Prototype = centroid of members of class Assign test documents to the category with the closest prototype vector based on cosine similarity. Relevance feedback methods can be adapted for text categorization. Prasad L14VectorClassify 14.2

Illustration of Rocchio Text Categorization Magnitude of the resultant does not matter for cosine similarity Prasad L14VectorClassify

Definition of centroid Where Dc is the set of all documents that belong to class c and v(d) is the vector space representation of d. Note that centroid will in general not be a unit vector even when the inputs are unit vectors. sum unit vectors

Rocchio Properties Forms a simple generalization of the examples in each class (a prototype). Prototype vector does not need to be averaged or otherwise normalized for length since cosine similarity is insensitive to vector length. Classification is based on similarity to class prototypes. Does not guarantee classifications are consistent with the given training data. Why not? Prasad L14VectorClassify

Rocchio Anomaly Prototype models have problems with polymorphic (disjunctive) categories. Prasad L14VectorClassify

3 Nearest Neighbor Comparison Nearest Neighbor tends to handle polymorphic categories better. Prasad L14VectorClassify

Rocchio is a linear classifier The separation boundary is the locus of all points equidistant from the two centroids

Two-class Rocchio as a linear classifier Line or hyperplane defined by: For Rocchio, set: Exercise 14.15: Locus of points equidistant from two centroids : ||u(c1) – x|| = ||u(c2) – x|| (a1 – x1)^2 + (a2 – x2)^2 = (b1 –x1)^2 + (b2 –x2)^2 (a1^2 -- 2 a1 x1 + x1^2) + (a2^2 -- 2 a2 x2 + x2^2) = (b1^2 -- 2 b1 x1 + x1^2) + (b2^2 -- 2 b2 x2 + x2^2) (a1^2 -- 2 a1 x1 ) + (a2^2 -- 2 a2 x2 ) = (b1^2 -- 2 b1 x1 ) + (b2^2 -- 2 b2 x2 ) (a1^2 + a2^2) -- (b1^2 + b2^2 ) = (2 a1 x1 + 2 a2 x2) -- (2 b1 x1 + 2 b2 x2 ) ||u(c1)||^2 - ||u(c2)||^2 = 2 (u(c1) – u(c2))^T x Hyperplane wT x = b such that w = u(c1) – u(c2) and b = 0.5 [ ||u(c1)||^2 - ||u(c2)||^2 ]

Rocchio classification Rocchio forms a simple representation for each class: the centroid/prototype Classification is based on similarity to / distance from the prototype/centroid It does not guarantee that classifications are consistent with the given training data It is little used outside text classification, but has been used quite effectively for text classification Again, cheap to train and test documents

Rocchio cannot handle nonconvex, multimodal classes Exercise: Why is Rocchio not expected to do well for the classification task a vs. b here? A is centroid of the a’s, B is centroid of the b’s. The point o is closer to A than to B. But o is a better fit for the b class. A is a multimodal class with two prototypes. But in Rocchio we only have one prototype. a a a a a a a a a a a a X A X a a a a a a a a a a O a a b b b b b b B b b b b b b b b 55

SKIP WHAT FOLLOWS Prasad L14VectorClassify

Decision Tree Classification Tree with internal nodes labeled by terms Branches are labeled by tests on the weight that the term has Leaves are labeled by categories Classifier categorizes document by descending tree following tests to leaf The label of the leaf node is then assigned to the document Most decision trees are binary trees (advantageous; may require extra internal nodes) DT make good use of a few high-leverage features

Decision Tree Categorization: Example Geometric interpretation of DT? Prasad

Decision Tree Learning Learn a sequence of tests on features, typically using top-down, greedy search At each stage choose the unused feature with highest Information Gain (feature/class MI) Binary (yes/no) or continuous decisions f1 !f1 f7 !f7 P(class) = .9 P(class) = .6 P(class) = .2 Prasad L14VectorClassify

Entropy Calculations If we have a set with k different values in it, we can calculate the entropy as follows: Where P(valuei) is the probability of getting the ith value when randomly selecting one from the set. So, for the set R = {a,a,a,b,b,b,b,b} Prasad a-values b-values

Looking at some data Color Size Shape Edible? Yellow Small Round + - Green Irregular Large

Entropy for our data set 16 instances: 9 positive, 7 negative. This equals: 0.9836 This makes sense – it’s almost a 50/50 split; so, the entropy should be close to 1. Prasad L14VectorClassify

Visualizing Information Gain Entropy of set = 0.9836 (16 examples) Entropy = 0.8113 (from 8 examples) Entropy = 0.9544 (from 8 examples) Color Size Shape Edible? Yellow Small Round + - Green Irregular Large Size Small Large Color Size Shape Edible? Yellow Small Round + - Green Irregular Color Size Shape Edible? Green Large Irregular - Yellow Round + Prasad L14VectorClassify

Visualizing Information Gain The data set that goes down each branch of the tree has its own entropy value. We can calculate for each possible attribute its expected entropy. This is the degree to which the entropy would change if we branch on this attribute. You add the entropies of the two children, weighted by the proportion of examples from the parent node that ended up at that child. 0.9836 Size (16 examples) Small Large 0.8113 0.9544 (8 examples) (8 examples) Entropy of left child is 0.8113 I(size=small) = 0.8113 Entropy of right child is 0.9544 I(size=large) = 0.9544 Prasad 8 examples with ‘small’ 8 examples with ‘large’

G(attrib) = I(parent) – I(attrib) We want to calculate the information gain (or entropy reduction). This is the reduction in ‘uncertainty’ when choosing our first branch as ‘size’. We will represent information gain as “G.” G(size) = I(parent) – I(size) G(size) = 0.9836 – 0.8828 G(size) = 0.1008 Entropy of all data at parent node = I(parent) = 0.9836 Child’s expected entropy for ‘size’ split = I(size) = 0.8828 So, we have gained 0.1008 bits of information about the dataset by choosing ‘size’ as the first branch of our decision tree. Prasad L14VectorClassify

Decision Tree Learning Fully grown trees tend to have decision rules that are overly specific and therefore unable to categorize documents well Therefore, pruning or early stopping methods for Decision Trees are normally a standard part of classification packages Use of small number of features is potentially bad in text categorization, but in practice decision trees do well for some text classification tasks Decision trees are easily interpreted by humans – compared to probabilistic methods like Naive Bayes Decision Trees are normally regarded as a symbolic machine learning algorithm, though they can be used probabilistically Prasad L14VectorClassify

Category: “interest” – Dumais et al. (Microsoft) Decision Tree rate=1 rate.t=1 lending=0 prime=0 discount=0 pct=1 year=0 year=1

Summary: Representation of Text Categorization Attributes Representations of text are usually very high dimensional (one feature for each word) High-bias algorithms that prevent overfitting in high-dimensional space generally work best For most text categorization tasks, there are many relevant features and many irrelevant ones Methods that combine evidence from many or all features (e.g. naive Bayes, kNN, neural-nets) often tend to work better than ones that try to isolate just a few relevant features (standard decision-tree or rule induction)* *Although one can compensate by using many rules Explain supervised learning Prasad

Which classifier do I use for a given text classification problem? Is there a learning method that is optimal for all text classification problems? No, because of bias-variance tradeoff. Factors to take into account: How much training data is available? How simple/complex is the problem? (linear vs. nonlinear decision boundary) How noisy is the problem? How stable is the problem over time? For an unstable problem, it’s better to use a simple and robust classifier. References: Fabrizio Sebastiani. Machine Learning in Automated Text Categorization. ACM Computing Surveys, 34(1):1-47, 2002. Tom Mitchell, Machine Learning. McGraw-Hill, 1997. Yiming Yang & Xin Liu, A re-examination of text categorization methods. Proceedings of SIGIR, 1999. Evaluating and Optimizing Autonomous Text Classification Systems (1995) David Lewis. Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval Trevor Hastie, Robert Tibshirani and Jerome Friedman, Elements of Statistical Learning: Data Mining, Inference and Prediction. Springer-Verlag, New York. Open Calais: Automatic Semantic Tagging Free (but they can keep your data), provided by Thompson/Reuters Weka: A data mining software package that includes an implementation of many ML algorithms