Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Power of Word Clusters for Text Classification Noam Slonim and Naftali Tishby Presented by: Yangzhe Xiao.

Published by Modified over 9 years ago

Similar presentations

Presentation on theme: "The Power of Word Clusters for Text Classification Noam Slonim and Naftali Tishby Presented by: Yangzhe Xiao."— Presentation transcript:

1 The Power of Word Clusters for Text Classification Noam Slonim and Naftali Tishby Presented by: Yangzhe Xiao

2 Word-clusters vs words –Reduced feature dimensionality. –More robust. –18% increase in accuracy. Challenge: Group similar words into word-clusters that preserve the information about document categories. --Information Bottleneck (IB) Method.

3 IB method is based on following idea: Given the empirical joint distribution of two variables, one variable is compressed so that the mutual information about the other variable is preserved as much as possible. find clusters of the members of the set X, denoted here by, such that the mutual information I( ;Y) is maximized, under a constraint on the information extracted from X, I ( ;X).

4 The problem has optimal formal solution without any assumption about the origin of the joint distribution p(x,y).

5 Kullback-Leibler divergence between the conditional distributions p(y|x) and Z(β,x) is a normalization factor. Single positive β determines the softness of the classification.

6 Agglomerative IB Algorithm

7

8

9 Normalized information curves for all 10 iterations in large and small sample sizes

Download ppt "The Power of Word Clusters for Text Classification Noam Slonim and Naftali Tishby Presented by: Yangzhe Xiao."

Similar presentations

Mission-based Joint Optimal Resource Allocation in Wireless Multicast Sensor Networks Yun Hou Prof Kin K. Leung Archan Misra.

Mission-based Joint Optimal Resource Allocation in Wireless Multicast Sensor Networks Yun Hou Prof Kin K. Leung Archan Misra.
January 23 rd, 2011 1. Document classification task We are interested to solve a task of Text Classification, i.e. to automatically assign a given document.

January 23 rd, Document classification task We are interested to solve a task of Text Classification, i.e. to automatically assign a given document.
ECG Signal processing (2)

ECG Signal processing (2)
Feature Selection as Relevant Information Encoding Naftali Tishby School of Computer Science and Engineering The Hebrew University, Jerusalem, Israel NIPS.

Feature Selection as Relevant Information Encoding Naftali Tishby School of Computer Science and Engineering The Hebrew University, Jerusalem, Israel NIPS.
Unsupervised Learning

Unsupervised Learning
Semi-supervised Learning Rong Jin. Semi-supervised learning  Label propagation  Transductive learning  Co-training  Active learning.

Semi-supervised Learning Rong Jin. Semi-supervised learning  Label propagation  Transductive learning  Co-training  Active learning.
Minimum Redundancy and Maximum Relevance Feature Selection

Minimum Redundancy and Maximum Relevance Feature Selection
Information Bottleneck EM School of Engineering & Computer Science The Hebrew University, Jerusalem, Israel Gal Elidan and Nir Friedman.

Information Bottleneck EM School of Engineering & Computer Science The Hebrew University, Jerusalem, Israel Gal Elidan and Nir Friedman.
Middle Term Exam 03/01 (Thursday), take home, turn in at noon time of 03/02 (Friday)

Middle Term Exam 03/01 (Thursday), take home, turn in at noon time of 03/02 (Friday)
Clustering (1) Clustering Similarity measure Hierarchical clustering Model-based clustering Figures from the book Data Clustering by Gan et al.

Clustering (1) Clustering Similarity measure Hierarchical clustering Model-based clustering Figures from the book Data Clustering by Gan et al.
Bregman Information Bottleneck NIPS’03, Whistler December 2003 Koby Crammer Hebrew University of Jerusalem Noam Slonim Princeton University.

Bregman Information Bottleneck NIPS’03, Whistler December 2003 Koby Crammer Hebrew University of Jerusalem Noam Slonim Princeton University.
On feature distributional clustering for text categorization Bekkerman, El-Yaniv, Tishby and Winter The Technion. June, 27, 2001.

On feature distributional clustering for text categorization Bekkerman, El-Yaniv, Tishby and Winter The Technion. June, 27, 2001.
Middle Term Exam 03/04, in class. Project It is a team work No more than 2 people for each team Define a project of your own Otherwise, I will assign.

Middle Term Exam 03/04, in class. Project It is a team work No more than 2 people for each team Define a project of your own Otherwise, I will assign.
K nearest neighbor and Rocchio algorithm

K nearest neighbor and Rocchio algorithm
Distributional Clustering of Words for Text Classification Authors: L.Douglas Baker Andrew Kachites McCallum Presenter: Yihong Ding.

Distributional Clustering of Words for Text Classification Authors: L.Douglas Baker Andrew Kachites McCallum Presenter: Yihong Ding.
Local one class optimization Gal Chechik, Stanford joint work with Koby Crammer, Hebrew university of Jerusalem.

Local one class optimization Gal Chechik, Stanford joint work with Koby Crammer, Hebrew university of Jerusalem.
Information Bottleneck presented by Boris Epshtein & Lena Gorelick Advanced Topics in Computer and Human Vision Spring 2004.

Information Bottleneck presented by Boris Epshtein & Lena Gorelick Advanced Topics in Computer and Human Vision Spring 2004.
Logistic Regression Rong Jin. Logistic Regression Model  In Gaussian generative model:  Generalize the ratio to a linear model Parameters: w and c.

Logistic Regression Rong Jin. Logistic Regression Model  In Gaussian generative model:  Generalize the ratio to a linear model Parameters: w and c.
Logistic Regression Rong Jin. Logistic Regression Model  In Gaussian generative model:  Generalize the ratio to a linear model Parameters: w and c.

Logistic Regression Rong Jin. Logistic Regression Model  In Gaussian generative model:  Generalize the ratio to a linear model Parameters: w and c.
Announcements  Project proposal is due on 03/11  Three seminars this Friday (EB 3105) Dealing with Indefinite Representations in Pattern Recognition.

Announcements  Project proposal is due on 03/11  Three seminars this Friday (EB 3105) Dealing with Indefinite Representations in Pattern Recognition.

Similar presentations

Ads by Google