Example 16,000 documents 100 topic Picked those with large p(w|z)

Given a new document, compute and words allocated to each topic approximates p(z n |w) See cases where these values are relatively large 4 topics found New document?

Unseen document (contd.) Bag of words - William Randolph Hearst Foundation assigned to different topics

Applications and empirical results Document modeling Document classification Collaborative filtering

Document modeling Task: density estimation, high likelihood to unseen document Measure of goodness: perplexity Monotonically decreases in the likelihood

The experiment ArticlesTerms Scientific abstracts 5,22528,414 Newswire articles 16,33323,075

The experiment (contd.) Preprocessed –stop words –appearing once 10% held for training Trained with the same stopping criteria

Results

Overfitting in Mixture of unigrams Peaked posterior in the training set Unseen document with unseen word Word will have very small probability Remedy: smoothing

Overfitting in pLSI Mixture of topics allowed Marginalize over d to find p(w) Restriction to having the same topic proportions as training documents “Folding in” ignore p(z|d) parameters and refit p(z|d new )

LDA Documents can have different proportions of topics No heuristics

Document classification Generative or discriminative Choice of features in document classification LDA as dimensionality reduction technique as LDA features

The experiment Binary classification 8000 documents, 15,818 words True label not known 50 topic Trained SVM on the LDA features Compared with SVM on all word features LDA reduced feature space by 99.6%

GRAIN vs NOT GRAIN

EARN vs NOT EARN

LDA in document classification Feature space reduced, performance improved Results need further investigation Use for feature selection

Collaborative filtering Collection of users and movies they prefer Trained on observed users Task: given unobserved user and all movies preferred but one, predict the held out movie Only users who positively rated 100 movies Trained on 89% of data

Some quantities required… Probability of held out movie p(w|w obs ) –For mixture of unigrams and pLSI sum out topic variable –For LDA sum out topic and Dirichlet variables (quantity efficient to compute)

Results

Further work Other approaches for inference and parameter estimation Embedded in another model Other types of data Partial exchangeability

Example – Visual words Document = image Words = image features: bars, circles Topics = face, airplane Bag of words = no spatial relationship between objects

Visual words

Identifying the visual words and topics

Conclusion Exchangeability, De Finetti Theorem Dirichlet distribution Generative  Bag of words  Independence assumption in Dirichlet distribution - correlated topics

Implementations In C (by one of the authors) – In C and Matlab –

References Latent Dirichlet allocation, D. Blei, A. Ng, and M. Jordan. In Journal of Machine Learning Research, 3: , 2003 Discovering object categories in image collections. J. Sivic, B. C. Russell, A. A. Efros, A. Zisserman, W. T. Freeman. MIT AI Lab Memo AIM , February, 2005 Correlated topic models, David Blei and John Lafferty, Advances in Neural Information Processing Systems 18, 2005.