1. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 1 龙星计划课程 : 信息检索 Topic Models for Text Mining ChengXiang Zhai ( 翟成祥 ) Department of Computer Science Graduate School of Library & Information Science Institute for Genomic Biology, Statistics University of Illinois, Urbana-Champaign http://www-faculty.cs.uiuc.edu/~czhai, czhai@cs.uiuc.edu

2. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 2 Text Management Applications Access Mining Organization Select information Create Knowledge Add Structure/Annotations

3. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 3 What Is Text Mining? “The objective of Text Mining is to exploit information contained in textual documents in various ways, including …discovery of patterns and trends in data, associations among entities, predictive rules, etc.” (Grobelnik et al., 2001) “Another way to view text data mining is as a process of exploratory data analysis that leads to heretofore unknown information, or to answers for questions for which the answer is not currently known.” (Hearst, 1999) (Slide from Rebecca Hwa’s “Intro to Text Mining”)

4. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 4 Two Different Views of Text Mining Data Mining View: Explore patterns in textual data –Find latent topics –Find topical trends –Find outliers and other hidden patterns Natural Language Processing View: Make inferences based on partial understanding natural language text –Information extraction –Question answering Shallow mining Deep mining

5. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 5 Applications of Text Mining Direct applications: Go beyond search to find knowledge –Question-driven (Bioinformatics, Business Intelligence, etc): We have specific questions; how can we exploit data mining to answer the questions? –Data-driven (WWW, literature, email, customer reviews, etc): We have a lot of data; what can we do with it? Indirect applications –Assist information access (e.g., discover latent topics to better summarize search results) –Assist information organization (e.g., discover hidden structures)

6. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 6 Text Mining Methods Data Mining Style: View text as high dimensional data –Frequent pattern finding –Association analysis –Outlier detection Information Retrieval Style: Fine granularity topical analysis –Topic extraction –Exploit term weighting and text similarity measures Natural Language Processing Style: Information Extraction –Entity extraction –Relation extraction –Sentiment analysis –Question answering Machine Learning Style: Unsupervised or semi-supervised learning –Mixture models –Dimension reduction Topic of this lecture

7. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 7 Outline The Basic Topic Models: –Probabilistic Latent Semantic Analysis (PLSA) [Hofmann 99] –Latent Dirichlet Allocation (LDA) [Blei et al. 02] Extensions –Contextual Probabilistic Latent Semantic Analysis (CPLSA) [Mei & Zhai 06] –Other extensions

8. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 8 Basic Topic Model: PLSA

9. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 9 PLSA: Motivation What did people say in their blog articles about “Hurricane Katrina”? Query = “Hurricane Katrina” Results:

10. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 10 Probabilistic Latent Semantic Analysis/Indexing (PLSA/PLSI) [Hofmann 99] Mix k multinomial distributions to generate a document Each document has a potentially different set of mixing weights which captures the topic coverage When generating words in a document, each word may be generated using a DIFFERENT multinomial distribution (this is in contrast with the document clustering model where, once a multinomial distribution is chosen, all the words in a document would be generated using the same model) We may add a background distribution to “attract” background words

11. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 11 PLSA as a Mixture Model Topic  1 Topic  k Topic  2 … Document d Background B warning 0.3 system 0.2.. aid 0.1 donation 0.05 support 0.02.. statistics 0.2 loss 0.1 dead 0.05.. is 0.05 the 0.04 a 0.03.. kk 11 22 B B W  d,1  d, k 1 - B  d,2 “Generating” word w in doc d in the collection Parameters: B =noise-level (manually set)  ’s and  ’s are estimated with Maximum Likelihood ? ? ? ? ? ? ? ? ? ? ?

12. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 12 Special Case: Model-based Feedback Simple case: there is only one topic P(w|  F ) P(w|  B ) 1- P(source) Background words Topic words Maximum Likelihood: What about there are k topics?

13. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 13 How to Estimate  j : EM Algorithm the 0.2 a 0.1 we 0.01 to 0.02 … Known Background p(w | B) … text =? mining =? association =? word =? … Unknown topic model p(w|  1 )=? “Text mining” Observed Doc(s) ML Estimator … … information =? retrieval =? query =? document =? … Unknown topic model p(w|  2 )=? “information retrieval” Suppose, we know the identity of each word...

14. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 14 How the Algorithm Works 14 aid price oil π d1,1 ( P(θ 1 |d 1 ) ) π d1,2 ( P(θ 2 |d 1 ) ) π d2,1 ( P(θ 1 |d 2 ) ) π d2,2 ( P(θ 2 |d 2 ) ) aid price oil Topic 1Topic 2 aid price oil P(w| θ) Initial value Initializing π d, j and P(w| θ j ) with random values Iteration 1: E Step: split word counts with different topics (by computing z’ s) Iteration 1: M Step: re- estimate π d, j and P(w| θ j ) by adding and normalizing the splitted word counts Iteration 2: E Step: split word counts with different topics (by computing z’ s) Iteration 2: M Step: re- estimate π d, j and P(w| θ j ) by adding and normalizing the splitted word counts Iteration 3, 4, 5, … Until converging 7 5 6 8 7 5 d1d1 d2d2 c(w, d) c(w,d)p(z d,w = B) c(w,d)(1 - p(z d,w = B))p(z d,w =j)

15. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 15 Parameter Estimation E-Step: Word w in doc d is generated - from cluster j - from background Application of Bayes rule M-Step: Re-estimate - mixing weights - cluster LM Fractional counts contributing to - using cluster j in generating d - generating w from cluster j Sum over all docs (in multiple collections) m = 1 if one collection

16. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 16 PLSA with Prior Knowledge There are different ways of choosing aspects (topics) –Google = Google News + Google Map + Google scholar, … –Google = Google US + Google France + Google China, … Users have some domain knowledge in mind, e.g., –We expect to see “retrieval models” as a topic in IR. –We want to show the aspects of “history” and “statistics” for Youtube A flexible way to incorporate such knowledge as priors of PLSA model In Bayesian, it’s your “belief” on the topic distributions 16

17. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 17 Adding Prior Topic  1 Topic  k Topic  2 … Document d Background B warning 0.3 system 0.2.. aid 0.1 donation 0.05 support 0.02.. statistics 0.2 loss 0.1 dead 0.05.. is 0.05 the 0.04 a 0.03.. kk 11 22 B B W  d,1  d, k 1 - B  d,2 “Generating” word w in doc d in the collection Parameters: B =noise-level (manually set)  ’s and  ’s are estimated with Maximum Likelihood Most likely 

18. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 18 Adding Prior as Pseudo Counts 18 the 0.2 a 0.1 we 0.01 to 0.02 … Known Background p(w | B) … text =? mining =? association =? word =? … Unknown topic model p(w|  1 )=? “Text mining” … information =? retrieval =? query =? document =? … … Unknown topic model p(w|  2 )=? “information retrieval” Suppose, we know the identity of each word... Observed Doc(s) MAP Estimator Pseudo Doc Size = μ text mining

19. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 19 Maximum A Posterior (MAP) Estimation +  p(w|  ’ j ) ++ Pseudo counts of w from prior  ’ Sum of all pseudo counts What if  =0? What if  =+  ?

20. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 20 Basic Topic Model: LDA The following slides about LDA are taken from Michael C. Mozer’s course lecture http://www.cs.colorado.edu/~mozer/courses/ProbabilisticModels/

21. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 LDA: Motivation –“Documents have no generative probabilistic semantics” i.e., document is just a symbol –Model has many parameters linear in number of documents need heuristic methods to prevent overfitting –Cannot generalize to new documents

22. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 Unigram Model

23. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 Mixture of Unigrams

24. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 Topic Model / Probabilistic LSI d is a localist representation of (trained) documents LDA provides a distributed representation

25. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 LDA Vocabulary of |V| words Document is a collection of words from vocabulary. N words in document ww = (w 1,..., w N ) Latent topics random variable z, with values 1,..., k Like topic model, document is generated by sampling a topic from a mixture and then sampling a word from a mixture. But topic model assumes a fixed mixture of topics (multinomial distribution) for each document. LDA assumes a random mixture of topics (Dirichlet distribution) for each topic.

26. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 Generative Model “Plates” indicate looping structure Outer plate replicated for each document Inner plate replicated for each word Same conditional distributions apply for each replicate Document probability

27. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 Fancier Version

28. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 Inference

29. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 Inference In general, this formula is intractable: Expanded version: 1 if w n is the j'th vocab word

30. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 Variational Approximation Computing log likelihood and introducing Jensen's inequality: log(E[x]) >= E[log(x)] Find variational distribution q such that the above equation is computable. –q parameterized by γ and φ n –Maximize bound with respect to γ and φ n to obtain best approximation to p(w | α, β) –Lead to variational EM algorithm Sampling algorithms (e.g., Gibbs sampling) are also common

31. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 Data Sets C. Elegans Community abstracts 5,225 abstracts 28,414 unique terms TREC AP corpus (subset) 16,333 newswire articles 23,075 unique terms Held-out data – 10% Removed terms 50 stop words, words appearing once

32. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 C. Elegans Note: fold in hack for pLSI to allow it to handle novel documents. Involves refitting p(z|d new ) parameters -> sort of a cheat

33. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 AP

34. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 34 Summary: PLSA vs. LDA LDA adds a Dirichlet distribution on top of PLSA to regularize the model Estimation of LDA is more complicated than PLSA LDA is a generative model, while PLSA isn’t PLSA is more likely to over-fit the data than LDA Which one to use? –If you need generalization capacity, LDA –If you want to mine topics from a collection, PLSA may be better (we want overfitting!)

35. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 35 Extension of PLSA: Contextual Probabilistic Latent Semantic Analysis (CPLSA)

36. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 36 A General Introduction to EM Data: X (observed) + H(hidden) Parameter:  “Incomplete” likelihood: L(  )= log p(X|  ) “Complete” likelihood: L c (  )= log p(X,H|  ) EM tries to iteratively maximize the incomplete likelihood: Starting with an initial guess  (0), 1. E-step: compute the expectation of the complete likelihood 2. M-step: compute  (n) by maximizing the Q-function

37. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 37 Convergence Guarantee Goal: maximizing “Incomplete” likelihood : L(  )= log p(X|  ) I.e., choosing  (n), so that L(  (n) )-L(  (n-1) )  0 Note that, since p(X,H|  ) =p(H|X,  ) P(X|  ), L(  ) =L c (  ) -log p(H|X,  ) L(  (n) )-L(  (n-1) ) = L c (  (n) )-L c (  (n-1) )+log [p(H|X,  (n-1) )/p(H|X,  (n) )] Taking expectation w.r.t. p(H|X,  (n-1) ), L(  (n) )-L(  (n-1) ) = Q(  (n) ;  (n-1) )-Q(  (n-1) ;  (n-1) ) + D(p(H|X,  (n-1) )||p(H|X,  (n) )) KL-divergence, always non-negative EM chooses  (n) to maximize Q Therefore, L(  (n) )  L(  (n-1) )! Doesn’t contain H

38. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 38 Another way of looking at EM Likelihood p(X|  )  current guess Lower bound (Q function) next guess E-step = computing the lower bound M-step = maximizing the lower bound L(  (n-1) ) + Q(  ;  (n-1) ) -Q(  (n-1) ;  (n-1) ) + D(p(H|X,  (n-1) )||p(H|X,  )) L(  (n-1) ) + Q(  ;  (n-1) ) -Q(  (n-1) ;  (n-1) )

39. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 39 Why Contextual PLSA?

40. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 40 Motivating Example: Comparing Product Reviews Common Themes“IBM” specific“APPLE” specific“DELL” specific Battery LifeLong, 4-3 hrsMedium, 3-2 hrsShort, 2-1 hrs Hard diskLarge, 80-100 GBSmall, 5-10 GBMedium, 20-50 GB SpeedSlow, 100-200 MhzVery Fast, 3-4 GhzModerate, 1-2 Ghz IBM Laptop Reviews APPLE Laptop Reviews DELL Laptop Reviews Unsupervised discovery of common topics and their variations

41. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 41 Motivating Example: Comparing News about Similar Topics Common Themes“Vietnam” specific“Afghan” specific“Iraq” specific United nations ……… Death of people ……… … ……… Vietnam WarAfghan War Iraq War Unsupervised discovery of common topics and their variations

42. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 42 Motivating Example: Discovering Topical Trends in Literature Unsupervised discovery of topics and their temporal variations Theme Strength Time 19801990 19982003 TF-IDF Retrieval IR Applications Language Model Text Categorization

43. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 43 Motivating Example: Analyzing Spatial Topic Patterns How do blog writers in different states respond to topics such as “oil price increase during Hurricane Karina”? Unsupervised discovery of topics and their variations in different locations

44. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 44 Motivating Example: Sentiment Summary Unsupervised/Semi-supervised discovery of topics and different sentiments of the topics

45. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 45 Research Questions Can we model all these problems generally? Can we solve these problems with a unified approach? How can we bring human into the loop?

46. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 46 Contextual Text Mining Given collections of text with contextual information (meta- data) Discover themes/subtopics/topics (interesting word clusters) Compute variations of themes over contexts Applications: –Summarizing search results –Federation of text information –Opinion analysis –Social network analysis –Business intelligence –..

47. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 47 Context Features of Text (Meta-data) Weblog Article Author Author’s Occupation Location Time communities source

48. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 48 Context = Partitioning of Text 1999 2005 2006 1998 …… papers written in 1998 WWWSIGIRACLKDDSIGMOD papers written by authors in US Papers about Web

49. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 49 Themes/Topics Uses of themes: –Summarize topics/subtopics –Navigate in a document space –Retrieve documents –Segment documents –… Theme  1 Theme  k Theme  2 … Background B government 0.3 response 0.2.. donate 0.1 relief 0.05 help 0.02.. city 0.2 new 0.1 orleans 0.05.. Is 0.05 the 0.04 a 0.03.. [ Criticism of government response to the hurricane primarily consisted of criticism of its response to the approach of the storm and its aftermath, specifically in the delayed response ] to the [ flooding of New Orleans. … 80% of the 1.3 million residents of the greater New Orleans metropolitan area evacuated ] …[ Over seventy countries pledged monetary donations or other assistance]. …

50. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 50 View of Themes: Context-Specific Version of Views Context: After 1998 (Language models) Context: Before 1998 (Traditional models) vector space TF-IDF Okapi LSI vector Rocchio weighting feedback term retrieval feedback language model smoothing query generation mixture estimate EM pseudo model feedback judge expansion pseudo query Theme 2: Feedback Theme 1: Retrieval Model retrieve model relevance documen t query

51. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 51 Coverage of Themes: Distribution over Themes Background Theme coverage can depend on context Oil Price Government Response Aid and donation Criticism of government response to the hurricane primarily consisted of criticism of its response to … The total shut-in oil production from the Gulf of Mexico … approximately 24% of the annual production and the shut-in gas production … Over seventy countries pledged monetary donations or other assistance. … Background Oil Price Government Response Aid and donation Context: Texas Context: Louisiana

52. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 52 General Tasks of Contextual Text Mining Theme Extraction: Extract the global salient themes –Common information shared over all contexts View Comparison: Compare a theme from different views –Analyze the content variation of themes over contexts Coverage Comparison: Compare the theme coverage of different contexts –Reveal how closely a theme is associated to a context Others: –Causal analysis –Correlation analysis

53. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 53 A General Solution: CPLSA CPLAS = Contextual Probabilistic Latent Semantic Analysis An extension of PLSA model ([Hofmann 99]) by –Introducing context variables –Modeling views of topics –Modeling coverage variations of topics Process of contextual text mining –Instantiation of CPLSA (context, views, coverage) –Fit the model to text data (EM algorithm) –Compute probabilistic topic patterns

54. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 54 Document context: Time = July 2005 Location = Texas Author = xxx Occup. = Sociologist Age Group = 45+ … “Generation” Process of CPLSA View1View2View3 Themes government donation New Orleans government 0.3 response 0.2.. donate 0.1 relief 0.05 help 0.02.. city 0.2 new 0.1 orleans 0.05.. TexasJuly 2005 sociolo gist Theme coverages: Texas July 2005 document …… Choose a view Choose a Coverage government donate new Draw a word from  i response aid help Orleans Criticism of government response to the hurricane primarily consisted of criticism of its response to … The total shut-in oil production from the Gulf of Mexico … approximately 24% of the annual production and the shut- in gas production … Over seventy countries pledged monetary donations or other assistance. … Choose a theme

55. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 55 To generate a document D with context feature set C: –Choose a view v i according to the view distribution –Choose a coverage к j according to the coverage distribution –Choose a theme according to the coverage к j –Generate a word using –The likelihood of the document collection is: Probabilistic Model

56. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 56 Parameter Estimation: EM Algorithm Interesting patterns: –Theme content variation for each view: –Theme strength variation for each context Prior from a user can be incorporated using MAP estimation

57. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 57 Regularization of the Model Why? –Generality  high complexity (inefficient, multiple local maxima) –Real applications have domain constraints/knowledge Two useful simplifications: –Fixed-Coverage: Only analyze the content variation of themes (e.g., author-topic analysis, cross-collection comparative analysis ) –Fixed-View: Only analyze the coverage variation of themes (e.g., spatiotemporal theme analysis) In general –Impose priors on model parameters –Support the whole spectrum from unsupervised to supervised learning

58. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 58 Interpretation of Topics Statistical topic models term 0.1599 relevance 0.0752 weight 0.0660 feedback 0.0372 independence 0.0311 model 0.0310 frequent 0.0233 probabilistic 0.0188 document 0.0173 … term 0.1599 relevance 0.0752 weight 0.0660 feedback 0.0372 independence 0.0311 model 0.0310 frequent 0.0233 probabilistic 0.0188 document 0.0173 … term 0.1599 relevance 0.0752 weight 0.0660 feedback 0.0372 independence 0.0311 model 0.0310 frequent 0.0233 probabilistic 0.0188 document 0.0173 … Multinomial topic models NLP Chunker Ngram stat. database system, clustering algorithm, r tree, functional dependency, iceberg cube, concurrency control, index structure … Candidate label pool Collection (Context) Ranked List of Labels clustering algorithm; distance measure; … Relevance Score Re-ranking Coverage; Discrimination

59. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 59 Relevance: the Zero-Order Score Intuition: prefer phrases covering high probability words Clustering dimensional algorithm birch shape Latent Topic  … Good Label ( l 1 ): “clustering algorithm” body Bad Label ( l 2 ): “body shape” … p(w|  )

60. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 60 Relevance: the First-Order Score Intuition: prefer phrases with similar context (distribution) Clustering dimension partition algorithm hash Clustering hash dimension algorithm partition C: SIGMOD Proceedings Topic  … … P(w|  ) P(w| l 1 ) D(  || l 1 ) < D(  || l 2 ) Good Label ( l 1 ): “clustering algorithm” Clustering hash dimension join algorithm … Bad Label ( l 2 ): “hash join” P(w| l 2 ) Score (l,  )

61. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 61 Sample Results Comparative text mining Spatiotemporal pattern mining Sentiment summary Event impact analysis Temporal author-topic analysis

62. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 62 Comparing News Articles Iraq War (30 articles) vs. Afghan War (26 articles) Cluster 1Cluster 2Cluster 3 Common Theme united 0.042 nations 0.04 … killed 0.035 month 0.032 deaths 0.023 … … Iraq Theme n 0.03 Weapons 0.024 Inspections 0.023 … troops 0.016 hoon 0.015 sanches 0.012 … … Afghan Theme Northern 0.04 alliance 0.04 kabul 0.03 taleban 0.025 aid 0.02 … taleban 0.026 rumsfeld 0.02 hotel 0.012 front 0.011 … … The common theme indicates that “United Nations” is involved in both wars Collection-specific themes indicate different roles of “United Nations” in the two wars

63. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 63 Comparing Laptop Reviews Top words serve as “labels” for common themes (e.g., [sound, speakers], [battery, hours], [cd,drive]) These word distributions can be used to segment text and add hyperlinks between documents

64. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 64 Spatiotemporal Patterns in Blog Articles Query= “Hurricane Katrina” Topics in the results: Spatiotemporal patterns

65. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 65 Theme Life Cycles for Hurricane Katrina city 0.0634 orleans 0.0541 new 0.0342 louisiana 0.0235 flood 0.0227 evacuate 0.0211 storm 0.0177 … price 0.0772 oil 0.0643 gas 0.0454 increase 0.0210 product 0.0203 fuel 0.0188 company 0.0182 … Oil Price New Orleans

66. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 66 Theme Snapshots for Hurricane Katrina Week4: The theme is again strong along the east coast and the Gulf of Mexico Week3: The theme distributes more uniformly over the states Week2: The discussion moves towards the north and west Week5: The theme fades out in most states Week1: The theme is the strongest along the Gulf of Mexico

67. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 67 Theme Life Cycles: KDD Global Themes life cycles of KDD Abstracts gene 0.0173 expressions 0.0096 probability 0.0081 microarray 0.0038 … marketing 0.0087 customer 0.0086 model 0.0079 business 0.0048 … rules 0.0142 association 0.0064 support 0.0053 …

68. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 68 Theme Evolution Graph: KDD T SVM 0.007 criteria 0.007 classifica – tion 0.006 linear 0.005 … decision 0.006 tree 0.006 classifier 0.005 class 0.005 Bayes 0.005 … Classifica - tion 0.015 text 0.013 unlabeled 0.012 document 0.008 labeled 0.008 learning 0.007 … Informa - tion 0.012 web 0.010 social 0.008 retrieval 0.007 distance 0.005 networks 0.004 … ………… 1999 … web 0.009 classifica – tion 0.007 features0.006 topic 0.005 … mixture 0.005 random 0.006 cluster 0.006 clustering 0.005 variables 0.005 … topic 0.010 mixture 0.008 LDA 0.006 semantic 0.005 … … 20002001200220032004

69. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 69 Blog Sentiment Summary (query=“Da Vinci Code”) NeutralPositiveNegative Facet 1: Movie... Ron Howards selection of Tom Hanks to play Robert Langdon. Tom Hanks stars in the movie,who can be mad at that? But the movie might get delayed, and even killed off if he loses. Directed by: Ron Howard Writing credits: Akiva Goldsman... Tom Hanks, who is my favorite movie star act the leading role. protesting... will lose your faith by... watching the movie. After watching the movie I went online and some research on... Anybody is interested in it?... so sick of people making such a big deal about a FICTION book and movie. Facet 2: Book I remembered when i first read the book, I finished the book in two days. Awesome book.... so sick of people making such a big deal about a FICTION book and movie. I’m reading “Da Vinci Code” now. … So still a good book to past time. This controversy book cause lots conflict in west society.

70. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 70 Results: Sentiment Dynamics Facet: the book “ the da vinci code”. ( Bursts during the movie, Pos > Neg ) Facet: religious beliefs ( Bursts during the movie, Neg > Pos )

71. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 71 Event Impact Analysis: IR Research vector 0.0514 concept 0.0298 extend 0.0297 model 0.0291 space 0.0236 boolean 0.0151 function 0.0123 feedback 0.0077 … xml 0.0678 email 0.0197 model 0.0191 collect 0.0187 judgment 0.0102 rank 0.0097 subtopic 0.0079 … probabilist 0.0778 model 0.0432 logic 0.0404 ir 0.0338 boolean 0.0281 algebra 0.0200 estimate 0.0119 weight 0.0111 … model 0.1687 language 0.0753 estimate 0.0520 parameter 0.0281 distribution 0.0268 probable 0.0205 smooth 0.0198 markov 0.0137 likelihood 0.0059 … 1998 Publication of the paper “A language modeling approach to information retrieval” Starting of the TREC conferences year 1992 term 0.1599 relevance 0.0752 weight 0.0660 feedback 0.0372 independence 0.0311 model 0.0310 frequent 0.0233 probabilistic 0.0188 document 0.0173 … Theme: retrieval models SIGIR papers

72. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 72 Temporal-Author-Topic Analysis pattern 0.1107 frequent 0.0406 frequent-pattern 0.039 sequential 0.0360 pattern-growth 0.0203 constraint 0.0184 push 0.0138 … project 0.0444 itemset 0.0433 intertransaction 0.0397 support 0.0264 associate 0.0258 frequent 0.0181 closet 0.0176 prefixspan 0.0170 … research 0.0551 next 0.0308 transaction 0.0308 panel 0.0275 technical 0.0275 article 0.0258 revolution 0.0154 innovate 0.0154 … close 0.0805 pattern 0.0720 sequential 0.0462 min_support 0.0353 threshold 0.0207 top-k 0.0176 fp-tree 0.0102 … index 0.0440 graph 0.0343 web 0.0307 gspan 0.0273 substructure 0.0201 gindex 0.0164 bide 0.0115 xml 0.0109 … 2000 time Author Author B Author A Global theme: frequent patterns Jiawei Han Rakesh Agrawal

73. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 73 Modeling Topical Communities (Mei et al. 08) 73 Community 1: Information Retrieval Community 2: Data Mining Community 3: Machine Learning

74. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 74 Other Extensions (LDA Extensions) Many extensions of LDA, mostly done by David Blei, Andrew McCallum and their co-authors Some examples: –Hierarchical topic models [Blei et al. 03] –Modeling annotated data [Blei & Jordan 03] –Dynamic topic models [Blei & Lafferty 06] –Pachinko allocation [Li & McCallum 06]) Also, some specific context extension of PLSA, e.g., author-topic model [Steyvers et al. 04]

75. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 75 Future Research Directions Topic models for text mining –Evaluation of topic models –Improve the efficiency of estimation and inferences –Incorporate linguistic knowledge –Applications in new domains and for new tasks Text mining in general –Combination of NLP-style and DM-style mining algorithms –Integrated mining of text (unstructured) and unstructured data (e.g., Text OLAP) –Interactive mining: Incorporate user constraints and support iterative mining Design and implement mining languages

76. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 76 What You Should Know How PLSA works How EM algorithm works in general Contextual PLSA can be used to perform many quite different interesting text mining tasks

77. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 77 Roadmap This lecture: Topic models for text mining Next lecture: Next generation search engines