Context Analysis in Text Mining and Search

Context Analysis in Text Mining and Search
Qiaozhu Mei Department of Computer Science University of Illinois at Urbana-Champaign Joint work with ChengXiang Zhai 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Motivating Example: Personalized Search
MSR Metropolis Street Racer Magnetic Stripe Reader Molten salt reactor Mars Sample Return … Mountain safety research Actually Looking for Microsoft Research… 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Motivating Example: Comparing Product Reviews
IBM Laptop Reviews APPLE Laptop Reviews DELL Laptop Reviews Common Themes “IBM” specific “APPLE” specific “DELL” specific Battery Life Long, 4-3 hrs Medium, 3-2 hrs Short, 2-1 hrs Hard disk Large, GB Small, 5-10 GB Medium, GB Speed Slow, Mhz Very Fast, 3-4 Ghz Moderate, 1-2 Ghz Unsupervised discovery of common topics and their variations 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Motivating Example: Discovering Topical Trends in Literature
SIGIR topics Topic Strength Time Explain the Plots. Temporal theme analysis separate to make the fonts bigger… more explanations. Title: sample ETP: theme evolutionary graph 1980 1990 1998 2003 TF-IDF Retrieval Language Model IR Applications Text Categorization Unsupervised discovery of topics and their temporal variations 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Motivating Example: Analyzing Spatial Topic Patterns
How do bloggers in different states respond to topics such as “oil price increase during Hurricane Karina”? Unsupervised discovery of topics and their variations in different locations 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Motivating Example: Summarizing Sentiments
Query: Dell Laptops Topic-sentiment summary positive negative Facet 2 (Battery) Facet 1 (Price) neutral my Dell battery sucks Stupid Dell laptop battery One thing I really like about this Dell battery is the Express Charge feature. i still want a free battery from dell.. …… it is the best site and they show Dell coupon code as early as possible Even though Dell's price is cheaper, we still don't want it. mac pro vs. dell precision: a price comparis.. DELL is trading at $24.66 time strength Positive Negative Topic-sentiment dynamics (Topic = Price) Neutral Unsupervised/Semi-supervised discovery of topics and different sentiments of the topics 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Motivating Example: Analyzing Topics on a Social Network
Bruce Croft Publications of Gerard Salton Publications of Bruce Croft Information retrieval Machine learning Data mining Gerard Salton Unsupervised discovery of topics and correlated research communities 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Research Questions What are these problems in common?
Can we model all these problems generally? Can we solve these problems with a unified approach? How can we bring human into the loop? 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Rest of Talk Background: Language Models in Text Mining and Retrieval
Definition of context General methodology to model context Models, example applications, results Conclusion and Discussion 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Generative Models of Text
Text as observations: words; tags; links, etc Use a unified probabilistic model to explain the appearance (generation) of observations Documents are generated by sampling every observation from such a generative model Different generation assumption  different model Document Language Models Probabilistic Topic Models: PLSA, LDA, etc. Hidden Markov Models … 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Multinomial Language Models
A multinomial distribution of words as a text representation retrieval information 0.15 model query language feedback …… Known as a Topic model when there are k of them in text: e.g., semi-supervised learning; boosting; spectral clustering, etc. 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Language Models in Information Retrieval (e.g., KL-Div. Method)
Doc Language Model (LM) θd : p(w|d) text 4/100=0.04 mining 3/100=0.03 clustering 1/100=0.01 … data = 0 computing = 0 … Smoothed Doc LM θd' : p(w|d’) Document d text =0.039 mining =0.028 clustering =0.01 … data = computing = … A text mining paper Similarity function Data ½=0.5 Mining ½=0.5 Query Language Model θq : p(w|q) Query q Data ½=0.4 Mining ½=0.4 Clustering =0.1 … ? p(w|q’) data mining 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Probabilistic Topic Models for Text Mining
term relevance weight feedback independ model … Topic models (Multinomial distributions) Text Collections Probabilistic Topic Modeling Subtopic discovery Opinion comparison Summarization Topical pattern analysis … Passage segmentation PLSA [Hofmann 99] LDA [Blei et al. 03] Author-Topic [Steyvers et al. 04] CPLSA [Mei & Zhai 06] … Pachinko allocation [Li & McCallum 06] CTM [Blei et al. 06] … web search link graph … … 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Importance of Context Science in the year 2000 and Science in the year 1500: Are we still working on the same topics? For a computer scientist and a gardener: Does “tree, root, prune” mean the same? “Football” means soccer in Europe. What about in US? Context affects topics! 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Context Features of Text (Meta-data)
Weblog Article communities Author Compared with other kinds of data, Weblogs have some interesting special characteristics, which make it interesting to exploit for text mining. source Location Time Author’s Occupation 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Context = Partitioning of Text
Papers about Web papers written in 1998 1998 papers written by authors in US 1999 …… …… 2005 2006 WWW SIGIR ACL KDD SIGMOD 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Rich Context Information in Text
News articles: time, publisher, etc. Blogs: time, location, author, … Scientific Literature: author, publication year, conference, citations, … Query Logs: time, IP address, user, clicks, … Customer reviews: product, source, time, sentiments.. s: sender, receiver, time, thread, … Web pages: domain, time, click rate, etc. More? entity-relations, social networks, …… 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Categories of Context Some partitions of text are explicit  explicit context Time; location; author; conference; user; IP; etc Similar to metadata Some partitions are implicit  implicit context Sentiments; missions; goals; intents; Some partitions are at document level Some are at a finer granularity Context of a word; an entity; a pattern; a query, etc. Sentences; sliding windows; adjacent words; etc 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Context Analysis Use context to infer semantics
Annotating frequent patterns; labeling of topic models Use context to provide targeted service Personalized search; intent-based search; etc. Compare contextual patterns of topics Evolutionary topic patterns; spatiotemporal topic patterns; topic-sentiment patterns; etc. Use context to help other tasks Social network analysis; impact summarization; etc. 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

General Methodology to Model Context
Context  Generative Model Observations in the same context are generated with a unified model Observations in different contexts are generated with different models Observations in similar contexts are generated with similar models Text is generated with a mixture of such generative models Example Task; Model; Sample results 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Model a unique context with a unified model (Generation)
2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Probabilistic Latent Semantic Analysis (Hofmann ’99)
Documents about “Hurricane Katrina” Topics θ1…k government donation New Orleans P(w|θj) Draw a word from i A Document d Criticism of government response to the hurricane primarily consisted of criticism of its response to … The total shut-in oil production from the Gulf of Mexico … approximately 24% of the annual production and the shut-in gas production … Over seventy countries pledged monetary donations or other assistance. … government 0.3 response donate 0.1 relief 0.05 help city 0.2 new orleans government response donate πd : P(θi|d) help aid Orleans new Choose a topic N D Wd,n θk Zd,n πd K πd θk 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Latent Dirichlet Allocation (Blei ‘03)
PLSA: no natural way to assign probability to a unseen document. Number of parameters grow linearly with size of training set  overfits data. Not a fully generative model. LDA solves these problems But need to inference p(topic|d) and p(w|topic) Parameter estimation using Gibbs Sampling or variational inference 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Example: Topics in Science (D. Blei 05)

Label a Multinomial Topic Model
Semantically close (relevance) Understandable – phrases? High coverage inside topic Discriminative across topics Retrieval models term relevance weight feedback independence model frequent probabilistic document … iPod Nano じょうほうけんさく Pseudo-feedback Mei and Zhai 06: a topic in SIGIR Information Retrieval 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Automatic Labeling of Topics
NLP Chunker Ngram Stat. information retrieval, retrieval model, index structure, relevance feedback, … Candidate label pool 1 Collection (e.g., SIGIR) term relevance weight feedback independence 0.03 model … Discrimination 3 information retriev retrieval models IR models pseudo feedback …… Relevance Score Information retrieval retrieval models IR models pseudo feedback …… 2 4 Coverage retrieval models IR models pseudo feedback …… information retrieval 0.01 filtering collaborative … trec evaluation … 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Label Relevance: Context Comparison
Intuition: expect the label with similar context (distribution) Clustering dimension partition algorithm hash Topic  … P(w|) Clustering hash dimension algorithm partition … p(w | clustering algorithm ) Good Label (l1) “clustering algorithm” l2: “hash join” Clustering hash dimension key algorithm … p(w | hash join) key …hash join … code …hash table …search …hash join… map key…hash …algorithm…key …hash…key table…join… Score (l,  ) = D(||l) 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Results: Sample Topic Labels
north case trial iran documents walsh reagan charges the, of, a, and, to, data, > 0.02 … clustering time clusters databases large performance 0.01 quality iran contra … clustering algorithm clustering structure … tree trees spatial b r disk array cache r tree b tree … large data, data quality, high data, data application, … indexing methods 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Model different contexts with different models (Discrimination, Comparison)

Example: Finding Evolutionary Patterns of Topics
1999 2000 2001 2002 2003 2004 T KDD web classifica –tion features0.006 topic … SVM criteria classifica – tion linear … mixture random cluster clustering variables … topic mixture LDA semantic … decision tree classifier class Bayes … … Classifica - tion text unlabeled document labeled learning … Informa - tion web social retrieval distance networks 0.004 … Content Variations over Contexts … … 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Example: Finding Evolutionary Patterns of Topics (II)
Figure from (Mei ‘05) Strength Variations over Contexts 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

View of Topics: Context-Specific Version of Views
Context 1: 1998 ~ 2006 (e.g. After “Language Modeling”) One context  one view A document selects from a mix of views language model smoothing query generation feedback mixture estimate EM pseudo vector Rocchio weighting feedback term space TF-IDF Okapi LSI retrieval Topic 1: Retrieval Model retrieve model relevance document query feedback judge expansion pseudo query Topic 2: Feedback Context 2: 1977 ~ 1998 (i.e. Before “Language Modeling”) 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Coverage of Topics: Distribution over Topics
Criticism of government response to the hurricane primarily consisted of criticism of its response to … The total shut-in oil production from the Gulf of Mexico … approximately 24% of the annual production and the shut-in gas production … Over seventy countries pledged monetary donations or other assistance. … Oil Price Government Response Aid and donation Background Context: Texas A coverage of topics: a (strength) distribution over the topics. One context  one coverage A document selects from a mix of multiple coverages. Oil Price Government Response Aid and donation Background Context: Louisiana 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

A General Solution: CPLSA
CPLAS = Contextual Probabilistic Latent Semantic Analysis An extension of PLSA model ([Hofmann 99]) by Introducing context variables Modeling views of topics Modeling coverage variations of topics Process of contextual text mining Instantiation of CPLSA (context, views, coverage) Fit the model to text data (EM algorithm) Compare a topic from different views Compute strength dynamics of topics from coverages Compute other probabilistic topic patterns 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

The “Generation” Process
Choose a theme View1 View2 View3 Texas July 2005 sociologist Topics government donation New Orleans Draw a word from i Criticism of government response to the hurricane primarily consisted of criticism of its response to … The total shut-in oil production from the Gulf of Mexico … approximately 24% of the annual production and the shut-in gas production … Over seventy countries pledged monetary donations or other assistance. … Context of Document: Time = July 2005 Location = Texas Author = Eric Brill Occup. = Sociologist Age = 45+ … government 0.3 response donate 0.1 relief 0.05 help city 0.2 new orleans government response donate help aid Orleans new Choose a view Topic coverages: Texas July 2005 document …… sociologist Choose a Coverage 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

An Intuitive Example Two topics: web search; machine learning
I am writing a WWW paper.  I will cover more about “web search” instead of “machine learning”. But of course I have my own taste. I am from a search engine company, so when I write about “web search”, I will focus on “search engine” and “online advertisements”… Coverage donate 0.1 relief 0.05 help city 0.2 new orleans View 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

The Probabilistic Model
A probabilistic model explaining the generation of a document D and its context features C: if an author wants to write such a document, he will Choose a view vi according to the view distribution Choose a coverage кj according to the coverage distribution . Choose a theme according to the coverage кj . Generate a word using . The likelihood of the document collection is: 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Example results: Query Log Analysis Context = Days of week
Query & Clicks: more query/clicks on weekdays Search Difficulty: more difficult to predict on weekends

Query Log Analysis Context = Type of Query
Business Queries: clear day-week pattern; weekdays more frequent than weekends Consumer Queries: no clear day-week pattern; weekends are comparable, even more frequent than weekdays

Bursting Topics in SIGMOD: Context = Time (Years)

Spatiotemporal Text Mining: Context = Time & Location
Week4: The theme is again strong along the east coast and the Gulf of Mexico Week3: The theme distributes more uniformly over the states Week2: The discussion moves towards the north and west Week5: The theme fades out in most states Week1: The theme is the strongest along the Gulf of Mexico About Government Response in Hurricane Katrina 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Faceted Opinions Context = Sentiments
Neutral Positive Negative Topic 1: Movie ... Ron Howards selection of Tom Hanks to play Robert Langdon. Tom Hanks stars in the movie,who can be mad at that? But the movie might get delayed, and even killed off if he loses. Directed by: Ron Howard Writing credits: Akiva Goldsman ... Tom Hanks, who is my favorite movie star act the leading role. protesting ... will lose your faith by ... watching the movie. After watching the movie I went online and some research on ... Anybody is interested in it? ... so sick of people making such a big deal about a FICTION book and movie. Topic 2: Book I remembered when i first read the book, I finished the book in two days. Awesome book. I’m reading “Da Vinci Code” now. … So still a good book to past time. This controversy book cause lots conflict in west society. Can click on the cells to get to the original articles.. 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Sentiment Dynamics Context = Time & Sentiments
“ the da vinci code” Facet: the book “ the da vinci code”. ( Bursts during the movie, Pos > Neg ) Facet: the impact on religious beliefs. ( Bursts during the movie, Neg > Pos ) 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Event Impact Analysis: IR Research
xml model collect judgment rank subtopic … vector concept extend model space boolean function feedback … Publication of the paper “A language modeling approach to information retrieval” Starting of the TREC conferences year 1992 term relevance weight feedback independence model frequent probabilistic document … Theme: retrieval models SIGIR papers 1998 model language estimate parameter distribution probable smooth markov likelihood … probabilist model logic ir boolean algebra estimate weight … 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Model similar context with similar models (Smoothing, Regularization)

Personalization with Backoff
Ambiguous query: MSG Madison Square Garden Monosodium Glutamate Disambiguate based on user’s prior clicks We don’t have enough data for everyone! Backoff to classes of users Proof of Concept: Classes defined by IP addresses Better: Market Segmentation (Demographics) Collaborative Filtering (Other users who click like me)

Context = IP Full personalization: every context has a different model: sparse data! * Personalization with backoff: similar contexts have similar models *.* 156.*.*.* *.*.*.* No personalization: all contexts share the same model 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Backing Off by IP Sparse Data Missed Opportunity
λs estimated with EM and CV A little bit of personalization Better than too much Or too little λ4 : weights for first 4 bytes of IP λ3 : weights for first 3 bytes of IP λ2 : weights for first 2 bytes of IP ……

Social Network as Correlated Contexts
Linked contexts are similar to each other Optimization of Relevance Feedback Weights Parallel Architecture in IR ... Predicting query performance … A Language Modeling Approach to Information Retrieval ... 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Social Network Context for Topic Modeling
e.g. coauthor network Context = author Coauthor = similar contexts Intuition: I work on similar topics to my neighbors Smoothed Topic distributions over context  2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Topic Modeling with Network Regularization (NetPLSA)
Basic Assumption (e.g., co-author graph) Related authors work on similar topics PLSA topic distribution of a document tradeoff between topic and smoothness difference of topic distribution on neighbor vertices Graph Harmonic Regularizer, Generalization of [Zhu ’03], importance (weight) of an edge 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Topical Communities with PLSA
term peer visual interface question patterns analog towards protein mining neurons browsing training clusters vlsi xml weighting stream motion generation 0.01 multiple frequent 0.01 chip design recognition 0.01 e natural engine relations page cortex service library gene spike social Noisy community assignment ? ? ? ? 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Topical Communities with NetPLSA
retrieval mining neural web information data learning services document discovery 0.03 networks semantic query databases 0.02 recognition 0.02 services text rules analog peer search association 0.02 vlsi ontologies evaluation 0.02 patterns neurons rdf user frequent gaussian management 0.01 relevance streams network ontology Web Coherent community assignment Data mining Information Retrieval Machine learning 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Smoothed Topic Map Map a topic on the network (e.g., using p(θ|a))
Core contributors Intermediate Irrelevant NetPLSA PLSA (Topic : “information retrieval”) 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Smoothed Topic Map The Windy States Blog articles: “weather”
NetPLSA PLSA The Windy States Blog articles: “weather” US states network: Topic: “windy” Real reference 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Related Work Specific Contextual Text Mining Problems
Multi-collection Comparative Mining (e.g., [Zhai et al. 04]) Temporal theme pattern (e.g., [Mei et al. 05], [Blei et al. 06], [Wang et al. 06]) Spatiotemporal theme analysis (e.g., [Mei et al. 06], [Wang et al. 07]) Author-topic analysis (e.g., [Steyvers et al. 04], [Zhou et al 06]) … Probabilistic topic models: Probabilistic latent semantic analysis (PLSA) (e.g. [Hofmann 99]) Latent Dirichlet allocation (LDA) (e.g., [Blei et al. 03]) Many extensions (e.g., [Blei et al. 05], [Li and McCallum 06]) 2007 © ChengXiang Zhai LLNL, Aug 15, 2007

Conclusions Context analysis in text mining and search
General methodology to model context in text A unified generative model for observations in the same context Different models for different context Similar models for similar contexts Generation  discrimination  smoothing Many applications 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Discussion: Context in Search
Not all contexts are useful E.g. personalized search v.s. search by time of day How can we know which contexts are more useful? Many contexts are useful E.g., personalized search; task-based search; localized search; How can we combine them? Can we do better than market segmentations? Backoff to users who search like me – Collaborative Search But who searches like you? 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

References CPLSA NetPLSA Labeling Personalization: Applications:
Q. Mei, C. Zhai. A Mixture Model for Contextual Text Mining, In Proceedings of KDD' 06. NetPLSA Q. Mei, D. Cai, D. Zhang, C. Zhai, Topic Modeling with Network Reguarization, Proceedings of WWW’ 08 Labeling Q. Mei, X.Shen, C. Zhai, Automatic Labeling of Multinomial Topic Models, Proceedings KDD'07 Personalization: Q.Mei, K.Church, Entropy of Search Logs: How Hard is Search? With Personalization? With Backoff? In Proceedings of WSDM’08. Applications: Q. Mei, C. Zhai, Discovering Evolutionary Theme Patterns from Text - An Exploration of Temporal Text Mining, In Proceedings KDD' 05 Q. Mei, C. Liu, H. Su, and C. Zhai, A Probabilistic Approach to Spatiotemporal Theme Pattern Mining on Weblogs, In Proceedings of WWW' 06 Q. Mei, X. Ling, M. Wondra, H. Su, C. Zhai, Topic Sentiment Mixture: Modeling Facets and Opinions in Weblogs, Proceedings of WWW’ 07 2007 © ChengXiang Zhai LLNL, Aug 15, 2007

Experiments Bibliography data and coauthor networks
DBLP: text = titles; network = coauthors Four conferences (expect 4 topics): SIGIR, KDD, NIPS, WWW Blog articles and Geographic network Blogs from spaces.live.com containing topical words, e.g. “weather” Network: US states (adjacent states) 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Coherent Topical Communities
PLSA visual analog neurons vlsi motion chip natural cortex spike PLSA peer patterns mining clusters stream frequent 0.01 e page gene NetPLSA neural learning networks recognition 0.02 analog vlsi neurons gaussian network NetPLSA mining data discovery 0.03 databases 0.02 rules association 0.02 patterns frequent streams Semantics of community: “machine learning (NIPS)” Semantics of community: “Data Mining (KDD) ” 2008 © Qiaozhu Mei University of Illinois at Urbana-Champaign

Context Analysis in Text Mining and Search

Similar presentations

Presentation on theme: "Context Analysis in Text Mining and Search"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Context Analysis in Text Mining and Search

Similar presentations

Presentation on theme: "Context Analysis in Text Mining and Search"— Presentation transcript:

Similar presentations

About project

Feedback