1. Modeling Diversity in Information Retrieval
ChengXiang (“Cheng”) Zhai
Department of Computer Science
Graduate School of Library & Information Science
Institute for Genomic Biology
Department of Statistics
University of Illinois, Urbana-Champaign
Joint work with John Lafferty, William Cohen, and Xuehua Shen
ACM SIGIR 2009 Workshop on Redundancy, Diversity, and Interdependent Document Relevance, July 23, 2009, Boston, MA

2. Different Reasons for Diversification
Redundancy reduction
Diverse information needs
Mixture of users
Single user with an under-specified query
Aspect retrieval
Overview of results
Active relevance feedback …

3. Outline Risk minimization framework
Capturing different needs for diversification
Language models for diversification

4. IR as Sequential Decision Making
(Information Need)
(Model of Information Need)
User
System
A1 : Enter a query
Which documents to present?
How to present them?
Which documents
to view?
Ri: results (i=1, 2, 3, …)
Which part of the document
to show? How?
A2 : View document
View more?
R’: Document content
A3 : Click on “Back” button

5. Retrieval Decisions Given U, C, At , and H, choose
the best Rt from all possible
responses to At
History H={(Ai,Ri)}
i=1, …, t-1
Query=“Jaguar”
User U: A1 A2 … … At At
System: R1 R2 … … Rt-1
Click on “Next” button
Rt  r(At)
Rt =?
The best ranking for the query
The best k unseen docs C
All possible rankings of C
Document
Collection
All possible size-k subsets of unseen docs

6. A Risk Minimization Framework
Observed
User Model
M=(S, U…)
Seen docs
Information need
User: U
Interaction history: H
Current user action: At
Document collection: C
Optimal response: r* (minimum loss)
All possible responses:
r(At)={r1, …, rn}
L(ri,At,M)
Loss Function
Inferred
Observed
Bayes risk

7. A Simplified Two-Step Decision-Making Procedure
Approximate the Bayes risk by the loss at the mode of the posterior distribution
Two-step procedure
Step 1: Compute an updated user model M* based on the currently available information
Step 2: Given M*, choose a response to minimize the loss function

8. Optimal Interactive Retrieval
User U C
Collection A1
M*1
P(M1|U,H,A1,C)
L(r,A1,M*1) R1 A2
M*2
P(M2|U,H,A2,C)
L(r,A2,M*2) R2 A3 …
IR system

9. Refinement of Risk Minimization
At {“enter a query”, “click on Back button”, “click on Next button, …}
r(At): decision space (At dependent)
r(At) = all possible subsets of C + presentation strategies
r(At) = all possible rankings of docs in C
r(At) = all possible rankings of unseen docs …
M: user model
Essential component: U = user information need
S = seen documents
n = “Topic is new to the user”
L(Rt ,At,M): loss function
Generally measures the utility of Rt for a user modeled as M
Often encodes retrieval criteria (e.g., using M to select a ranking of docs)
P(M|U, H, At, C): user model inference
Often involves estimating a unigram language model U

10. Generative Model of Document & Query [Lafferty & Zhai 01]
inferred
observed
Partially U
User q
Query R d
Document S
Source

11. Risk Minimization with Language Models [Lafferty & Zhai 01, Zhai & Lafferty 06]
Loss L
...
query q
user U
doc set C
source S q 1 N
Choice: (D1,1)
Choice: (D2,2)
Choice: (Dn,n)

12. Optimal Ranking for Independent Loss
Decision space = {rankings}
Sequential browsing
Independent loss
Independent risk
= independent scoring
“Risk ranking principle”
[Zhai 02, Zhai & Lafferty 06]

13. Risk Minimization for Diversification
Redundancy reduction: loss function includes a redundancy/novelty measure
Special case: list presentation + MMR [Zhai et al. 03]
Diverse information needs: loss function defined on latent topics
Special case: PLSA/LDA + aspect retrieval [Zhai 02]
Active relevance feedback: loss function considers both relevance and benefit for feedback
Special case: feedback only (hard queries) [Shen & Zhai 05]

14. Need to model interdependent document relevance
Subtopic Retrieval
Query: What are the applications of robotics in the world today?
Find as many DIFFERENT applications as possible.
Subtopic judgments
A1 A2 A3 … Ak
d …
d …
d … …. dk
Example subtopics:
A1: spot-welding robotics
A2: controlling inventory
A3: pipe-laying robots
A4: talking robot
A5: robots for loading & unloading
memory tapes
A6: robot [telephone] operators
A7: robot cranes
… …
Need to model interdependent document relevance

15. Diversify = Remove Redundancy [Zhai et al. 03]
Greedy Algorithm for Ranking: Maximal Marginal Relevance (MMR)
“Willingness to tolerate redundancy”
C2<C3, since a redundant relevant doc is
better than a non-relevant doc

16. A Mixture Model for Redundancy
P(w|Old)
Ref. document
1- =? 
P(w|Background)
Collection
p(New|d)= = probability of “new” (estimated using EM)
p(New|d) can also be estimated using KL-divergence

17. Evaluation metrics Intuitive goals: How do we quantify these?
Should see documents from many different subtopics appear early in a ranking (subtopic coverage/recall)
Should not see many different documents that cover the same subtopics (redundancy).
How do we quantify these?
One problem: the “intrinsic difficulty” of queries can vary.

18. Evaluation metrics: a proposal
Definition: Subtopic recall at rank K is the fraction of subtopics a so that one of d1,..,dK is relevant to a.
Definition: minRank(S,r) is the smallest rank K such that the ranking produced by IR system S has subtopic recall r at rank K.
Definition: Subtopic precision at recall level r for IR system S is:
This generalizes ordinary recall-precision metrics.
It does not explicitly penalize redundancy.

19. Evaluation metrics: rationale
precision
1.0
0.0 K
minRank(S,r)
For subtopics, the
minRank(Sopt,r) curve’s shape is not predictable and linear.
minRank(Sopt,r)
recall

20. Evaluating redundancy
Definition: the cost of a ranking d1,…,dK is
where b is cost of seeing document, a is cost of seeing a subtopic inside a document (before a=0).
Definition: minCost(S,r) is the minimal cost at which recall r is obtained.
Definition: weighted subtopic precision at r is
will use a=b=1

21. Evaluation Metrics Summary
Measure performance (size of ranking minRank, cost of ranking minCost) relative to optimal.
Generalizes ordinary precision/recall.
Possible problems:
Computing minRank, minCost is NP-hard!
A greedy approximation seems to work well for our data set

22. Experiment Design Dataset: TREC “interactive track” data.
London Financial Times: 210k docs, 500Mb
20 queries from TREC 6-8
Subtopics: average 20, min 7, max 56
Judged docs: average 40, min 5, max 100
Non-judged docs assumed not relevant to any subtopic.
Baseline: relevance-based ranking (using language models)
Two experiments
Ranking only relevant documents
Ranking all documents

23. S-Precision: re-ranking relevant docs

24. WS-precision: re-ranking relevant docs

25. Results for ranking all documents
“Upper bound”: use subtopic names to build an explicit subtopic model.

26. Summary: Remove Redundancy
Mixture model is effective for identifying novelty in relevant documents
Trading off novelty and relevance is hard
Relevance seems to be dominating factor in TREC interactive-track data

27. Diversity = Satisfy Diverse Info. Need [Zhai 02]
Need to directly model latent aspects and then optimize results based on aspect/topic matching
Reducing redundancy doesn’t ensure complete coverage of diverse aspects

28. Aspect Generative Model of Document & Query
User q
Query 
=( 1,…, k) S
Source d
Document
PLSI:
LDA:

29. Aspect Loss Function  U q S d

30. Aspect Loss Function: Illustration
New candidate
p(a|k)
non-relevant
redundant
perfect
Combined coverage
p(a|k)
Desired coverage
p(a|Q)
“Already covered”
p(a|1)... p(a|k -1)

31. Evaluation Measures
Aspect Coverage (AC): measures per-doc coverage
#distinct-aspects/#docs
Aspect Uniqueness(AU): measures redundancy
#distinct-aspects/#aspects
Examples 1 1 1
… ... d1 d2 d3
#doc … …
#asp … …
#uniq-asp
AC: /1= /2= /3=1.67
AU: /2= /5= /8=0.625

32. Effectiveness of Aspect Loss Function (PLSI)

33. Effectiveness of Aspect Loss Function (LDA)

34. Comparison of 4 MMR Methods
CC - Cost-based Combination
QB - Query Background Model
MQM - Query Marginal Model
MDM - Document Marginal Model

35. Summary: Diverse Information Need
Mixture model is effective for capturing latent topics
Direct modeling of latent aspects/topics is more effective than indirect modeling through MMR in improving aspect coverage, but MMR is better for improving aspect uniqueness
With direct topic modeling and matching, aspect coverage can be improved at the price of lower relevance-based precision

36. Diversify = Active Feedback [Shen & Zhai 05]
Decision problem:
Decide subset of documents for relevance judgment

37. Independent Loss
Independent Loss

38. Independent Loss (cont.)
Top K
Uncertainty Sampling

39. Dependent Loss … MMR K Cluster Centroid Gapped Top K
Heuristics: consider relevance
first, then diversity
Select Top N documents …
Cluster N docs into K clusters
MMR
K Cluster Centroid
Gapped Top K

40. Illustration of Three AF Methods
Top-K
(normal feedback) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 …
Gapped
Top-K
K-cluster centroid
Aiming at high diversity …

41. Evaluating Active Feedback
Select K
docs
K docs
Query
Initial
Results
No feedback
(Top-k, gapped, clustering)
Judgment
File +
Judged docs -
Feedback
Results

42. Retrieval Methods (Lemur toolkit)
Results
Kullback-Leibler Divergence
Scoring
Document D
Query Q
Feedback Docs
F={d1, …, dn}
Active Feedback
Mixture Model Feedback
Only learn from relevant docs
Default parameter settings
unless otherwise stated

43. Comparison of Three AF Methods
bold font = worst * = best
Collection
Active FB Method
#Rel
Include judged docs
MAP
HARD
Top-K
146
0.325
0.527
Gapped
150
0.330
0.548
Clustering
105
0.332
0.565
AP88-89
198
0.228
0.351
180
0.234*
0.389*
118
0.237
0.393
Top-K is the worst!
Clustering uses fewest relevant docs

44. Appropriate Evaluation of Active Feedback
Original DB
with judged docs
(AP88-89, HARD)
Original DB
without judged docs
New DB
(AP88-89, AP90) + + - - + +
Can’t tell if the ranking of un-judged documents
is improved
Different methods have
different test documents
See the learning effect
more explicitly
But the docs must be
similar to original docs

45. Comparison of Different Test Data
Top-K is consistently the worst!
Test Data
Active FB Method
#Rel
MAP
AP88-89
Including
judged docs
Top-K
198
0.228
0.351
Gapped
180
0.234
0.389
Clustering
118
0.237
0.393
AP90
0.220
0.321
0.222
0.326
0.223
0.325
Clustering generates fewer, but higher quality examples

46. Summary: Active Feedback
Presenting the top-k is not the best strategy
Clustering can generate fewer, higher quality feedback examples

47. Conclusions
There are many reasons for diversifying search results (redundancy, diverse information needs, active feedback)
Risk minimization framework can model all these cases of diversification
Different scenarios may need different techniques and different evaluation measures

48. References Risk Minimization
[Lafferty & Zhai 01] John Lafferty and ChengXiang Zhai. Document language models, query models, and risk minimization for information retrieval. In Proceedings of the ACM SIGIR 2001, pages
[Zhai & Lafferty 06] ChengXiang Zhai and John Lafferty, A risk minimization framework for information retrieval, Information Processing and Management, 42(1), Jan. 2006, pages
Subtopic Retrieval
[Zhai et al. 03] ChengXiang Zhai, William Cohen, and John Lafferty, Beyond Independent Relevance: Methods and Evaluation Metrics for Subtopic Retrieval, In Proceedings of ACM SIGIR 2003.
[Zhai 02] ChengXiang Zhai, Language Modeling and Risk Minimization in Text Retrieval, Ph.D. thesis, Carnegie Mellon University, 2002.
Active Feedback
[Shen & Zhai 05] Xuehua Shen, ChengXiang Zhai, Active Feedback in Ad Hoc Information Retrieval, Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval ( SIGIR'05), 59-66, 2005
ACM SIGIR 2009 Workshop on Redundancy, Diversity, and Interdependent Document Relevance, July 23, 2009, Boston, MA

49. Thank You!