1. Relevance and Reinforcement in Interactive Browsing
Anton Leuski
Proceedings of CIKM'00, Washington, DC November 6-11, pp
Summarized by Seung-Joon Yi

2. (C) 2001, SNU Biointelligence Lab, http://bi.snu.ac.kr/
Introduction
The document selection procedure can be implemented as a “wizard” that comes up right after the documents are retrieved.
The wizard examines the relevance values of each unexamined document and highlight the most likely to be relevant documents
As the user examines the documents and marks them as relevant or not, the wizard reevaluates its estimations and changes the highlighting accordingly.
(C) 2001, SNU Biointelligence Lab, 

3. Introduction(cont’d)
This paper concentrate on analysis of documents that are already present in the retrieved set.
In addition, this paper focues only in the inter-document similarity information obtained after the original retrieval session and ignores all term-level statistics.
The feedback problem is formulated in terms of reinforcement learning.
(C) 2001, SNU Biointelligence Lab, 

4. (C) 2001, SNU Biointelligence Lab, http://bi.snu.ac.kr/
Related Work
Relevance feedback
Rocchio’s algorithm
Incremental relevance feedback
Connectionist approach
(C) 2001, SNU Biointelligence Lab, 

5. (C) 2001, SNU Biointelligence Lab, http://bi.snu.ac.kr/
Search Strategy
The problem of navigating the retrieved document set can be naturally expressed as a reinforcement learning problem.
Environment State Dt:defined by the inter-document similarities, what documents were examined, and what relevance jugments were assigned.
Action d: next document to examine.
Reward:whether the examined document is relevant or not.
Goal:find all relevant documents ASAP.
(C) 2001, SNU Biointelligence Lab, 

6. Search Strategy(cont’d)
Agent implementation is defined at each time step by a mapping between a state representation combined with an action and a numeric value, F(Dt,d) (search strategy function)
Agent computes the mapping for each unexamined document and selects d with the highest value of F(Dt,d)
TD-learning learning rule
(C) 2001, SNU Biointelligence Lab, 

7. (C) 2001, SNU Biointelligence Lab, http://bi.snu.ac.kr/
Simple Rocchio
Design search strategy function F1(Dt,d) as a single perceptron unit that has four inputs:bias or constant input, document similarity to query, average similarity between the document and all examined relevant documents, and average similarity between the document and all examined non-relevant documents.
(C) 2001, SNU Biointelligence Lab, 

8. Application Coefficients
Design secondary search strategy function F2(Dt,d) as a linear combination of three instances of the search strategy function from the previous section (F1(Dt,d) ), where the coefficients-called application coefficients-are smooth functions of the number of the documents.
(C) 2001, SNU Biointelligence Lab, 

9. (C) 2001, SNU Biointelligence Lab, http://bi.snu.ac.kr/
Tile Coding
The feature space is partitioned with a regular grid and a single number is assigned to each cell in the partition. The set of tilings define the final function F3(Dt,d) :given a point in the feature space, a tile containing the point is selected from each grid and the average of the corresponding numbers is returned.
Can approximate more complex functions.
Five dimensions(4 of them are the same feature that used in F2, and the 5th feature is the number of examined documents squared)
(C) 2001, SNU Biointelligence Lab, 

10. (C) 2001, SNU Biointelligence Lab, http://bi.snu.ac.kr/
Experiments
Retrieved document sets for experiments are generated by running the Inquery retrieval engine on two standard TREC collections.
The engine assigns what is called a belief score to each document in the collection, which is used as the query-document similarity value.
(C) 2001, SNU Biointelligence Lab, 

11. Document Representation
To compute the inter-document similarities, a vector-space approach where each document is represented by a vector term V is used.
Weight of the ith term is computed using the Inquery weighting fomula.
The inter-document similarity is measured by the cosine of the angle between two vectors.
(C) 2001, SNU Biointelligence Lab, 

12. (C) 2001, SNU Biointelligence Lab, http://bi.snu.ac.kr/
Experimental setup
TREC ad-hoc queries with their corresponding collections and the relevance judgments supplied bu NIST accessors.
For each TREC topic
A query constructed by extensive analysis and expansion
The description foeld of the topic
The title of the topic
A query constructed from the title by expanding it using Local Text Analysis(LCA)
(C) 2001, SNU Biointelligence Lab, 

13. Training and Evaluation Procedure
Model:a situation where a user located the first relevant document by following the ranked list.
Experimental task:given the highest ranked relevant document as the starting point, find the rest of the relevant information.
Perforcemece measure:Average precision on the unexamined portion of the document set.
Data sets:training,testing,evaluation subsets from 8 data sets-one for each query type on each collection
(C) 2001, SNU Biointelligence Lab, 

14. Training and Evaluation Procedure(cont’d)
Parameters
Each search strategy function began with all parameters initialized to zero.
Learning rate η:0.1
Discount factor ρ:0.4
Application coefficients μ:1,25,50 and σ=6.
Tiles: 256 tiles.
The learning process terminated when the average precision failed to improve for several iterations.
(C) 2001, SNU Biointelligence Lab, 

15. Baseline Evaluation Procedure
Starting with the ranked list, follow it until the first relevant document is found.
At that point all examined documents analyzed and a new query is created by expanding the old query with several top ranked terms from the examined documents.
The rest of the unexamined documents are re-ordered using the modified query and the process continues until all documents are examined.
(C) 2001, SNU Biointelligence Lab, 

16. (C) 2001, SNU Biointelligence Lab, http://bi.snu.ac.kr/
Results
1st strategy: 5% improvement
2nd strategy:9% improvement
3rd strategy:10% improvement
(C) 2001, SNU Biointelligence Lab, 

17. (C) 2001, SNU Biointelligence Lab, http://bi.snu.ac.kr/
Results
Used more documents(100)
Similar improvements
(C) 2001, SNU Biointelligence Lab, 

18. (C) 2001, SNU Biointelligence Lab, http://bi.snu.ac.kr/
Results
No starting point case
The search strategies were to start without any relevance information and to explore the whole retrieved set.
Small but significant improvement
(C) 2001, SNU Biointelligence Lab, 

19. (C) 2001, SNU Biointelligence Lab, http://bi.snu.ac.kr/
Discussions
Reward function
Using relevance value:RL maximizes total discounted relevance
In this paper, perforce was evaluated using the average precision
Alternative reward function:use P at the end of the search and all intermediate reward set to zero
Learned similar search strategy
(C) 2001, SNU Biointelligence Lab, 

20. (C) 2001, SNU Biointelligence Lab, http://bi.snu.ac.kr/
Conclusions
Formalized the relevance feedback problem in terms of reinforcement learning.
The technique is very successful when only the inter-document similarity data is available and no term information is provided.
(C) 2001, SNU Biointelligence Lab,