1. Carnegie Mellon 1 Maximum Likelihood Estimation for Information Thresholding Yi Zhang & Jamie Callan Carnegie Mellon University {yiz,callan}@cs.cmu.edu

2. 2 Overview u Adaptive filtering: definition and challenges u Threshold based on score distribution and the sampling bias problem u Maximum likelihood estimation for score distribution parameters u Results of Experiments u Conclusion

3. 3 Given an initial description of information needs, a filtering system sifts through a stream of documents,and delivers relevant documents to a user as soon as the document arrives. Relevance feedback maybe available for some of the delivered documents, thus user profiles can be updated adaptively. Filtering System Adaptive Filtering

4. 4 u Three major problems l Learning corpus statistics, such as idf l Learning user profile, such as adding or deleting key words and adjusting term weights. (Scoring method) l Learning delivery threshold. (Binary judgment) u Evaluation Measures l Linear utility = r1*RR+r2*NR+r3*RN+r4*NN Optimizing linear utility => Finding P(relevant|document) In one dimension: P(relevant|document) = P(relevant|score) l F measure

5. 5 A Model of Score Distribution: Assumptions and Empirical Justification u Relevant: u Non-relevant: u According to other researchers, this is generally true for various statistical searching systems (scoring methods, Manmatha’s paper, Arampatzis’s paper) Figure 1. Density of document scores: TREC9 OHSU Topic 3 and Topic 5

6. 6 Optimize for Linear Utility Measure: from Score Distribution to Probability of Relevancy u p: p(r) ratio of relevant documents

7. 7 Optimize for F Measure: From Score Distribution to Precision and Recall If set threshold at θ:

8. 8 What We Have Now? u A model for score distribution u Algorithms to find the optimal threshold for different evaluation measures given the model u Learning task: find the parameters for the model?

9. 9 Bias Problem for Parameter Estimation while Filtering u We only receive feedback for documents delivered u Parameter estimation based on random sampling assumption is biased u Sampling criteria depends on threshold, which changes over time u Solution: maximum likelihood principle, which is guaranteed to be unbiased Figure: Estimation of parameters for relevant document scores of TREC9 OHSU Topic 3 with a fixed dissemination threshold 0.4435

10. 10 Unbiased Estimation of Parameters Based on Maximum Likelihood Principle (1) ML: the best estimation of parameters is the one that maximizes the probability of training data:

11. 11 Unbiased Estimation of Parameters Based on Maximum Likelihood Principle (2) For each item inside the sum operation of the previous formula:

12. 12 Unbiased Estimation of Parameters Based on Maximum Likelihood Principle (3) Calculating the denominator:

13. 13 Unbiased Estimation of Parameters Based on Maximum Likelihood Principle (4) For a relevant document delivered: For a non-relevant document delivered:

14. 14 Relationship to Arampatzis’s Estimation If no threshold exists The previous formula becomes: For a relevant document delivered: For a non-relevant document delivered: Corresponding result will be the same as Arampatzis’s

15. 15 Unbiased Estimation of Parameters Based on Maximum Likelihood Principle (5) u Optimization using conjugate gradient descent algorithm u Smoothing using conjugate prior: l Prior for p: beta distribution: l Prior for variance: l Set:

16. 16 Experimental Methodology (1) u Optimization goal (similar to the measure used by TREC9): T9U’=2*Relevant_Retrieved-Non_Relevant_Retrieved=2RR-NR Corresponding rule: deliver if : u Dataset l OHSUMED data (348566 articles from 1887 to 1991. 63 OHSUMED queries and 500 MeSH headings to simulate user profiles) l FT data (210158 articles from Financial Times 1991 to 1994. TREC topics 351-400 to simulate user profiles) u Each profile begins with 2 relevant documents and an initial user profile u No profile updating for simplicity.

17. 17 Experimental Methodology (2) u Four runs for each profile l Run1 : biased estimation of parameters because sampling bias was not considered l Run3 : maximum likelihood estimation. Both runs will stop delivering documents if the threshold is set too high, especially in the early stages of filtering. We introduced a minimum delivery ratio: If a profile has not achieved the minimum delivery ratio, its threshold will be decreased automatically: l Run 2: biased estimation + minimum delivery ratio l Run 4: maximum likelihood estimation + minimum delivery ratio u Time: 21 minutes for the whole process of 63 OHSU topics on 4 years of OHSUMED data (ML algorithm)

18. 18 Results: OHSUMED Data Run 1: Biased estimation Run 2: Biased estimation+ min. delivery Ratio Run 3: Unbiased estimation Run4:Unbiased estimation+min. delivery ratio OHSU topics T9U’ utility1.84 3.25 2.7 8.17 Avg. docs. delivered per profile 3.839.655.7318.40 Precision0.370.290.360.32 Recall0.0360.0800.0520.137 MESH topics T9U’ utility1.89 4.28 2.44 13.10 Avg. docs. delivered per profile 3.5111.826.2227.91 Precision0.420.390.400.34 Recall0.0180.0460.0250.068

19. 19 Results: Financial Times Run 1: Biased estimation Run 2: Biased estimation + min. delivery ratio Run 3: Unbiased estimation Run 4: Unbiased estimation + min. delivery ratio T9U’ utility 1.44 -0.209 0.65 0.84 Avg. docs. Delivered per profile 9.5810.449.0512.27 Precision 0.200.170.220.26 Recall 0.1610.1670.150.193

20. 20 Result Analysis: Difference Between Run 4 and Run 2 on TREC9 OHSU Topics For most of the topics, ML (Run 4) delivered more documents than Run 2 For some of the topics, ML (run 4) has a much higher utility than Run 2, while they are similar in most of the other topics Utility: ML - Biased Docs delivered:ML -Biased Topics

21. 21 Conclusion u Score density distribution l Relevant documents: normal distribution l Non-relevant documents: exponential distribution u Bias problem due to non-random sampling can be solved based on the maximum likelihood principle u Significant improvement in the TREC-9 filtering task. u Future work l Thresholding while updating profiles l Non-random sampling problem in other task