1. Information Retrieval Performance Measurement Using Extrapolated Precision William C. Dimm DESI VI June 8, 2015

2. Comparing Precision-Recall Curves

3. What if we only know one point?

4. Comparing Precision-Recall Points

5. F 1 Comparison: Wrong Conclusion!

6. F 1 Depends Strongly on Recall

7. F 1 Contours

9. Danger Zone

11. Want Contours Like P-R Curves

12. Less Cutting Across Contours

13. How to Quantify?

14. Precision's Relationship with Cost ● Precision is meaningful – inversely proportional to number of docs to review: n = ρNR/P

15. Extrapolated Precision

16. X is Fairly Independent of R

17. Precision-Recall Curve Math

18. Actual Precision-Recall Curves

19. Actual Probability Curves

20. Model Probability vs Actual

21. Model Precision vs Actual

22. Extrapolation Limitations ● P < 0.99 ● R < 0.99 ● P >= 2ρ / (1 + ρ + R*(1 - ρ))

23. Summary ● Proportionality dictates recall – Need performance measure less sensitive to recall ● Extrapolate precision-recall point to target recall level using model curves ● Model precision-recall curves are constant performance contours ● When close to target recall, performance measure inversely proportional to review cost