Information Retrieval Performance Measurement Using Extrapolated Precision William C. Dimm DESI VI June 8, 2015

Comparing Precision-Recall Curves

What if we only know one point?

Comparing Precision-Recall Points

F 1 Comparison: Wrong Conclusion!

F 1 Depends Strongly on Recall

F 1 Contours

Danger Zone

Want Contours Like P-R Curves

Less Cutting Across Contours

How to Quantify?

Precision's Relationship with Cost ● Precision is meaningful – inversely proportional to number of docs to review: n = ρNR/P

Extrapolated Precision

X is Fairly Independent of R

Precision-Recall Curve Math

Actual Precision-Recall Curves

Actual Probability Curves

Model Probability vs Actual

Model Precision vs Actual

Extrapolation Limitations ● P < 0.99 ● R < 0.99 ● P >= 2ρ / (1 + ρ + R*(1 - ρ))

Summary ● Proportionality dictates recall – Need performance measure less sensitive to recall ● Extrapolate precision-recall point to target recall level using model curves ● Model precision-recall curves are constant performance contours ● When close to target recall, performance measure inversely proportional to review cost