Presentation is loading. Please wait.

Presentation is loading. Please wait.

Modern Information Retrieval

Similar presentations


Presentation on theme: "Modern Information Retrieval"— Presentation transcript:

1 Modern Information Retrieval
Chapter 3 Retrieval Evaluation

2 The most common measures of system performance are time and space
an inherent tradeoff Data retrieval time and space indexing Information retrieval precision of the answer set also important

3

4 evaluation considerations
query with/without feedback query interface design real data/synthetic data real life/laboratory environment repeatability and scalability

5 recall and precision recall: fraction of relevant documents which has been retrieved precision: fraction of retrieved documents which is relevant

6 can we precisely compute precisions? can we precisely compute recalls?

7 precision versus recall curve: a standard evaluation strategy

8

9 interpolation procedure for generating the 11 standard recall levels
Rq={d3,d56,d129} where j is in {0,1,2,…,10} and P(r) is a known precision

10

11 to evaluate the retrieval strategy over all test queries, the precisions at each recall level are averaged

12 another approach: compute average precision at given relevant document cutoff values
advantages?

13 single value summary for each query
average precision at seen relevant documents example in Figure 3.2 favor systems which retrieve relevant documents quickly can have a poor overall recall performance R-precision R: total number of relevant documents examples in Figures 3.2 and 3.3

14 precision histogram

15 combining recall and precision
the harmonic mean it assumes a high value only when both recall and precision are high

16 the E measure b=1, complement of the harmonic mean
b>1, the user is more interested in precision b<1, the user is more interested in recall

17 user-oriented measures

18 coverage ratio: fraction of the documents known to be relevant which has been retrieved
the system finds the relevant documents the user expected to see

19 novelty ratio: fraction of the relevant documents retrieved which was previously unknown to the user
the system reveals new relevant documents previously unknown to the user

20 relative recall: the ratio between the number of relevant documents found and the number of relevant documents the user expected to find relative recall= when the relative recall equals to 1 (the user finds enough relevant documents), the user stops searching

21 recall effort: the ratio between the number of relevant documents the user expected to find and the number of documents examined in an attempt to find the expected relevant documents research in IR lack a solid formal framework lack robust and consistent testbeds and benchmarks Text REtrieval Conference

22 retrieval techniques methods using automatic thesauri sophisticated term weighting natural language techniques relevance feedback advanced pattern matching document collection over 1 million documents newspaper, patents, etc. topics in natural language conversion done by the system

23 relevant documents the pooling method: for each topic, collect the top k documents generated by each participating system and decide their relevance by human assessors the benchmark tasks ad hoc task filtering task Chinese cross languages spoken document retrieval high precision very large collection

24 evaluation measures summary table statistics: number of documents retrieved, number of relevant documents retrieved, number of relevant documents not retrieved, etc. recall-precision averages document level averages: average precision at seen relevant documents average precision histogram


Download ppt "Modern Information Retrieval"

Similar presentations


Ads by Google