1 Evaluations in information retrieval

2 Evaluations in information retrieval: summary The following gives an overview of approaches that are applied to assess the quality of »information retrieval systems, and more concretely of search systems »the resulting set of records obtained after performing a query in an information retrieval system Note: This should not be confused with assessing the quality and value of the content of an information source.

3 Evaluations in information retrieval: introduction The quality of the results, the outcome of any search using any retrieval system depends on many components / factors. These components can be evaluated and modified to increase the quality of the results more or less independently.

4 Evaluations in information retrieval: important factors The information retrieval system ( = contents + system) The user of the retrieval system and the search strategy applied to the system Result of a search

5 Evaluations in information retrieval: why? (Part 1) To study the differences in outcome/results when a component of a retrieval system is changed, such as »the user interface »the retrieval algorithm »addition by the database of uncontrolled, natural language keywords versus keywords selected from a more rigid, controlled vocabulary

6 Evaluations in information retrieval: why? (Part 2) To study the differences in outcome/results when a search strategy is changed. To study the differences in outcome/results when searches are performed by different groups of users, such as »children versus adults »inexperienced users versus more experienced, professional information intermediaries/professionals

7 Evaluations in information retrieval: the simple Boolean model Boolean model: # items in database = # items selected + # items not selected # Items selected = # relevant items + # irrelevant items Relevant Yes 1 In Irrelevant No 0 Out

8 Relevant items in a database: scheme Dependent on the aims, independent of the search strategy Relevant items! (In most cases the small subset) Irrelevant / NOT relevant items (In most cases the large subset)

9 Selecting relevant items by searching a database: scheme Dependent on the aims, independent of the search strategy Selected and relevant! Selected but not relevant Not selected but relevant Not selected and not relevant Dependent on the aims and dependent on the search strategy

10 Recall: definition and meaning Definition: # Of selected relevant items “Recall” = * 100% Total # of relevant items in database Aim: high recall Problem: in most practical cases, the total # of relevant items in a database cannot be measured.

11 Selecting relevant items: recall Selected and relevant! Selected but not relevant Not selected but relevant Not selected and not relevant

12 !? Question !? Task !? Problem !? How to use of the concept “recall”, when you do not know the total number of relevant items in the database ?

13 Recall: how to use the concept of recall Using the same database, variations in recall express the effect of search variations »Variations in search terms »Use of a classification scheme »Use of a thesaurus »...

14 !? Question !? Task !? Problem !? How can you change your search strategy to increase the recall?

15 Precision: definition and meaning Definition: # Of selected relevant items “Precision” = * 100% Total # of selected items Aim: high precision

16 Selecting relevant items: precision Selected and relevant! Selected but not relevant Not selected but relevant Not selected and not relevant

17 !? Question !? Task !? Problem !? How can you change your search strategy to increase the precision?

18 !? Question !? Task !? Problem !? When you change your search strategy to increase the precision, which consequence do you expect for the recall, in most cases?

19 Relation between recall and precision of searches 100% Recall 0 0 Precision 100% Ideal = Impossible to reach in most systems Ideal = Impossible to reach in most systems Search (results)

20 !? Question !? Task !? Problem !? Indicate on the figure that a user improves a search.

21 !? Question !? Task !? Problem !? Indicate on the figure that a database producer and / or the retrieval system improves the retrieval quality.

22 !? Question !? Task !? Problem !? Indicate the relation between the recall and precision in a classical information retrieval system in the form of a figure. Indicate in that figure a good and a bad search.

23 Recall and precision should be considered together Examples: Increase in retrieved number of relevant items may be accompanied by an impractical decrease in precision. Precision of a search close to 100% may NOT be ideal, because the recall of the search may be too low. Make search / query broader to increase recall ! Poor (low) precision is more noticeable than bad (low) recall.

24 Evaluation in the case of systems offering relevance ranking Many modern information retrieval systems offer output with relevance ranking. This is more complicated than simple Boolean retrieval, and the simple concepts of recall and precision cannot be applied. To compare retrieval systems or search strategies, decide to consider for comparison a particular number of items ranked highest in each output. This brings us to for instance: “first-20 precision”.

25 !? Question !? Task !? Problem !? Give examples of retrieval systems that offer relevance ranking.