1. Evaluation of IR Performance
Dr. Bilal
IS 530
Fall 2005

2. Searching for Information
Imprecise
Incomplete
Tentative
Challenging

3. IR Performance Precision Ratio =
the number of relevant documents retrieved
the total number of documents retrieved

4. IR Performancel Recall Ratio =
the number of relevant documents retrieved
the total number of relevant documents

5. Why Do We Miss Items? Indexing errors Wrong search terms
Wrong database
Language variations
Other???

6. Why Do We Get Unwanted Items?
Indexing errors
Wrong search terms
Homographs
Incorrect term relations
Other???

7. Boolean Operators OR increases recall AND increases precision
NOT increases precision

8. Recall and Precision in Practice
Inversely related
Search strategies designed for high precision or high recall (or medium)
Needs of users dictate search strategy towards recall or precision
Practice helps changing queries to favor recall or precision

9. Recall and Precision
1.0
Recall
1.0
Precision

10. Problems with Relevance, Recall, and Precision
Yes or no decision
Things are more or less relevant
In practice not easy to measure
Not focused on user needs

11. Relevance A match between a query and information retrieved
Is a judgment
Can be judged by anyone who is informed of the query and views the retrieved information

12. Relevance (cont.) Judgments may differ
Is the base for information retrieval evaluation methods (recall and precision)
Documents can be ranked by likely relevance

13. Pertinence Based on information need rather than request and documents
Can only be judged by user
May differ from relevance judgments

14. Pertinence (cont.) Transient, varies with many factors
Not often used in evaluation
May be used as a measure of satisfaction

15. High Precision Searching
Controlled vocabulary
Limits: Specific fields, major descriptors, Date, language, etc.
AND operator
Proximity
Careful with truncation

16. High Recall Searching OR logic Keyword searching No limits Truncate
Broader terms

17. Related Concepts
Topicality
Aboutness
Utility
Pertinence
Satisfaction

18. Hints for Improving Performance
Good interview
User presence, if possible
Preliminary search and user response
Evaluation during search (you or you and user)
User feedback
Search refinement as you progress