1. 1 CS 430 / INFO 430 Information Retrieval Lecture 10 Evaluation of Retrieval Effectiveness 1

2. 2 Course administration Assignment 1 Everybody should have received an email message with their grades and comments. If you have not received a message, please contact: cs430-l@lists.cs.cornell.edu

3. 3 Course administration Regrading requests If you have a question about your grade, send a message to cs430-l@lists.cs.cornell.edu. The original grader and I will review your query and I will reply to you. If we clearly made a mistake, we will correct the grade. If we discover a mistake that was missed in the first grading, we will deduct points. If the program did not run for a simple reason (e.g., wrong files submitted), we will run it and regrade it, but remove the points for poor submission. When the grade is a matter of judgment, no changes will be made.

4. 4 Course administration Assignment 2 We suggest the JAMA package for singular value decomposition in Java. If you wish to use C++, you will need to find a similar package. You will have to select a suitable value of k, the number of dimensions. In your report explain how you selected k. With only 20 short files, the appropriate value of k for this corpus is almost certainly much less than the value of 100 used in the reading.

5. 5 Course administration Discussion Class 4 (a) Check the Web site for which sections to concentrate on. (b) The PDF version of the file on the TREC site is damaged. Use the PostScript version, or the PDF version on the course Web site.

6. 6 Retrieval Effectiveness Designing an information retrieval system has many decisions: Manual or automatic indexing? Natural language or controlled vocabulary? What stoplists? What stemming methods? What query syntax? etc. How do we know which of these methods are most effective? Is everything a matter of judgment?

7. 7 Evaluation To place information retrieval on a systematic basis, we need repeatable criteria to evaluate how effective a system is in meeting the information needs of the user of the system. This proves to be very difficult with a human in the loop. It proves hard to define: the task that the human is attempting the criteria to measure success

8. 8 Studies of Retrieval Effectiveness The Cranfield Experiments, Cyril W. Cleverdon, Cranfield College of Aeronautics, 1957 -1968 SMART System, Gerald Salton, Cornell University, 1964-1988 TREC, Donna Harman and Ellen Voorhees, National Institute of Standards and Technology (NIST), 1992 -

9. 9 Evaluation of Matching: Recall and Precision If information retrieval were perfect... Every hit would be relevant to the original query, and every relevant item in the body of information would be found. Precision: percentage (or fraction) of the hits that are relevant, i.e., the extent to which the set of hits retrieved by a query satisfies the requirement that generated the query. Recall: percentage (or fraction) of the relevant items that are found by the query, i.e., the extent to which the query found all the items that satisfy the requirement.

10. 10 Recall and Precision with Exact Matching: Example Corpus of 10,000 documents, 50 on a specific topic Ideal search finds these 50 documents and reject all others Actual search identifies 25 documents; 20 are relevant but 5 were on other topics Precision: 20/25 = 0.8 (80% of hits were relevant) Recall: 20/50 = 0.4(40% of relevant were found)

11. 11 Measuring Precision and Recall Precision is easy to measure: A knowledgeable person looks at each document that is identified and decides whether it is relevant. In the example, only the 25 documents that are found need to be examined. Recall is difficult to measure: To know all relevant items, a knowledgeable person must go through the entire collection, looking at every object to decide if it fits the criteria. In the example, all 10,000 documents must be examined.

12. 12 Relevance as a Set Comparison D = set of documents A = set of documents that satisfy some user- based criterion B = set of documents identified by the search system

13. 13 Measures based on relevance retrieved relevant | A  B | relevant | A | retrieved relevant | A  B | retrieved | B | retrieved not-relevant | B - A  B | not-relevant | D - A | recall = = precision = = fallout = =

14. 14 Combining Recall and Precision: Normalized Symmetric Difference Relevant Retrieved D = set of documents A B Symmetric difference, S = A  B - A  B Normalized symmetric difference = |S| / 2 (|A| + |B|) = 1 - 1 (1/recall + 1/precision) Symmetric Difference: The set of elements belonging to one but not both of two given sets. 1212 { }

15. 15 Relevance Recall and precision: depend on concept of relevance Relevance is a context-, task-dependent property of documents "Relevance is the correspondence in context between an information requirement statement... and an article (a document), that is, the extent to which the article covers the material that is appropriate to the requirement statement." F. W. Lancaster, 1979

16. 16 Relevance How stable are relevance judgments? For textual documents, knowledgeable users have good agreement in deciding whether a document is relevant to an information requirement. There is less consistency with non-textual documents, e.g., a photograph. Attempts to have users give a level of relevance, e.g., on a five point scale, are inconsistent.

17. 17 Relevance judgments (TREC) In the TREC experiments, each topic was judged by a single assessor, who also set the topic statement. In TREC-2, a sample of the topics and documents was rejudged by second expert assessor. The average agreement was about 80%. In TREC-4, all topics were rejudged by two additional assessors, with 72% agreement among all three assessors.

18. 18 Relevance judgments (TREC) However: In the TREC-4 tests, most of the agreement was among the documents that all assessors agreed were non-relevant 30% of documents judged relevant by the first assessor, were judged non-relevant by both additional assessors. Using data from TREC-4 and TREC-6, Voorhees estimates a practical upper bound of 65% precision at 65% recall, as the level at which human experts agree with one another.

19. 19 Cranfield Collection The first Information Retrieval test collection Test collection: 1,400 documents on aerodynamics Queries: 225 queries, with a list of the documents that should be retrieved for each query

20. 20 Cranfield Second Experiment Comparative efficiency of indexing systems: (Universal Decimal Classification, alphabetical subject index, a special facet classification, Uniterm system of co-ordinate indexing) Four indexes prepared manually for each document in three batches of 6,000 documents -- total 18,000 documents, each indexed four times. The documents were reports and paper in aeronautics. Indexes for testing were prepared on index cards and other cards. Very careful control of indexing procedures.

21. 21 Cranfield Second Experiment (continued) Searching: 1,200 test questions, each satisfied by at least one document Reviewed by expert panel Searches carried out by 3 expert librarians Two rounds of searching to develop testing methodology Subsidiary experiments at English Electric Whetstone Laboratory and Western Reserve University

22. 22 The Cranfield Data The Cranfield data was made widely available and used by other researchers Salton used the Cranfield data with the SMART system (a) to study the relationship between recall and precision, and (b) to compare automatic indexing with human indexing Sparc Jones and van Rijsbergen used the Cranfield data for experiments in relevance weighting, clustering, definition of test corpora, etc.

23. 23 Cranfield Experiments -- Measures of Effectiveness for Matching Methods Cleverdon's work was applied to matching methods. He made extensive use of recall and precision, based on concept of relevance. recall (%) precision (%) x x x x x x x x x x Each x represents one search. The graph illustrates the trade-off between precision and recall.

24. 24 Typical precision-recall graph for different queries 1.0 0.75 0.5 0.25 1.0 0.75 0.5 0.25 recall precision Broad, general query Narrow, specific query Using Boolean type queries Note: Some authors plot recall against precision.

25. 25 A Crucial Cranfield Results The various manual indexing systems have similar retrieval efficiency Retrieval effectiveness using automatic indexing can be at least as effective as manual indexing with controlled vocabularies -> original results from the Cranfield + SMART experiments (published in 1967) -> considered counter-intuitive -> other results since then have supported this conclusion

26. 26 Precision and Recall with Ranked Results Precision and recall are defined for a fixed set of hits, e.g., Boolean retrieval. Their use needs to be modified for a ranked list of results.

27. 27 Evaluation: Ranking Methods Precision and recall measure the results of a single query using a specific search system applied to a specific set of documents. Matching methods: Precision and recall are single numbers. Ranking methods: Precision and recall are functions of the rank order.

28. 28 Evaluating Ranking: Recall and Precision If information retrieval were perfect... Every document relevant to the original information need would be ranked above every other document. With ranking, precision and recall are functions of the rank order. Precision(n): fraction (or percentage) of the n most highly ranked documents that are relevant. Recall(n): fraction (or percentage) of the relevant items that are in the n most highly ranked documents.

29. 29 Precision and Recall with Ranking Example "Your query found 349,871 possibly relevant documents. Here are the first eight." Examination of the first 8 finds that 5 of them are relevant.

30. 30 Graph of Precision with Ranking: P(r) Precision P(r) Rank r 0 1 1 2 3 4 5 6 7 8 Relevant? Y N Y Y N Y N Y 1/1 1/2 2/3 3/4 3/5 4/6 4/7 5/8

31. 31 Ranked retrieval: Recall and precision after retrieval of n documents nrelevantrecallprecision 1yes0.21.0 2yes0.41.0 3no0.40.67 4yes0.60.75 5no0.60.60 6yes0.80.67 7no0.80.57 8no0.80.50 9no0.80.44 10no0.80.40 11no0.80.36 12no0.80.33 13yes1.00.38 14no1.00.36 SMART system using Cranfield data, 200 documents in aeronautics of which 5 are relevant

32. 32 Precision-recall graph 1.0 0.75 0.5 0.25 1.0 0.75 0.5 0.25 recall precision 1 2 3 4 5 6 12 13 200 Note: Some authors plot recall against precision.

33. 33 11 Point Precision (Recall Cut Off) p(n) is precision at that point where recall has first reached n Define 11 standard recall points p(r 0 ), p(r 1 ),... p(r 10 ), where p(r j ) = p(j/10) Note: if p(r j ) is not an exact data point, use interpolation

34. 34 Example: SMART System on Cranfield Data Recall Precision 0.0 1.0 0.1 1.0 0.2 1.0 0.3 1.0 0.4 1.0 0.5 0.75 0.6 0.75 0.7 0.67 0.8 0.67 0.9 0.38 1.0 0.38 Precision values in blue are actual data. Precision values in red are by interpolation (by convention equal to the next actual data value).

35. 35 Recall cutoff graph: choice of interpolation points 1.0 0.75 0.5 0.25 1.0 0.75 0.5 0.25 recall precision 1 2 3 4 5 6 12 13 200 The blue line is the recall cutoff graph.

36. 36 Average precision Average precision for a single topic is the mean of the precision obtained after each relevant document is obtained. Example: p = (1.0 + 1.0 + 0.75 + 0.67 + 0.38) / 5 = 0.75 Mean average precision for a run consisting of many topics is the mean of the average precision scores for each individual topic in the run. Definitions from TREC-8

37. 37 Normalized recall measure 51015200195 ideal ranks actual ranks worst ranks recall ranks of retrieved documents

38. 38 Normalized recall area between actual and worst area between best and worst Normalized recall = R norm = 1 - (after some mathematical manipulation) r i - i n(N - n) i = 1 n  n 

39. 39 Statistical tests Suppose that a search is carried out on systems i and j System i is superior to system j if, for all test cases, recall(i) >= recall(j) precisions(i) >= precision(j) In practice, we have data from a limited number of test cases. What conclusions can we draw?

40. 40 Recall-precision graph 1.0 0.75 0.5 0.25 1.0 0.75 0.5 0.25 recall precision The red system appears better than the black, but is the difference statistically significant?

41. 41 Statistical tests The t-test is the standard statistical test for comparing two table of numbers, but depends on statistical assumptions of independence and normal distributions that do not apply to this data. The sign test makes no assumptions of normality and uses only the sign (not the magnitude) of the the differences in the sample values, but assumes independent samples. The Wilcoxon signed rank uses the ranks of the differences, not their magnitudes, and makes no assumption of normality but but assumes independent samples.