1. 1 CS 430: Information Discovery Lecture 8 Evaluation of Retrieval Effectiveness II

2. 2 Course administration

3. 3 The Cranfield methodology Recall and precision depend on concept of relevance -> Is relevance a context-, task-independent property of documents? "Relevance is the correspondence in context between an information requirement statement (a query) and an article (a document), that is, the extent to which the article covers the material that is appropriate to the requirement statement." F. W. Lancaster, 1979

4. 4 Relevance Recall and precision values are for a specific set of documents and a specific set of queries Relevance is subjective, but experimental evidence suggests that, for textual documents, different experts have similar judgments about relevance Estimates of relevance level are less consistent Query types are important, depending on specificity -> subject-heading queries -> title queries -> paragraphs or free text Tests should use realistic queries

5. 5 Text Retrieval Conferences (TREC) Led by Donna Harman (NIST), with DARPA support Annual since 1992 (initial experiment ended 1999) Corpus of several million textual documents, total of more than five gigabytes of data Researchers attempt a standard set of tasks -> search the corpus for topics provided by surrogate users -> match a stream of incoming documents against standard queries Participants include large commercial companies, small information retrieval vendors, and university research groups.

6. 6 The TREC Corpus (Examples) SourceSize# DocsMedian (Mbytes)words/doc Wall Street Journal, 87-8926798,732245 Associated Press newswire, 8925484,678446 Computer Selects articles24275,180200 Federal Register, 8926025,960391 abstracts of DOE publications184226,087111 Wall Street Journal, 90-9224274,520301 Associated Press newswire, 8823779,919438 Computer Selects articles17556,920182 Federal Register, 8820919,860396

7. 7 The TREC Corpus (continued) SourceSize# DocsMedian (Mbytes)words/doc San Jose Mercury News 9128790,257379 Associated Press newswire, 9023778,321451 Computer Selects articles345161,021122 U.S. patents, 932436,7114,445 Financial Times, 91-94564210,158316 Federal Register, 9439555,630588 Congressional Record, 9323527,922288 Foreign Broadcast Information470130,471322 LA Times475131,896351

8. 8 The TREC Corpus (continued) Notes: 1. The TREC corpus consists mainly of general articles. The Cranfield data was in a specialized engineering domain. 2. The TREC data is raw data: -> No stop words are removed; no stemming -> Words are alphanumeric strings -> No attempt made to correct spelling, sentence fragments, etc.

9. 9 TREC Experiments 1.NIST provides text corpus on CD-ROM Participant builds index using own technology 2.NIST provides 50 natural language topic statements Participant converts to queries (automatically or manually) 3.Participant run search, returns up to 1,000 hits to NIST. NIST analyzes for recall and precision (all TREC participants use rank based methods of searching)

10. 10 TREC Topic Statement Number: 409 legal, Pan Am, 103 Description: What legal actions have resulted from the destruction of Pan Am Flight 103 over Lockerbie, Scotland, on December 21, 1988? Narrative: Documents describing any charges, claims, or fines presented to or imposed by any court or tribunal are relevant, but documents that discuss charges made in diplomatic jousting are not relevant. A sample TREC topic statement

11. 11 Relevance Assessment For each query, a pool of potentially relevant documents is assembled, using the top 100 ranked documents from each participant. The human expert who set the query looks at every document in the pool and determines whether it is relevant. Documents outside the pool are not examined. In a TREC-8 example, with 71 participants: 7,100 documents in the pool 1,736 unique documents (eliminating duplicates) 94 judged relevant

12. 12 A Cornell Footnote The TREC analysis uses a program developed by Chris Buckley, who spent 17 years at Cornell before completing his Ph.D. in 1995. Buckley has continued to maintain the SMART software and has been a participant at every TREC conference. SMART is used as the basis against which other systems are compared. During the early TREC conferences, the tuning of SMART with the TREC corpus led to steady improvements in retrieval efficiency, but after about TREC-5 a plateau was reached. TREC-8, in 1999, was the final year for this experiment.

13. 13 Measures based on relevance RR NN NR RN not retrieved not relevant retrieved not relevant retrieved relevant not retrieved relevant retrieved not retrieved not relevant relevant

14. 14 Measures based on relevance retrieved relevant relevant retrieved relevant retrieved retrieved not-relevant not-relevant recall = precision = fallout =

15. 15 Estimates of Recall Pooled used by TREC depends on the pool of nominated documents. Are there relevant documents not in the pool? An Example of Estimating Recall Litigation support system using IBM STAIRS system Corpus 40,000 documents 51 queries Random samples of document examined by lawyers in blind sampling experiment Estimate that only 20% of relevant documents found by STAIRS Blair and Mahon, 1981

16. 16 Recall-precision after retrieval of n documents nrelevantrecallprecision 1yes0.21.0 2yes0.41.0 3no0.40.67 4yes0.60.75 5no0.60.60 6yes0.80.67 7no0.80.57 8no0.80.50 9no0.80.44 10no0.80.40 11no0.80.36 12no0.80.33 13yes1.00.38 14no1.00.36 SMART system using Cranfield data, 200 documents in aeronautics of which 5 are relevant

17. 17 Recall-precision graph 1.0 0.75 0.5 0.25 1.0 0.75 0.5 0.25 recall precision 1 2 3 4 5 6 12 13 200

18. 18 Typical recall-precision graph 1.0 0.75 0.5 0.25 1.0 0.75 0.5 0.25 recall precision Broad, general query Narrow, specific query

19. 19 Normalized recall measure 51015200195 ideal ranks actual ranks worst ranks recall ranks of retrieved documents

20. 20 Normalized recall area between actual and worst area between best and worst Normalized recall = R norm = 1 - r i - i n(N - n) i = 1 n  n 

21. 21 Normalized Symmetric Difference Retrieved Relevant All documents A B Symmetric difference, S = A  B - A  B Normalized symmetric difference = S / ½ (|A| + |B|) = 1 - 1 2(1/recall + 1/precision)

22. 22 Statistical tests Suppose that a search is carried out on systems i and j System i is superior to system j if, for all test cases, recall(i) >= recall(j) precisions(i) >= precision(j)

23. 23 Recall-precision graph 1.0 0.75 0.5 0.25 1.0 0.75 0.5 0.25 recall precision The red system appears better than the black, but is the difference statistically significant?

24. 24 Statistical tests The t-test is the standard statistical test for comparing two table of numbers, but depends on statistical assumptions of independence and normal distributions that do not apply to this data. The sign test makes no assumptions of normality and uses only the sign (not the magnitude) of the the differences in the sample values, but assumes independent samples. The Wilcoxon signed rank uses the ranks of the differences, not their magnitudes, and makes no assumption of normality but but assumes independent samples.

25. 25 User criteria System-centered and user-centered evaluation -> Is user satisfied? -> Is user successful? System efficiency -> What efforts are involved in carrying out the search? Suggested criteria recall and precision response time user effort form of presentation content coverage

26. 26 System factors that affect user satisfaction Collection Input policies -- coverage, error rates, timeliness Document characteristics -- title, abstract, summary, full text Indexing Rules for assigning terms, specificity, exhaustively Query Formulation, operators