1. Special Topics in Computer Science The Art of Information Retrieval Chapter 3: Retrieval Evaluation Alexander Gelbukh www.Gelbukh.com

2. 2 Previous chapter Modeling is needed for formal operations Boolean model is the simplest Vector model is the best combination of quality and simplicity oTF-IDF term weighting oThis (or similar) weighting is used in all further models Many interesting and not well-investigated variations opossible future work

3. 3 Previous chapter: Research issues How people judge relevance? oranking strategies How to combine different sources of evidence? What interfaces can help users to understand and formulate their Information Need? ouser interfaces: an open issue Meta-search engines: combine results from different Web search engines oThey almost do not intersect oHow to combine ranking?

4. 4 Evaluation! How do you measure if your system is good or bad? To go to the right direction, need to know where you want to get to. We can do it this way vs. This way it performs better oI think it is better... oWe do it this way... oOur method takes into account syntax and semantics... oI like the results... Criterion of truth. Crucial for any science. Enables competition financial policy attracts people oTREC international competitions

5. 5 Methodology Define formally your task and constraints Define formally your evaluation criterion (argue if needed) oOne numerical value, not several! Demonstrate that your method gives better value than othe baseline (the simple obvious way) Retrieve all. Retrieve none. Retrieve at random. Use Google. ostate-of-the-art (the best reported method) That your parameter settings are optimal oConsider singular (extreme) settings oSet your parameters to 0. To infinity.

6. 6 Methodology The only valid way of reasoning But we want the clusters to be non-trivial oThen add this as a penalty to your criteria or as constraints Divide your acceptability considerations: oConstraints: yes/no. oEvaluation: better/worse. Check that your evaluation criteria are well justified oMy formula gives it this way oMy result is correct since this is what my algorithm gives oReason in terms of the user task, not your algorithm / formulas Are your good/bad judgments in accord with intuition?

7. 7 Evaluation? IR: user satisfaction oDifficult to model formally oExpensive to measure directly (experiments with subjects) At least two contradicting parameters oCompleteness vs. quality oNo good way to combine into one single numerical value oSome user-defined weights of importance of the two Not formal, depend on situation Art

8. 8 Parameters to evaluate Performance evaluation oSpeed oSpace Tradoff oCommon for all systems. Not discussed here. Retrieval performance (quality?) evaluation o= goodness of a retrieval strategy oA test reference collection: docs and queries. oThe correct set (or ordering) provided by experts oA similarity measure to compare system output with the correct one.

9. 9 Evaluation: Model User Satisfaction User task oBatch query processing? Interaction? Mixed? Way of use oReal-life situation: what factors matter? oInterface type This chapter: laboratory settings oRepeatability oScalability

10. 10 Precision & Recall Tradeoff (as with time and space) Assumes the retrieval results are sets oas Boolean; in Vector use threshold Measures closeness between two sets Recall: Of relevant docs, how many (%) were retrieved? Others are lost. Precision: Of retrieved docs, how many (%) are relevant? Others are noise. Nowadays with huge collections Precision is more important!

11. 11 Precision & Recall Recall = Precision =

12. 12 Ranked Output... Truth: unordered relevant set Output: ordered guessing Compare ordered set with an unordered one

13. 13...Ranked Output Plot precision vs. recall curve In the initial part of the list containing n% of all relevant docs, what the precision is? o11 standard recall levels: 0%, 10%,..., 90%, 100%. o0%: interpolated

14. 14 Many experiments Average precision and recall Ranked output: Average precision at each recall level To get equal (standard) recall levels, interpolation oof 3 relevant docs, there is no 10% level! oInterpolated value at level n = maximum known value between n and n + 1 oIf none known, use the nearest known.

15. 15 Precision vs. Recall Figures Alternative method: document cutoff values oPrecision at first 5, 10, 15, 20, 30, 50, 100 docs Used to compare algorithms. oSimple oIntuitive NOT a one-value comparison!

16. Which one is better?

17. 17 Single-value summaries Performance for an individual query oCan be averaged over several queries, too oHistogram for several queries can be made oTables can be made oCurves cannot be used for this! Precision at first relevant doc? Average precision at (each) seen relevant docs oFavors systems that give several relevant docs first R-precision oprecision at Rth retrieved (R = total relevant)

18. Precision histogram Two algs: A, B R(A)-R(B). Which is better?

19. 19 Alternative measures Problems with Precision & Recall measure: oRecall cannot be estimated with large collections oTwo values, but we need one value to compare oDesigned for batch mode, not interactive. Informativeness! oDesigned for linear ordering of docs (not weak ordering) Alternative measures: combine both in one F-measure: E-measure: user preference Rec vs. Prec

20. User-oriented measures Definitions:

21. 21 User-oriented measures Coverage ratio oMany expected docs Novelty ratio oMany new docs Relative recall: # found / # expected Recall effort: # expected / # examined until those are found Other: oexpected search length (good for weak order) osatisfaction (considers only relevant docs) ofrustration (considers only non-relevant docs)

22. 22 Reference collections Texts with queries and relevant docs known TREC Text REtrieval Conference. Different in different years Wide variety of topics. Document structure marked up. 6 GB. See NIST website: available at small cost Not all relevant docs marked! oPooling method: otop 100 docs in ranking of many search engines omanually verified oWas tested that is a good approximation to the real set

23. 23...TREC tasks Ad-hoc (conventional: query answer) Routing (ranked filtering of changing collection) Chinese ad-hoc Filtering (changing collection; no ranking) Interactive (no ranking) NLP: does it help? Cross-language (ad-hoc) High precision (only 10 docs in answer) Spoken document retrieval (written transcripts) Very large corpus (ad-hoc, 20 GB = 7.5 M docs) Query task (several query versions; does strategy depends on it?) Query transforming oAutomatic oManual

24. 24...TREC evaluation Summary table statistics o# of requests used in the task o# of retrieved docs; # of relevant retrieved and not retrieved Recall-precision averages o11 standard points. Interpolated (and not) Document level averages oAlso, can include average R-value Average precision histogram oBy topic. oE.g., difference between R-precision of this system and average of all systems

25. 25 Smaller collections Simpler to use Can include info that TREC does not Can be of specialized type (e.g., include co-citations) Less sparse, greater overlap between queries Examples: oCACM oISI othere are others

26. 26 CACM collection Communications of ACM, 1958-1979 3204 articles Computer science Structure info (author, date, citations,...) Stems (only title and abstract) Good for algorithms relying on cross-citations oIf a paper cites another one, they are related oIf two papers cite the same ones, they are related 52 queries with Boolean form and answer sets

27. 27 ISI collection On information sciences 1460 docs For similarity in terms and cross-citation Includes: oStems (title and abstracts) oNumber of cross-citations 35 natural-language queries with Boolean form and answer sets

28. 28 Cystic Fibrosis (CF) collection Medical 1239 docs MEDLINE data okeywords assigned manually! 100 requests 4 judgments for each doc oGood to see agreement Degrees of relevance, from 0 to 2 Good answer set overlap ocan be used for learning from previous queries

29. 29 Research issues Different types of interfaces; interactive systems: oWhat measures to use? oSuch as infromativeness

30. 30 Conclusions Main measures: Precision & Recall. oFor sets oRankings are evaluated through initial subsets There are measures that combine them into one oInvolve user-defined preferences Many (other) characteristics oAn algorithm can be good at some and bad at others oAverages are used, but not always are meaningful Reference collection exists with known answers to evaluate new algorithms

31. 31 Thank you! Till October 9 October 23: midterm exam