INFORMATION RETRIEVAL TECHNIQUES BY DR. ADNAN ABID Lecture # 23 Performance Evaluation of Information Retrieval Systems

ACKNOWLEDGEMENTS The presentation of this lecture has been taken from the underline sources “Introduction to information retrieval” by Prabhakar Raghavan, Christopher D. Manning, and Hinrich Schütze “Managing gigabytes” by Ian H. Witten, ‎Alistair Moffat, ‎Timothy C. Bell “Modern information retrieval” by Baeza-Yates Ricardo, ‎  “Web Information Retrieval” by Stefano Ceri, ‎Alessandro Bozzon, ‎Marco Brambilla

Outline Why System Evaluation? Difficulties in Evaluating IR Systems Measures for a search engine Measuring user happiness How do you tell if users are happy?

Why System Evaluation? There are many retrieval models/ algorithms/ systems, which one is the best? What is the best component for: Ranking function (dot-product, cosine, …) Term selection (stopword removal, stemming…) Term weighting (TF, TF-IDF,…) How far down the ranked list will a user need to look to find some/all relevant documents? 00:01:59  00:02:30(there are) 00:03:00  00:03:30(ranking) 00:05:10  00:05:25(term selection) 00:06:20  00:06:35(term weighting) 00:06:49  00:07:05 (how far down)

Difficulties in Evaluating IR Systems Effectiveness is related to the relevancy of retrieved items. Relevancy is not typically binary but continuous. Even if relevancy is binary, it can be a difficult judgment to make. Relevancy, from a human standpoint, is: Subjective: Depends upon a specific user’s judgment. Situational: Relates to user’s current needs. Cognitive: Depends on human perception and behavior. Dynamic: Changes over time. 00:07:50  00:08:18 (effectiveness) 00:09:15  00:09:40 (relevancy is not) 00:10:00  00:10:20 (even) 00:12:30  00:13:15 (relevancy from human + subjective + situational) 00:13:55  00:14:50 (cognitive + dynamic)

Measures for a search engine How fast does it index Number of documents/hour (Average document size) How fast does it search Latency as a function of index size Expressiveness of query language Ability to express complex information needs Speed on complex queries Uncluttered UI Is it free? 00:15:30  00:15:45 (how fast does it index) 00:18:27  00:18:46 (how fast does it search) 00:20:40  00:21:00 (expressiveness) 00:24:50  00:25:15 (Uncluttered UI + is it free)

Measuring user happiness Issue: who is the user we are trying to make happy? Depends on the setting Web engine: User finds what s/he wants and returns to the engine Can measure rate of return users User completes task – search as a means, not end See Russell http://dmrussell.googlepages.com/JCDL-talk- June-2007-short.pdf eCommerce site: user finds what s/he wants and buys Is it the end-user, or the eCommerce site, whose happiness we measure? Measure time to purchase, or fraction of searchers who become buyers? 00:30:50  00:31:30 (issue) 00:34:25  00:35:00 (web engine) 00:35:25  00:35:50 (eCommerce)

Measuring user happiness Enterprise (company/govt/academic): Care about “user productivity” How much time do my users save when looking for information? Many other criteria having to do with breadth of access, secure access, etc. 00:37:38  00:38:00 (enterprise)

How do you tell if users are happy? Search returns products relevant to users How do you assess this at scale? Search results get clicked a lot Misleading titles/summaries can cause users to click Users buy after using the search engine Or, users spend a lot of $ after using the search engine Repeat visitors/buyers Do users leave soon after searching? Do they come back within a week/month/… ? 00:48:00  00:48:25 (search return) 00:49:10  00:49:30 (search results) 00:50:50  00:51:10 (user buy) 00:51:50  00:50:10 (repeat)