1. INFORMATION RETRIEVAL TECHNIQUES BY DR. ADNAN ABID
Lecture # 23
Performance Evaluation of Information Retrieval Systems

2. ACKNOWLEDGEMENTS
The presentation of this lecture has been taken from the underline sources
“Introduction to information retrieval” by Prabhakar Raghavan, Christopher D. Manning, and Hinrich Schütze
“Managing gigabytes” by Ian H. Witten, ‎Alistair Moffat, ‎Timothy C. Bell
“Modern information retrieval” by Baeza-Yates Ricardo, ‎ 
“Web Information Retrieval” by Stefano Ceri, ‎Alessandro Bozzon, ‎Marco Brambilla

3. Outline Why System Evaluation? Difficulties in Evaluating IR Systems
Measures for a search engine
Measuring user happiness
How do you tell if users are happy?

4. Why System Evaluation?
There are many retrieval models/ algorithms/ systems, which one is the best?
What is the best component for:
Ranking function (dot-product, cosine, …)
Term selection (stopword removal, stemming…)
Term weighting (TF, TF-IDF,…)
How far down the ranked list will a user need to look to find some/all relevant documents?
00:01:59  00:02:30(there are)
00:03:00  00:03:30(ranking)
00:05:10  00:05:25(term selection)
00:06:20  00:06:35(term weighting)
00:06:49  00:07:05 (how far down)

5. Difficulties in Evaluating IR Systems
Effectiveness is related to the relevancy of retrieved items.
Relevancy is not typically binary but continuous.
Even if relevancy is binary, it can be a difficult judgment to make.
Relevancy, from a human standpoint, is:
Subjective: Depends upon a specific user’s judgment.
Situational: Relates to user’s current needs.
Cognitive: Depends on human perception and behavior.
Dynamic: Changes over time.
00:07:50  00:08:18 (effectiveness)
00:09:15  00:09:40 (relevancy is not)
00:10:00  00:10:20 (even)
00:12:30  00:13:15 (relevancy from human + subjective + situational)
00:13:55  00:14:50 (cognitive + dynamic)

6. Measures for a search engine
How fast does it index
Number of documents/hour
(Average document size)
How fast does it search
Latency as a function of index size
Expressiveness of query language
Ability to express complex information needs
Speed on complex queries
Uncluttered UI
Is it free?
00:15:30  00:15:45 (how fast does it index)
00:18:27  00:18:46 (how fast does it search)
00:20:40  00:21:00 (expressiveness)
00:24:50  00:25:15 (Uncluttered UI + is it free)

7. Measuring user happiness
Issue: who is the user we are trying to make happy?
Depends on the setting
Web engine:
User finds what s/he wants and returns to the engine
Can measure rate of return users
User completes task – search as a means, not end
See Russell June-2007-short.pdf
eCommerce site: user finds what s/he wants and buys
Is it the end-user, or the eCommerce site, whose happiness we measure?
Measure time to purchase, or fraction of searchers who become buyers?
00:30:50  00:31:30 (issue)
00:34:25  00:35:00 (web engine)
00:35:25  00:35:50 (eCommerce)

8. Measuring user happiness
Enterprise (company/govt/academic): Care about “user productivity”
How much time do my users save when looking for information?
Many other criteria having to do with breadth of access, secure access, etc.
00:37:38  00:38:00 (enterprise)

9. How do you tell if users are happy?
Search returns products relevant to users
How do you assess this at scale?
Search results get clicked a lot
Misleading titles/summaries can cause users to click
Users buy after using the search engine
Or, users spend a lot of $ after using the search engine
Repeat visitors/buyers
Do users leave soon after searching?
Do they come back within a week/month/… ?
00:48:00  00:48:25 (search return)
00:49:10  00:49:30 (search results)
00:50:50  00:51:10 (user buy)
00:51:50  00:50:10 (repeat)