1. What Does the User Really Want ? Relevance, Precision and Recall

2. Overview What is relevance ? Define Precision Define Recall

3. Basic Web Retrieval Process Need Keyword Query Results

4. Relevance Good results are relevant to the need ! BUT –How do we know what the user needs and wants

5. Three Definition of Relevance Pages which match the user need –Expert centred Pages which the user thinks match their need –User centred Pages which match the user query –System centred

6. Expert Centred Agree a definition of what the query means Trawl through results identifying each page returned as relevant or irrelevant Disadvantage –Takes no account of users current knowledge or needs –static

7. User Centred Get user to trawl through pages judging them as relevant or irrelevant Problem –If they knew the answer would they be looking ? –Can we get the same result twice ? –expensive

8. System Centred Assumes keywords adequately express need Highly reproducible Really assessing accuracy of indexing and keyword matching

9. Formal evaluations Almost always use an Expert Centred or System Centred Approach

10. Formal Definition of relevance A relevant page is a page which meets the user’s need and can be recognised as such by the search result description provided by the search engine by a sufficiently expert user

11. Precision

12. Precision Simply Defined The proportion of documents returned which are actually relevant to the query or need

13. Precision Issues Needs to be averaged across a set of queries to be meaningful Needs to take account of rank order in results set

14. Recall

15. Recall Simply Defined The proportion of available relevant documents (anywhere on the web) which are actually returned in the results set

16. Retrieved versus Relevant Docs Retrieved Relevant High Precision Retrieval High Recall Retrieval

17. Recall Problems Need to find all the pages on the web relevant to the need –Theoretically get a team of experts to assess every page on the web for relevance In practice the results of a high recall search of a fixed web trawl is assessed

18. Different sorts of retrieval Today’s Sunderland result –High precision Legal Cases –Has anyone ever been acquitted of murder and subsequently used their presence at the murder as an alibi ? –High recall

19. Key Points Definition of Relevance –and its problems Definition of Precision Definition of Recall