1 Discussion Class 11 Click through Data as Implicit Feedback

2 Discussion Classes Format: Question Ask a member of the class to answer. Provide opportunity for others to comment. When answering: Stand up. Give your name. Make sure that the TA hears it. Speak clearly so that all the class can hear. Suggestions: Do not be shy at presenting partial answers. Differing viewpoints are welcome.

3 Question 1: Training data (a) Why is training data important in tuning a search engine? (b) Explain the difference between the use of "explicit relevance data" and "implicit feedback." (c) What is "click through data" and why might it be useful?

4 Question 2: Eye tracking (a) What is eye tracking? (b)Define "fixation" and "saccade"? (c)How are they measured?

5 Question 3: Explicit relevance judgments The method for obtaining explicit relevance judgments was very different from the TREC methodology. Explain the difference in: (a) Selection of assessors (b) Documents reviewed by assessors (c) Task required of assessors for each document (d) Technique for ensuring consistency between assessors Are these differences suitable for this experiment?

6 Question 4: Regular Google ranks How do the fixations differ from the click?

7 Question 5: Relevance, ranking, and click through data (a)When the rankings were reversed, how did the user behavior change? i) Average rank of pages clicked ii) Number of pages clicked (b)The evidence that user's click through behavior is influenced by trust in Google was studied by a statistical experiment. i) What was the hypothesis? ii) What was the methodology? iii) What was the conclusions?

8 Question 6: Clicks as relative relevance judgments Section 5.3 looks at a number of relative measures: click > skip above last click > click above click > click earlier last click > click previous click > no-click next (a)What are these tests attempting to measure? (b)Explain the rationale behind: click > skip above.

9 Question 7:

10 Question 8:

11 Question 9: