1. Eye Tracking Analysis of User Behavior in WWW Search Laura Granka Thorsten Joachims Geri Gay

2. why use eye-tracking for information retrieval? Understand how searchers evaluate online search results Enhanced interface design More accurate interpretation of implicit feedback (eg, clickthrough data) More targeted metrics for evaluating retrieval performance Figure: popular regions are highlighted through shadow-intensity

3. key research questions How long does it take searchers to select a document? How many abstracts do searchers look at before making a selection? Do searchers look at abstracts ranked lower than the selected document? Do searchers view abstracts linearly? Which parts of the abstract are most likely to be viewed?

4. what is eye-tracking? Device to detect and record where and what people look at Multiple applications: reading, usability, visual search, in both physical and virtual contexts Eye tracking device Figure: Cornell HCI eye-tracking configuration View of subject’s pupil on monitor; used for calibration

5. ocular indices for www tracking Fixations: ~200-300ms; information is acquired Saccades: extremely rapid movements between fixations Pupil dilation: size of pupil indicates interest, arousal Aggregate eye-tracking graphs depict viewing intensity in key regions “Scanpath” output depicts pattern of movement throughout screen. Black markers represent fixations.

6. experimental search tasks Ten search tasks given to all participants Search topics included travel, science, movies, local, television, college, and trivia Searches evenly split between informational and navigational tasks

7. experimental procedures Users conducted live Google searches Users allowed to search freely, with any queries Script removed all ad content Proxy stored all pages and log files Figure: Specific “zones” were created around each result, enabling eye-movements to be analyzed specific to the rankings

8. sample eye-tracking output

13. overall searching behavior How long does it take users to select a document? Overall mean: 5.7 seconds, St.D: 5.4 Time spent before a result is clicked less difficultmore difficult task

14. overall viewing behavior Mean: 3.07 Median/Mode: 2.00 Most likely to view only two documents per results set How many abstracts do we view, and in what order? Notice dip after page break

15. overall viewing behavior How many abstracts do we view, and in what order? Results viewed linearly

16. overall viewing behavior Time spent viewing each abstract compared with the frequency that each rank is selected. Error bars are 1 SE How much time do we spend viewing each abstract?

17. overall viewing behavior Number of abstracts viewed above and below selected link How thoroughly do we view the results set?

18. overall viewing behavior What information in the abstract is most useful? Title: 30% Snippet: 43% Category: 0.3% URL: 21% Other: 5% (includes, cached, similar pages, description) Percentage of time spent viewing each part of abstract Other: 5.3% Title: 30.5% Category: 0.3% Snippet: 42.8% URL: 21.1%

19. overall searching behavior *Difficulty and satisfaction are ranked on a 1-10 scale; 10 meaning very difficult and very satisfied, respectively Search task difficulty and satisfaction with Google less difficultmore difficult

20. overall searching behavior Mean rank of selected doc: 2.66 Median/ Mode: 1.00 Task difficulty influences rank of selected document and number of abstracts viewed less difficultmore difficult

21. overall searching behavior 1. Michael Jordan statistician20 2. Thousand acres dude ranch11 2. One thousand acres dude ranch11 3. 1000 acres dude ranch9 4. Time machine movie7 4. Carnegie mellon university graduate housing7 5. Imdb6 5. Emeril lagasse6 5. First modern antibiotic6 5. Greyhound bus6 5. Carnegie mellon graduate housing6 Top Query Terms Frequency

22. conclusions Searching Trends: Popularity of specialized, vertical portals Several students preferred conducting a Google search from the cmu.edu homepage Majority of students preferred an internal imdb.com search over a general Google search

23. conclusions Document selected in under 5 seconds Users click on the first promising link they see Results viewed linearly Top 2 results most likely to be viewed Users rather reformulate query than scroll Task type and difficulty affect viewing behavior Presentation of results affects selection

24. future research Impact on advertising With such fast selections being made, will searchers even view ads? ? Ads most likely to be seen: Difficult task Ambiguous info need Informational query Low searcher expertise

25. future research Relevance judgments Do we spent more time viewing relevant abstracts? Do we click the first relevant abstract viewed? Does pupil dilation increase for more relevant documents If results were re-ranked, would viewing behavior differ? If results were re-ranked, would viewing behavior differ?

26. Cornell University Computer Science & Human-Computer Interaction Thorsten Joachims tj@cs.cornell.edu Laura Granka granka@cornell.edu Matthew Feusner feusner@cornell.edu Geri Gay gkg1@cornell.edu Helene Hembrooke hah4@cornell.edu