1. Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London

2. Outline 1. Eye Movement Behaviour 2. Image Identification 3. Image Search 4. Conclusions & Future Work

3. Eye Gaze Computer (Eye Image Processing) Client Computer (Application Program) Chinrest Eye Monitor Real-Time Gazepoint Display Application Display Eye Tracking System

4. Eye Movement Behaviour saliency map image fixation and saccade map no ROI

5. Eye Movement Behaviour clear ROI saliency map image fixation and saccade map

6. Eye Movement Behaviour – no ROI participant Aparticipant B participant D participant C

7. Eye Movement Behaviour – clear ROI participant Aparticipant B participant D participant C

8. Variance of Attention Measure 149786212261246378Image6 1466120214531432246Image5 8571094687741443Image4 Obvious ROI 197365175389175Image3 629328496479500Image2 532333193325298Image1 Unclear ROI DCBA Participants Image Variance

9. Time Fixating Salient Regions (ms) 1240162098042406 19602220148036805 12802420234028204 Obvious ROI 204001003 4005004205802 1402060401 Unclear ROI DCBA Participants Images

10. Findings  No special fixation sequence although many look at salient regions first  Very salient regions inspected frequently and compared with background

11. Eye vs Mouse for Image Identification target image 1.Mouse click 2.Fixation > 40ms

12. Screen Display Sequence D = distractor Tn = target image

13. Eye vs Mouse Response Times 12 participants

14. Eye vs Mouse Response Times 6 participants in each group

15. Image Search Task target image steps to target 1000 images 13 participants

16. Image Selection Gaze selection of an image is determined by the sum of all fixations of 80ms or more on that image exceeding a threshold. Two thresholds 400ms and 800ms Successive sets of 15 images are retrieved based on their similarity with selected image. Performance compared with images randomly retrieved Participants not told what determines screen changes

17. Target Images easy to findhard to find

18. Similarity Links

20. Results 13 participants 8 sessions Main effect: Eye gaze 18 steps Random 22 steps p < 0.037

21. Results – Easy vs Hard Images

22. Other Selection Criteria 24 participants 8 sessions Main effect: fixation threshold not significant

23. Results - Lower Fixation Thresholds 6 participants 3 sessions Significant differences between random and 200ms + 300ms.

24. Results - Lower Fixation Thresholds

25. Conclusions  Eye tracking can be faster than tactile interfaces for visual tasks  Eye tracking interfaces are feasible for fast image search  Pre-attentive vision plays a part in very rapid search

26. Future Work  Further study of human visual behaviour  Use of higher performance similarity measures  Application to browsing large collections of photos/videos  Shared interaction