1. Windows of the Soul Slide 1 UVA - 8/24/98 Windows of the Soul? Windows of the Soul? What Eye Movements Tell Us About Software Usability University of Virginia Charlottesville, Virginia August 24, 1998

2. Windows of the Soul Slide 2 UVA - 8/24/98 Investigators Steve Ellis - Principal Investigator –Bell Labs / Lucent Technologies, Room 1K-411, 200 Laurel Ave, Middletown NJ 07748, shellis@lucent.com Ron Candrea –Bell Labs / Lucent Technologies, Room 2D-425, candrea@lucent.com Jason Misner –Systems Engineering, University of Virginia, Charlottesville, VA, 22903, jdm4s@virginia.edu Christopher Sean Craig –University of Virginia, csc3b@virginia.edu Christopher P. Lankford –University of Virginia, cpl2b@virginia.edu Thomas E. Hutchinson –University of Virginia, teh@virginia.edu

3. Windows of the Soul Slide 3 UVA - 8/24/98 Goals Of Our Investigation Assess the value of eye tracking technology for user interface evaluation and usability testing Identify eye tracking technologies most suitable for the usability lab Identify improvements in current eye tracking system capabilities for use in the usability lab Develop guidelines for purchase and use of commercial eye tracking systems

4. Windows of the Soul Slide 4 UVA - 8/24/98 Previous Eye Movement-Based Psychological Research and Applications Reading Research (Just & Carpenter, 1980; Rayner, 1978) Studies of Cognition and Elementary Information Processing (Just & Carpenter, 1976; Rayner, 1984) Social Psychology/Interpersonal Communications (Kleinke, 1986) Military Applications (Harris & Christhilf, 1980; Birkmire et al, 1993) Marketing, Advertising (Krugman et al, 1994) Usability Studies (Benel, Ottens & Horst, 1991; Card, 1984) Other Ergonomic Studies (Wierda & Maring, 1993) Bioengineering, Control (Jacob, 1991)

5. Windows of the Soul Slide 5 UVA - 8/24/98 Criteria for Choosing an Eye Tracking System for the Usability Lab Unobtrusive Easy and convenient to use Few or no restrictions on user population Gaze direction accuracy of 0.5º or better (i.e., 1 cm at 20 in). Effective tracking range of 30º or better High sampling rate Inexpensive Work with off-the-shelf applications Powerful and flexible set of tools for data collection and analysis

6. Windows of the Soul Slide 6 UVA - 8/24/98 The Human Eye

7. Windows of the Soul Slide 7 UVA - 8/24/98 HiMed Subjective Comparison of Eye Tracking Techniques Lo

8. Windows of the Soul Slide 8 UVA - 8/24/98 Pupil-Center/Corneal Reflection Method ERICA System

9. Windows of the Soul Slide 9 UVA - 8/24/98 ERICA System - Continued A view of the infrared image processed by ERICA. The relative position vector between the center of the bright-eye and glint determines the position of the user’s eye-gaze.

10. Windows of the Soul Slide 10 UVA - 8/24/98 A Pilot Study of Web Page Design Evaluate eye tracking data as a user interface evaluation and design tool Apply to a representative usability study of Web page design Understand how best to collect, analyze and interpret the eye tracking data

11. Windows of the Soul Slide 11 UVA - 8/24/98 Pilot Study Methodology 4 Web page designs 4 tasks performed by all testers 1. Do xyz 2. Do abc 3. Do re me 4. Do nada 16 independent testers 4 testers / page design; 1 page design / tester

12. Windows of the Soul Slide 12 UVA - 8/24/98 Original Web Page Design 12 items in web page. Each item ranged from 3 -12 lines of text with 5 - 9 words/line. About half the graphics/images were animated. Scroll about 5 times to see entire page. Testers saw only 2.5 items per screen.

13. Windows of the Soul Slide 13 UVA - 8/24/98 Many Hyperlinks Design Tripled number of text hyperlinks.

14. Windows of the Soul Slide 14 UVA - 8/24/98 No-Image Web Page Design Replaced graphics / images with text labels. Original hyperlinks.

15. Windows of the Soul Slide 15 UVA - 8/24/98 Dense Text Design Book-like, full-width layout. Eliminate graphics / images, two-column appearance and bordered rows. Scroll about 2 times to see entire page. Testers saw 5 of the 12 items on one screen.

16. Windows of the Soul Slide 16 UVA - 8/24/98 Tested Web Page Designs

17. Windows of the Soul Slide 17 UVA - 8/24/98 Data Collection and Analysis Playback Gaze Points - collected raw sampled data Gaze Trails Dynamically defined analysis measures –Fixation Points –LookZones Pupil Diameter

18. Windows of the Soul Slide 18 UVA - 8/24/98 Playback Real-time “video playback” Tracks scrolling events

19. Windows of the Soul Slide 19 UVA - 8/24/98 About 10 -12 samples/sec. Gaze Point Analysis — Raw Sampled Data

20. Windows of the Soul Slide 20 UVA - 8/24/98 Gaze Trails

21. Windows of the Soul Slide 21 UVA - 8/24/98 Gaze Trails with Gaze Points Overlay the sampled gaze points on the gaze trails Gaze trails optional Pop-up details on mouse-over each gaze point

22. Windows of the Soul Slide 22 UVA - 8/24/98 Fixation Points - Example 1 User defined during analysis, after capture. Parameters include: –Duration –Area –Minimum # gaze points More accurate indication of tester’s viewing & processing

23. Windows of the Soul Slide 23 UVA - 8/24/98 Fixation Points - Example 2

24. Windows of the Soul Slide 24 UVA - 8/24/98 Fixation Point Parameter Setting

25. Windows of the Soul Slide 25 UVA - 8/24/98 Determining Fixation Points

26. Windows of the Soul Slide 26 UVA - 8/24/98 LookZones

27. Windows of the Soul Slide 27 UVA - 8/24/98 LookZone Analysis Notice that the subject, searching for the Wireless links, skipped over most pictures and focused on scanning the text blocks for targets.

28. Windows of the Soul Slide 28 UVA - 8/24/98 Bitmap Screen Analysis Options

29. Windows of the Soul Slide 29 UVA - 8/24/98 3D LookZone Analysis In Bitmaps

30. Windows of the Soul Slide 30 UVA - 8/24/98 Vary Gaze Trail Width By Pupil Diameter

31. Windows of the Soul Slide 31 UVA - 8/24/98 Data Analysis - Review of Tasks Task 1: Free Browse Task 2: Directed Search –What is the name of Lucent’s new on-line technology magazine (“e-zine”)? Task 3: Directed Search –What is the name of the software that lets you search for patterns in large masses of data? Task 4: Directed Search and Judgment –How many links on this site will take you to pages dealing with “Wireless” technologies, such as that used in cellular phones.

32. Windows of the Soul Slide 32 UVA - 8/24/98 Comparison of Tasks Mean Time to Complete Tasks

33. Windows of the Soul Slide 33 UVA - 8/24/98 Review of Web Site Designs Four Screen Layouts –Original Text Block/Graphics Format –Many Hyperlinks –No Images –Dense Text

34. Windows of the Soul Slide 34 UVA - 8/24/98 Comparison of Web Designs Mean Time to Complete Tasks & & Average of e-zine and wireless search tasks. * Note: Testers of “Original” design were least experienced group and probably exaggerated the times on this simpler design compared with testers using the busier “Many Hyperlinks” page.

35. Windows of the Soul Slide 35 UVA - 8/24/98 & E-zine search task P-value = 0.03 Mean Time to First Fixation in Target LookZone &

36. Windows of the Soul Slide 36 UVA - 8/24/98 Mean Number of Fixations & & Wireless search task P-value = 0.37

37. Windows of the Soul Slide 37 UVA - 8/24/98 Mean Average Fixation Duration & & Wireless search task P-value = 0.02

38. Windows of the Soul Slide 38 UVA - 8/24/98 Total Fixations in Non-Target LookZones & & E-zine search task P-value = 0.13

39. Windows of the Soul Slide 39 UVA - 8/24/98 Subjective Comparison of Web Designs Highest

40. Windows of the Soul Slide 40 UVA - 8/24/98 Results and Conclusions Correlation among most results –Dense text design elicited best performance Eye movement data confirmed and provided insights into testers performance –Dense text design led to fewer, shorter fixations Results should NOT be viewed as conclusive –Too few subjects, groups differed in skill level –Some experimental design flaws, confounds

41. Windows of the Soul Slide 41 UVA - 8/24/98 Next Steps Replicate the study –Eliminate the potential confounds (Systematic group differences, scrolling, etc.) –Benchmark subjects’ reading speed Expand the study to see if results are generalizable Extend eye tracking methodology to other software applications

42. Windows of the Soul Slide 42 UVA - 8/24/98 References Benel, D. C. R., Ottens, D. & Horst, R. (1991). Use of an eyetracking system in the usability laboratory. Unpublished paper written for The Human Factors Society 35th Annual Meeting. San Francisco, CA Birkmire, D. P., Karsh, R., Barnette, R. D., Pillalamarri, R. & DeBastiani, S. (1993). Eye movements in search and target acquisition. In Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting (pp. 1305- 1309). Santa Monica, CA: Human Factors and Ergonomics Society Card, S. K. (1984). Visual search of computer command menus. In H. Bouma & D. G. Bouwhuis (Eds.). Attention and Performance X: Control of Language Processes (pp. 67-96). London: Erlbaum Harris, Sr. R. L. & Christhilf, D. M. (1980). What do pilots see in displays? In Proceedings of the Human Factors Society 24th Annual Meeting (pp. 22-26). Santa Monica, CA: Human Factors and Ergonomics Society Jacob, R. J. K. (1991). The use of eye movements in human-computer interaction techniques: What you look at is what you get. ACM Transactions on Information Systems, 9 (April), 152-169 Just, M. A. & Carpenter, P. A. (1976). The role of eye-fixation research in cognitive psychology. Behavior Research Methods and Instrumentation, 8, 139-143 Just, M. A. & Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87, 329-354 Kleinke, C. L. (1986). Gaze and eye contact: A research review. Psychological Bulletin, 100, 78-100 Krugman, D. M., Fox, R. J., Fletcher, J. E., Fischer, P. M. & Rojas, T. H. (1994). Do adolescents attend to warnings in cigarette advertising? A eye-tracking approach. Journal of Advertising Research, 34, 39-52 Rayner, K. (1978). Eye movements in reading and information processing. Psychological Bulletin, 85, 618-660 Rayner, K. (1984). Visual selection in reading, picture perception, and visual search: A tutorial review. In H. Bouma & D. G. Bouwhuis (Eds.), Attention and Performance X: Control of Language Processes (pp. 67-96). London: Erlbaum Wierda, M. & Maring, W. (1993). Interpreting eye movements of traffic participants. Chapter 21 in D. Brogan, A. Gale & K. Carr (Eds.), Visual Search 2 (pp. 287-300)

43. Windows of the Soul Slide 43 UVA - 8/24/98