1. Getting a Measure of Satisfaction from Eyetracking in Practice Workshop 24 Organisers: Tony Renshaw, Leeds Metropolitan University, UK Natalie Webb, Amberlight Partners, London Janet Finlay, Leeds Metropolitan University, UK

2. Aims: Initially –Define best practice in eye tracking Suggest solutions to problems Highlight unanswered questions Explore both scientific and commercial use –Explore how to measure satisfaction with eyetracking Adjusted through the day!

3. Format of Workshop: Presentations – 3 minutes per person identifying 3 key issues from position papers Participants note issues and questions of interest from presentations on post-its Participants organise their questions under themes for discussion and choose groups Break out groups – 3 AM, 3 PM Presentations from break out groups: –Areas of agreement –Unresolved issues –Future Work Summary – key take away points

4. Breakout Sessions 6 themes: 1.When should we use/not use eyetracking? 2.What are the best eye tracking methodologies? 3.How do we make eye tracking data analysis manageable? 4.How can we analyse gaze paths? 5.How can we tie eye tracking metrics into measuring user experience? 6.How do we deal with dynamic stimuli?

5. Participants Representing 6 countries and 3 continents Representing both research and commercial users Sune Alstrup Duncan Brumby Edward Cutrell Andrew Duchowski Laura Granka Ying-Hua Guan John Hansen Keith Karn Craig Lindley Janet Read Jens Riegelsberger Alain Robillard-Bastien Kerry Rodden Anthony Santella Charlotte Sennersten Katerina Tzanidou

6. 1. When should we use/not use eye tracking? Areas of agreement –Use when appropriate but not always! Context and purpose of project is important – and very different between commercial and research Should always consider complementary techniques Should recognise and work within the limitations of the technologies Unresolved issues –How can we best combine eye tracking with other methods How much influence and interference? –Replay gaze trails → post hoc rationalisation? –Concurrent think aloud → alter eye movement behaviour? –Dealing with limitations from mind-eye hypothesis –Common reporting standards needed e.g. providing full information about calibration and people who could not be tracked Future work –Investigate individual and population differences in eye gaze –Better understand the influence of sampling on eye tracking studies –Find ways to combine physiological and eye tracking data

7. 2. What are the best eye tracking methodologies? Areas of agreement –Must have a list of methods in order to provide “best practices” –Cannot make sense of data external to intentions Very different intentions for different research goals (usability vs cognitive models) Unresolved issues/Future work –Refining eye tracking methodologies –Further detailing the best usage of retrospective reporting vs. concurrent think aloud

8. 3. How do we make eye tracking data analysis manageable? Areas of agreement –Plan carefully with a clear view of purpose, problem being addressed and tasks –Choose appropriate numbers of participants, task sizes and session lengths –Select appropriate fixation parameters (duration and area) Unresolved Issues –Analysis software provided with eye tracking hardware not adequate – need better facilities for Filtering Summation Vizualization –Improving ways to combine data with other protocols; e.g. verbal –Studies involving different applications have different analysis requirements e.g. games, learning systems Future Work –Can borrow analysis algorithms from other areas e.g. bioinformatics, machine learning –Need Open Source analysis tools –Need more publications focusing on methodology and analysis techniques

9. 4. How can we analyse gaze paths? Areas of interest –Emerging standard E.g. String Edit Distance Unresolved issues –Dealing with Strings of varying lengths Comparison of multiple strings Visual presentation of analysis Very long sequences –Integration of data from other sources –Annotation of edited strings with comments Future Work –All of the above!

10. 5. How can we tie eye tracking metrics into measuring user experience? Areas of agreement/Unresolved Issues –Definition of satisfaction is problematic – depends on domain – engagement or immersion may be more useful –Currently lack of models connecting eye tracking metrics and higher level measures of user experience –Would like to be able to correlate particular behaviours with measures –Might be more helpful to approach this from behavioural rather than cognitive perspective Future work –Need more models linking eye behaviour to user experience e.g. Novice vs. expert behaviour Certain scan paths indicating a certain depth of engagement (e.g. in games) –Using eye movement behaviour to inform design through “design patterns”

11. 6. How do we deal with dynamic stimuli? Areas of agreement –Very little known! Unresolved issues –Large variety of stimuli formats e.g. games, web applications, java, flash –Problems of tracking moving AOI (areas of interest), definition of dynamic AOI –Labelling of data is labour intensive –What granularity of information is required? –How can scan paths be determined in this type of stimuli? Further work –Formulate appropriate cognitive and visual influence theories –Explore role of peripheral vision in dynamic environments –Explore influence of auditory information –Develop suitable strategies for determining AOI online or post hoc Extract stimuli information directly from the application and integrate with eye tracking data

13. Outcomes: Aim 1: “Best practice” is premature – we first need a better understanding of possibilities Aim 2: “Satisfaction” seen as a problematic construct – we need to develop more meaningful ways of interpreting eye movement data within specific contexts e.g. identifying common patterns of behaviour for specific scenarios

14. Wish list: High level models and metrics Better analysis algorithms More flexible and powerful analysis tools Methodologies for combining eye tracking with other usability approaches Better understanding of constraints and parameters for eye tracking studies in specific contexts More research!

15. If you are interested in eye tracking either as a researcher or for commercial use, we would like to talk to you! Please leave us your card in the tray provided and/or take one of ours. Thanks for your interest.