Eye Movements, Attention, and Working Memory in Natural Tasks Mary Hayhoe Center for Perceptual Systems

Even the simplest action must involve linkage between attention, memory, vision, eye movements, and body movements. from Land et al, 1999

How is vision used to acquire information from the world? What image information is needed for natural visually guided behavior?

Why Study Natural Behavior? Characterize what needs to be explained. Generalize (or not) from findings in simpler environments and perceptual tasks. Basis for clinical assessment/rehabilitation (brain injury/retinal disease) and for human-computer interactions

Development of visual displays To 1970’s: Optical Systems/Tachistoscopes: spots, flashes, line drawings 1970’s-2000+: CRT’s: sinusiodal grating stimuli Mid 1990’s- : 2D chromatic images of scenes Late 1990’s: HMD’s: complex, 3D virtual environments

Developments in Eye Tracking Head fixed /restricted : Contact lenses: mirror / magnetic coils Early infra-red systems Dual Purkinje Image tracker Head Free: Head mounted IR video-based systems Remote systems with head tracking Scene camera

Head-mounted, IR video-based trackers Portable: Pelz RIT ASL501 + Polhemus Fastracks for head and hands Freedom of movement allows a wider variety of behaviors mirror Scene camera Polhemus Fastrak

Driving: fixate tangent point while driving around a curve Fixation density Use of a single control variable: gaze angle > steering angle (Land & Lee, 1992)

Model Workspace Resource Area (Ballard et al 1995) eye hand “Just-in-time” strategy

Fixations tightly linked to actions.

Vision is active not passive. Specific information is usually acquired at the fixation point. Information is acquired “just-in-time”. Fixations patterns reflect learning at several levels: what objects are relevant/where information is located/order of sub-tasks/properties of world. Main Insights from Natural Tasks

In the context of natural behavior, humans acquire information from the scene sequentially and selectively. What controls this process?

Fundamental Constraints Acuity is limited. High acuity only in central retina. Attention is limited. Not all information in the image can be processed. Visual Working Memory is limited. Only a limited amount of information can be retained across gaze positions.

Standard paradigms: brief presentations - what changes? Subjects remember 3-4 items in form of “object files”. (eg Luck & Vogel, 1997)

How do attentional and memory limitations play out in natural behavior? Need to understand Usage, ie mechanisms than control allocation of gaze, attention, and memory, not just Capacity The Question - Natural behavior : sequences of operations over several sec - selection and timing under observer’s control. - Trial structure of standard paradigms : repeated instances of a single operation. Experimenter controls timing and nature of selection.

Cognitive Goal Sub-task Fixation Acquire Info How selective? What is stored? Make PBJ sandwich Get jelly Fixate jelly jar Object? Feature? Fixations alone don’t specify what information is selected. Understanding Task Micro-structure

Phantom Force Feedback System Haptic feedback for 2-fingered grasping. Eye tracker mounted inside the virtual reality helmet

Virtual Research V8 head Mounted Display Head position: Polhemus Fastrack ASL 501 Eye Tracker Infra-red, video based ASL limbus tracker Eye tracking in Virtual Environment Saccade detection for image changes during saccades

“Pick up any red brick.” PICK-UP CUE: Height Width Color Texture FEATURES RELEVANT IRRELEVANT FINGERTIPS

PUT-DOWN CUE: “Place the red brick on the right.”

Example of a One-Feature Trial

Example of a Two-Feature Trial

Predictable Same feature(s) repeatedly used for pick-up and put-down Trial Structure Two Types of Trial Blocks Unpredictable Same feature used for pick-up, ANY feature could be relevant for put-down One-Feature Two-Feature Pick up RED. RED on right. Pick up RED. TALL on right. Pick up RED. RED on right. Pick up RED. TALL on right.

Scene Acquire PU Cue Acquire FORGET PU Feature Acquire PD Cue Guide Hand to Belt Guide Hand ACQUIRE PD Feature PU Feature Low Memory Load Two Possible Strategies Acquire PU Cue Acquire RETAIN Acquire PD Cue Guide Hand to Belt Guide Hand RECALL Brick Features PD Feature “High” Memory Load Fixation Patterns

Fixation Pattern May Reveal Memory Strategy Acquire PU Cue Acquire FORGET PU Feature Acquire PD Cue Guide Hand to Belt Guide Hand ACQUIRE PD Feature PU Feature Acquire PU Cue Acquire RETAIN Acquire PD Cue Guide Hand to Belt Guide Hand RECALL Brick Features PD Feature Sorting based on information in scene Sorting based on working memory High Memory LoadLow Memory Load

One-FeatureTwo-Feature Unpredictable Sorting based on working memory Sorting based on information in scene Fixation Sequence/Time of Acquisition Depends on Memory Demands One-FeatureTwo-Feature Predictable Probability of Fixation Sequence

Fixation sequence during a trial depends on what information subjects need later in the trial. When Subjects are uncertain of what they need, re-fixate the brick to acquire the second feature later in trial, presumably to reduce memory load. Suggests subjects often acquire only partial information about brick features during pick-up. Fixation Patterns Reveal Subtle Control by Task

1.Fixation Sequence: Delay acquisition when unpredictable and greater memory load. 2.Change Detection

Experimental Logic Change both task relevant and irrelevant features of the object the subject is holding Greater sensitivity to relevant changes suggests task-specific representations. Normalized Trial Length Bricks in Array Pick-Up Cue Put-Down Cue Brick in Hand Conveyor Belts Left Belt Right Belt Probability of Fixation Brick 1 Brick 2 Brick 3 Brick 4 Brick 5 Fixations How Specific is Visually Acquired Information? ? memory

Up to 8 changes per 80 trials Feature Changes -- When the brick is being carried towards the area for sorting -- During a saccade Changed feature may be relevant or irrelevant to task

TRASH CAN: Dispose of any brick with a changed feature.

Subject fails to detect color change

Relevant Changes Noticed Most Often Predictable Unpredictable Relevant Changes Noticed Most Often Less Effect of Relevance when Unpredictable

Attention selects features, not objects (in this task). Features selected, and time of selection, is controlled by the current microtask and minimizes memory load. Object representations are not necessarily maintained in memory as bound entities or “object files”. Commonly accepted view: Attention binds features into object representations. Remember the attended objects in form of Object Files. Approximately 4 object files held in working memory across gaze positions. (Treisman, 1988; Irwin & Andrews, 1996; Luck & Vogel, 1997; Rensink, 2000; Wheeler & Treisman 2003) Present results suggest:

1.Fixation Sequence: Delay acquisition when unpredictable and greater memory load. 2.Change Detection: Visual acquisition and storage of information is highly selective 3. Sorting Performance

Unpredictable Irrelevant PU Relevant PD Relevant Two-Feature Less Effect of Relevance when Relevance is Unpredictable Predictable Rate of Change Detection Relevant Changes Noticed Most Often PU & PD Relevant Irrelevant One-Feature * Irrelevant PU Relevant PD Relevant Two-Feature * PU & PD Relevant Irrelevant One-Feature * Rate of Change Detection MISSED! MISS! What happens when a change is missed?

Why are changes not noticed? Perhaps changes are not noticed due to a failure to re-fixate/encode the new stimulus after the change. Insensitivity to changes has been interpreted as evidence for poor memory of the pre-change stimulus. (O’Regan, 1992; Rensink, 1997; Simons, 2000)

Sort by Old Working memory of old feature used for sorting decision Two Possible Sorting Decisions Failure to Update New Feature Sort by New New feature used for sorting decision Failure to Maintain Old Feature

Subject sorts by NEW feature.

Subject sorts by OLD feature

One- Feature Two- Feature Unpredictable Sort by Old Sort by New Sort by Old When Predictable Sort by OldSort by New Sort by New When Unpredictable One- Feature Two- Feature Predictable Fraction of Missed Trials

Sort by Old Working memory of old feature used for sorting decision Fixation Duration 997ms 747ms Sort by old feature despite long fixation on brick

Significance of Sorting by Old Use of memory rather than current sensory data, despite long fixation on brick, may result from attentional demands of brick placement. (“Inattentional blindness”: Mack & Rock, 1998 )

Observers may not re-sample the image to update information because features are typically stable. Hypothesis: Another factor in attentional selection and working memory use is subjects’ knowledge of properties of world. Implication: we need internal models of the scene after all! Failure to detect change doesn’t imply absence of representation, merely the wrong one. Significance of Sorting by Old (ctd)

Delay acquisition of task relevant information until necessary. (“Just-in-time” representations: Ballard et al., 1995) Strategy to delay acquisition depended on predictability and memory load. Presumably such trade-offs intrinsic to natural behavior. Sorting by new in unpredictable case presumably a consequence of greater probability of re-fixation after put-down cue. Sort by New When Unpredictable

1.Fixation Sequence: Delay acquisition when unpredictable and greater memory load. 2.Change Detection: Visual acquisition and storage of information is highly selective. 3. Sorting Performance: Working memory used instead of current image. 4. Duration of Fixations and Hand Movements

Subjects modify gaze patterns following a detected change. Brick Fixation Duration Noticed Change Trials After Change Trials Before Change Difference in Movement Duration (sec) Difference in Fixation Duration on Brick (sec) Hand Movement Duration Longer brick fixations and hand movement during trials following a noticed change. Subjects may have reprioritized the change detection task. (Predictable condition shown only) One-FeatureTwo-Feature

Change in fixation durations and hand movement duration following a detected change are consistent with re-allocation of attention to the change-detection task/ change in task priorities. Consistent with recent evidence showing sensitivity of oculomotor responses to stimulus probabilities and reward structure of the task. (Platt & Glimcher, 1998; Stuphorn & Schall, 2000; Hikosaka, 2000 etc) Sensitivity to reward provides a potential mechanism for mediating flexible task-specific modulation of attention. (cf Ballard & Sprague,2004)

Summary - In natural tasks, fixations are restricted to task-specific locations and tightly linked in time to immediate tasks demands (often use just-in-time strategy). - Selection of information within a fixation may be highly specific, and is determined by momentary task. -Subjects are sensitive to changes in environment that require changes in eye movement strategies. - Fixations reflect dynamic optimization with respect to a set of constraints, perhaps metabolic, computational, or time costs. -Once selected, information is usually retained. Suggests scene representations built up over multiple fixations, and updated depending on dynamic properties of world.

Regularity of natural behavior makes controlled investigation possible. The variety of behavioral measures, together with task context allows stronger inferences. Comments Understanding fixation patterns in natural behavior will require an understanding of way tasks are learnt represented in the brain. Reinforcement learning models of complex behavior necessary to explain fixation patterns/ sequential acquisition of visual information. There is always a “task”.

Acknowledgments: Jason Droll Brian Sullivan

What’s involved in making a saccadic eye movement? Behavioral goal: make a sandwich Sub-goal: get peanut butter Visual search for pb: requires memory for eg color of pb or location Visual search provides saccade goal - attend to target location Plan saccade to location (sensory-motor transformation) Coordinate with hands/head Calculate velocity/position signal Execute saccade/

target selection signals to muscles (forces) inhibits SC saccade decision saccade command (where to go) monitor/plan movements H V 1. Neural activity related to saccade 2. Microstimulation generates saccade 3. Lesions impair saccade Brain Circuitry for Saccades

Brain areas involved in making a saccadic eye movement Behavioral goal: make a sandwich (learn how to make sandwiches) Frontal cortex. Sub-goal: get peanut butter (secondary reward signal - dopamine - basal ganglia) Visual search for pb: requires memory for eg color of pb or location (memory for visual properties - Inferotemporal cortex; activation of color - V1, V4) Visual search provides saccade goal. LIP - target selection, also FEF Plan saccade - FEF, SEF Coordinate with hands/head Execute saccade/ control time of execution: basal ganglia (substantia nigra pars reticulata, caudate) Calculate velocity/position signal oculomotor nuclei Cerebellum?