Eye Movements and Working Memory Marc Pomplun Department of Computer Science University of Massachusetts at Boston Homepage:

Eye Movements and Working Memory Overview: Image Processing: Convolution Filters Iconic Memory Representations for Visual Search Working Memory Use in a Natural Task The Working Memory - Eye Movement Tradeoff

Convolution Filters Grayscale Image: /9 Averaging Filter:

Image Processing Original Image: Filtered Image: /9 value = 1  1/9 + 6  1/9 + 3  1/9 + 2  1/  1/9 + 3  1/9 + 5  1/  1/9 + 6  1/9 = 47/9 =

Image Processing Original Image: Filtered Image: /9 value = 6  1/9 + 3  1/9 + 2  1/  1/9 + 3  1/  1/  1/9 + 6  1/9 + 9  1/9 = 60/9 =

Image Processing Original Image: Filtered Image: Now you can see the averaging (smoothing) effect of the 3  3 filter that we applied.

Gaussian Filters Discrete version: 1/273 implement decreasing influence by more distant pixels

Gaussian Filters original 3333999915  15 Effect of Gaussian smoothing:

Sobel Filters Sobel filters are an example for edge detection filters. Two small convolution filters are used successively: SxSx SySy

Sobel Filters Sobel filters yield two interesting pieces of information: The magnitude of the gradient (local change in brightness): The angle of the gradient (tells us about the orientation of an edge):

Sobel Filters Original image (left) and result of calculating the magnitude of the brightness gradient with a Sobel filter (right)

Rao, Zelinsky, Hayhoe & Ballard (2002): Eye Movements in Iconic Visual Search Question: How do people represent items in their memory for efficient visual search? Idea: Iconic (appearance-based) multiscale representation Such representations were modeled using spatiochromatic convolution filters of different scales and orientations.

Rao, Zelinsky, Hayhoe & Ballard Convolution filters used for the model.

Rao, Zelinsky, Hayhoe & Ballard According to the model, iconic visual search proceeds as follows: The first saccade is aimed at the point in the visual scene whose low-frequency features have the best match with the low-frequency features of the memorized object. For the programming of the following saccades, higher and higher frequencies are included, until the target is found.

Rao, Zelinsky, Hayhoe & Ballard Coarse-to-fine scanning mechanism

Rao, Zelinsky, Hayhoe & Ballard Conclusion: Good correspondence between modeled and empirical scanpaths

Ballard, Hayhoe & Pelz (1995): Memory Representations in Natural Tasks Task: Copy a pattern of colored blocks

Ballard, Hayhoe & Pelz Possible strategies for completing the block copying task. Participants performed the following operations: (M)odel inspection, (P)ickup, and (D)ropoff.

Ballard, Hayhoe & Pelz Typical hand and gaze trajectories for a single copying step

Ballard, Hayhoe & Pelz Empirical frequency of individual strategies in the block copying task

Ballard, Hayhoe & Pelz Conclusions: In the block copying task, working memory is only sparsely used. Instead, subjects prefer to make additional eye movements. Because eye movements are “inexpensive”, subjects use the visual scene as an “external memory” rather than building an internal representation of it.

Eye Movement - Working Memory Tradeoff (Inamdar & Pomplun, 2003) Based on the previous study by Ballard et al, it seems that using working memory is clearly more “expensive” than performing eye movements. So maybe a “cost model” is an adequate way of describing and predicting behavior in visual tasks.

Inamdar & Pomplun The basic idea is that the visual system (including the cognitive mechanisms that are required for performing the task) optimizes visual behavior, i.e. minimizes its effort (cost). Is there such a tradeoff between the use of working memory and eye movements? If so, what exactly is minimized? Can this be quantified?

Inamdar & Pomplun Let subjects perform a visual task that requires eye movements and extensive use of visual working memory. Vary the “cost” of eye movements. Hypothesis: If the assumed tradeoff between eye movements and working memory exists, costlier eye movements should lead to increased use of working memory.

Stimuli in Experiment 1 Subjects were presented with two columns of simple geometrical objects in three different colors and three different shapes. The columns were identical except for one object that differed in either its color or its shape (in target- present trials). Subjects had to indicate whether such a target was present or not. The objects in the non-attended hemifield were always masked. The cost of eye movements was varied by changing the distance between the two columns.

Stimuli in Experiment 1

Eye Movements in Experiment 1

Results of Experiment 1

Experiment 2 What happens if the capacity limit of visual working memory is reached? By just varying the distance between columns, the cost of eye movements cannot be dramatically increased. Idea: “Artificially” increase the cost of eye movements in the present paradigm by delaying the unmasking of objects after any gaze switch between hemifields.

Stimuli in Experiment 2 We used the same stimuli as in Experiment 1, but only those for the “medium-distance” condition. Three visibility delays were used: 0ms, 500ms, and 1000ms. During the delays, objects in both hemifields were masked.

Results of Experiment 2

Conclusions There clearly is a cost-minimizing behavior with regard to eye movements and working memory. However, the current data does not allow to build a quantitative model of this phenomenon. It seems that people slightly overestimate their working memory capacity when they are forced to heavily increase their working memory load.