1 A biologically useful memory mechanism for the rapid deployment of visual attention Ken Nakayama

2 seeing = visibility X attention + R

3 Metaphors for vision Camera: It’s like a picture Hand: it’s more active Attention is the hand How is it controlled?

4 Background some robust examples of attention Change blindness Inattentional blindness (even at fovea) Attentive tracking “hand and fingers”

5 Rensink flicker experiment method: alternate two pictures –Ask subjects to identify changes If we were aware of everything in picture, should be easy raise your hand when you see the change don’t tell others At Nissan CBR

6 Rensink et al., Simons and Levin Change Blindness

7

8 summary of Rensink flicker experiment contrary to our phenomenological experience we are not aware of everything in our visual world large changes can escape our notice Conclusion: seeing requires attention

9 attention as an inertia-less hand can change size and position approx 4-6 times/sec

10 Other functions of attention Guidance of motor behavior Foraging for food

11

12

13 The intensity of predation depends... on the use of specific searching images. This implies that the birds perform a highly selective sieving operation on the visual stimuli that reach their retina... birds can only use a limited number of different search images at the same time. L. Tinbergen(1960)

14 Marian Dawkins (1971) Shifts of ‘attention’ in chicks during feeding

15 Search image widespread? Chicks (Dawkins, 1971) Pigeons (Reid and Shettleworth, 1992)) Blue Jays (Bond) Bumble Bees (1992) Butterflies (Stanton, 1984) Is it a mental image or could it be something else?

16 Claim There exists a primitive distributed memory system (seen in our human experiments) that could account for shifts of attention attributed to search images

17 It is part of a fast transient attentional system (Nakayama and Mackeben, 1989 Vision Research)

18 msec

19 Isolation of the transient component Keep location constant

20 Is the transient component due to its activation by a sensory transient ? NO such sensory transients not necessary analogy to “action potential ?”

21 Deploy transient attention without a local sensory transient

22 decoy cueing normal cueing

23 Transient attention is very fast, rises to peak within 100 msec. Can transient attention learn to go quickly to the appropriate position on a larger object ? Is it fast AND flexible Kristanjansson, Mackeben & Nakayama, 2001

24

25 fixed variable learning, Attention can be effectively deployed to a location within an object

26 Is the learning a property of sustained or transient attention ?

27 vary cue lead time Keep target position within the cue constant

28 Learning How fast does it occur ? How stable is it ? Method: use quasi random streaks of cue target regularities

time sequence of cues % correct Build-up of learning streak

30 color ? streak

31 percent correct local shape ?

32 Learning (summary so far) Is very rapid and is temporary Can be linked position within an object Can linked to a color within an object Can linked to a local shape within an object Are there things “attention” can’t learn ?

33 can it learn a 2nd order relation? + + color and position shape and position

34 random consistent AK AMH 2 subjects

35 Part of a fast mechanism of Attentional deployment, reaches peak with 100 msec

n-1 n Identify shape of the odd colored target Maljkovic and Nakayama Different paradigm to study the same process position and color of the target can change

n repeat color

Order in Sequence Repeat target position n..

39 Attention can learn colors, shapes and locations Is attention, (or are we) learning what to expect, then forming a search image in our minds ? Approach: manipulate target color uncertainty

40 prediction ?

41

42 make expectancy very explicit maximizing the possible use of search images, pitting it against repetition 1 2 Double alternation paradigm Target color over trials

43 21 Trial sequence: order in sequence repetition reaction time predicted outcomes 1 2 Search image (expectancy)

44 1 2

45 Learning is passive, mechanistic, piecemeal (color, position) Its not expectancy, Its not a search image Its not under conscious control

46 Its not just a linkage to the previous targets but active inhibition to non-targets

KN Distractor varies distractor same SS

48 Fine grain temporal analysis of the learning mechanism ~ 2nd order reverse correlation What is the influence of a single trial in the past ?

49 What is influence of a single trial in the past ? current trial pastfuture time

50

51

52 Linkage is not confined to color Probably any salient feature (spatial frequency, for example), will do

53

54 Learning restricted to what attracts attention (features, positions) Not the fine details that attention allows one to process

55

56 Is the build up and decay over time or over events?

57 Independence of features/location

58 Effect of target and distractor position in the past

59 influence of trial n-1

60 Independence of color and location learning

61 hypothesized properties of the memory system graded summates linearity (superposition) has independent components (features and locations)

62 leaky bucket

63 What is learned ? What is the reinforcer ? Simple identifiers of places where attention just went and didn’t go ? Memory kernel function

64 Generalizations beyond the measure of attention Speeds eye movements (human and monkey) Speeds motor behavior (manual pointing)

65 Saccadic eye movements (in human and in monkey)

66 Task: make a saccade to the odd colored target Human eye movements

67 saccadic latency position in same color sequence McPeek, Maljkovic & Nakayama Cumulative effects of learning latency difference pastfuture Memory kernel

68 Learning of speeded saccades in monkey Position in same color sequence Saccadic latency Robert McPeek and Ed Keller Smith Kettlewell

69 Learning generalizes to manual pointing + Trial 1 Measure RT with Touch sensitive screen (Song & Nakayama, 2003)

70 Learning generalizes to manual pointing + Trial 1 Trial 2 Measure RT with Touch sensitive screen (Song & Nakayama, 2003)

71 Touching target with finger ….. RT(ms) Position within “Same color” Sequence Target:

72 Implications for foraging Don’t need a cognitive concept like a search image Low level temporary, passive, graded connection strengths (plus and minus) may be sufficient

73 Marian Dawkins (1971) Shifts of ‘attention’ in chicks during feeding

74 Anatomical locale/mechanism? Object centered not retinotopic cortex Independent features simple 2 layered network

75 Relationship to other learning systems Is it a completely specialized sub- system for attention ? OR Is it the germ for short term memory more generally ? X Hypothesis: it’s a biologically conserved memory system for the rapid deployment of visual attention.

76 A biologically useful and conserved memory mechanism for the rapid deployment of visual attention (and other possible functions Ken Nakayama Manfred Mackeben Vera Maljkovic Robert McPeek Arni Kristjansson Joo-Hyun Song