1. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 1 CS 664, USC Spring 2002 Lecture 5. Visual Attention (bottom-up) Reading Assignments: None

2. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 2

3. 3

4. 4 Several Forms of Attention Attention and eye movements: - overt attention (with eye movements) - covert attention (without eye movements) Bottom-up and top-down control: - bottom-up control based on image features very fast (up to 20 shifts/s) involuntary / automatic - top-down control may target inconspicuous locations in visual scene slower (5 shifts/s or fewer; like eye movements) volitional Control and modulation: - direct attention towards specific visual locations - attention modulates early visual processing at attended location

5. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 5 What is attention then? Attention is often described as an information processing bottleneck. Controls access to higher levels of processing, short-term memory and consciousness. Hence, the strategy nature has developed to cope with information overload is to break down the problem of analyzing a visual scene: from a massively parallel approach to a rapid sequence of circumscribed recognitions.

6. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 6

7. 7

8. 8

9. 9

10. 10

11. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 11

12. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 12

13. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 13

14. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 14

15. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 15

16. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 16 First Computational Model Didday & Arbib, 1975 introduced a “two visual systems” framework Koch & Ullman, Hum. Neurobiol., 1985 Introduce concept of a single topographic saliency map. Most salient location selected by a winner-take-all network.

17. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 17

18. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 18 Shifter Circuits Anderson & van Essen, PNAS, 1987 Information dynamically routed through cortical hierarchy. Yields rotation- and scale-independent representation.

19. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 19 Shifter Circuits (cont.) Olshausen et al., J Neurosci, 1993 Implemented shifter circuits and demonstrated proof of concept. Control neurons in the pulvinar send the (attention-based) control signals that will determine the “passing” region of the circuit, through a modulation of intracortical connection weights. Perform recognition using associative memory at top level.

20. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 20 only attended item reaches output layer

21. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 21 Selective Tuning Model Tsotsos et al., Artificial Intelligence, 1995 - attention modulates neurons to earliest levels; wherever there is a many-to-one mapping many-to-one mapping - signal interference controlled by surround inhibition throughout processing network throughout processing network -task knowledge biases computations throughout processing network - attentional control is local, distributed and internal - competition is based on WTA (different form than previous models) (different form than previous models) - pyramid representation with reciprocal convergence and divergence neuron ‘sees’ this receptive field subject ‘attends’ to single item

22. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 22 The basic idea (BBS 1990)

23. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 23 Selective Tuning Model processing pyramid inhibited pathways pass pathways unit of interest at top input Caputo & Guerra 1998 Bahcall & Kowler 1999 Vanduffel, Tootell, Orban 2000 Smith et al. 2000 Kastner, De Weerd, Desimone, Ungerleider, 1998

24. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 24

25. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 25 Guided Search Wolfe, Psychonomic Bull. & Rev., 1994 How can we combine information from several modalities? Use top-down (task-dependent) weighting.

26. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 26

27. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 27

28. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 28

29. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 29

30. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 30

31. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 31

32. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 32

33. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 33

34. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 34

35. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 35

36. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 36

37. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 37

38. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 38

39. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 39

40. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 40

41. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 41

42. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 42 Evaluation of Advertising

43. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 43

44. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 44

45. Michael Arbib & Laurent Itti: CS664 – Spring 2002. Lecture 5: Visual Attention (bottom-up) 45 Brefczynski & DeYoe, Nature Neuroscience 1999