Michigan State University 1 “Saliency-Based Visual Attention” “Computational Modeling of Visual Attention”, Itti, Koch, (Nature Reviews – Neuroscience 2001 ) “A Model of Saliency-Based Visual Attention for Rapid Scene Analysis”, Itti, Koch and Niebur’s (IEEE PAMI 1998) Zhengping Ji

Michigan State University 2 Overview l Background l System architecture l The saliency map l Preprocessing l Feature maps l Feature integration l Focus of attention l Results l Conclusion

Michigan State University 3 Related Work l “Feature Integration Theory,” Treisman & Gelade, l Computational model of bottom-up attention, Koch and Ullman, 1985 l Saliency map is believed to be located in the posterior parietal cortex (Gotlieb, et al., 1998) and the pulvinar nuclei of the thalamus (Roinson & Peterson, 1992)

Michigan State University 4 Architecture

Michigan State University 5 Gaussian Pyramids l Repeated low-pass filtering l =0, 1, 2, 3,…,8 I(0) is original input 640 x x x x60 Scaling by a factor 2x2 * G 5x5 Scaling by a factor 2x2 * G 5x5 *

Michigan State University 6 Preprocessing l Original image with red, green, blue channels Intensity as I = (r + g + b)/3 l Broadly tuned color channels R = r - (g + b)/2 G = g - (r + b)/2 B = b - (r + g)/2 Y = (r + g)/2 - |r – g|/2 - b

Michigan State University 7 Preprocessing RG B Y Intensity

Michigan State University 8 Center-surround Difference l Achieve center-surround difference through across-scale difference Operated denoted by  Interpolation to finer scale and point-to- point subtraction One pyramid for each channel: I(  ), R(  ), G(  ), B(  ), Y(  ) where   [0..8] is the scale

Michigan State University 9 Intensity Feature Maps I(c, s) = | I(c)  I(s) | c  {2, 3, 4} s = c +  where   {3, 4} So I (2, 5) = | I (2)  I (5)| I (2, 6) = | I (2)  I (6)| I (3, 6) = | I (3)  I (6)| … l  6 Feature Maps

Michigan State University 10 Colour Feature Maps Similar to double-opponent cells (Prim. V. C) Red-Green and Yellow-Blue RG(c, s) = | (R(c) - G(c))  (G(s) - R(s)) | BY(c, s) = | (B(c) - Y(c))  (Y(s) - B(s)) | Same c and s as with intensity +R-G +G-R +B-Y +Y-B

Michigan State University 11 Orientation Feature Maps Create Gabor pyramids for  = {0º, 45º, 90º, 135º} c and s again similar to intensity

Michigan State University 12 Normalization Operator l Promotes maps with few strong peaks l Surpresses maps with many comparable peaks 1. Normalization of map to range [ 0…M ] 2. Compute average m of all local maxima 3. Find the global maximum M 4. Multiply the map by ( M – m ) 2

Michigan State University 13 Normalization Operator

Michigan State University 14 Conspicuity Maps

Michigan State University 15 Saliency Map l Average all conspicuity maps

Michigan State University 16 Neural Layers l Saliency Map (SM) modeled as layer of leaky integrate-and-fire neurons l SM feeds into winner-take-all (WTA) neural network l Inhibition of Return as transient inhibition of SM at FOA SM Stimulus WTA Inhibition of Return FOA shifted to position of winner

Michigan State University 17 Example of Operation Inhibition of return

Michigan State University 18 Results Image Saliency Map High saliency Locations (yellow circles)

Michigan State University 19 Shifting Attention l Using 2D “winner- take-all” neural network at scale 4 l FOA shifts every ms

Michigan State University 20 Summary l Saliecy map can be broken down into main steps l Create pyramids for 5 channels of original image l Determine feature maps then conspicuity maps l Combine into saliency map (after normalizing) l The key idea of saliency map is to extract local spatial discontinuities in the modalities of color, intensity and orientation. l Use two layers of neurons to model shifting attention. l Model appears to work accurately and robustly (but difficult to evaluate)

Michigan State University 21 Discussion l No top-down attention modeling, e.g., top-down spacial control, obejct-based attention. l Biologically plausible? l Neuromorphic architecture? l In which way the top-down and bottom-up processes are related? l In which way the attention and recognition are integrated and interacted with each other?

Michigan State University 22 References Itti, Koch, and Niebur: “ A Model of Saliency-Based Visual Attention for Rapid Scene Analysis ” IEEE PAMI Vol. 20, No. 11, November (1998) Itti, Koch: “ Computational Modeling of Visual Attention ” Nature Reviews – Neuroscience Vol. 2 (2001)