twelve visual perception
Processing in the human visual system (maybe, sort of) eye early vision (striate cortex) contour detection salienceattention object recognition (IT cortex) spatial memory (PP cortex) “what” pathway “where” pathway grouping
The eye Lens Depth of field Chromatic aberration Photoreceptors Rods Cones Fovea, macula, and periphery Retinal processing Gain control Edge enhancement? Simple motion detection lens/iris rods cones retinal ganglion
Photoreceptors Rods Found mostly in the macula and periphery Very sensitive to light But don’t detect color Cones Found in the fovea Less sensitive Color sensitive Colors seem to fade in low light
Trichromacy Having different cones for every possible wavelength would be bad We just have three kinds of cones “Blue” cones: short wavelengths “Green” cones: intermediate wavelengths “Red” cones: long wavelengths However, their responses overlap The eye reduces all the wavelengths at a given pixel to just the total “amount” of “red”, “green”, and “blue”
Components of a color image
Evolution of the color system S to cortex
Evolution of the color system S to cortex L + intensity − chroma (B-Y) cool/warm
Intensity and chromaticity Color image Intensity (R+G+B)Chromaticity (B-Y) white = cool black = warm grey = neurtral
Evolution of the color system S to cortex I + intensity − chroma (B-Y) L − chroma (R-G) cool/warm
Intensity and chromaticity Color image Intensity Chromaticity (B-Y) white = cool black = warn grey = neutral Chromaticity (R-G) white = red black = green grey = balanced
Code [define gray [i → [color i i i]]] [define signed-gray [i → [gray [+ 128 [∕ i 2]]]]] [define b-y [c → [− [blue c] [∕ [+ [red c] [green c]] 2]]]] [map-bitmap [c → [signed-gray [b-y c]]] cones]
Code [define r-g [c → [− [red c] [green c]]]] [map-bitmap [c → [signed-gray [r-g c]]] cones]
Image-level processing Pixel-level processing Massively parallel Preattentive Very fast
Contrast versus absolute intensity The visual system responds less to the amount of light or color Than to variations in the amount This is also known as contrast
Contrast detection Many of the neurons in the visual system are designed to respond to contrast Center/surround receptive fields Stimulated by the presence of light in a specific area Inhibited by its presence in the area immediately around it − + inhibitory region (light prevents firing) excitatory region (light stimulates firing)
Edge detection When applied to image intensity (R+G+B), Center/surround processing responds to lightness contrast Which often occurs at object edges
Scale space Contrast detection is performed at many different scales
Oriented edge detection Other neurons Are stimulated by light in an elongated region And inhibited by it in the region next to it These can detect the orientation of edges −+ −+ −+
Oriented edges
Constancy Illumination varies over a surface in intensity in chroma But you (usually don’t perceive the color of the surface as changing This is called perceptual constancy Color constancy Lightness constancy Georges de la tour, The Repentant Magdalene, C. 1640
Failures of lightness constancy
Failures of color constancy
Depth cues The world is 3D But the image is 2D Your vision system uses depth cues to recover lost information Stereopsis Motion Linear perspective Aerial perspective Relative size
Occlusion The interruption of edges of one object by the edges of another object Shows the interrupting object is in front C. Coles Phillips, Brittish Tank on Fifth Avenue, 1917
Stereo vision Your eyes are in slightly different positions So they receive slightly different images The disparity between images decreases with distance (simple case) So by matching features between the two eyes Your brain computes disparity And hence, depth
Motion parallax Objects farther away seem to move slower So the background seems to remain stable And the foreground moves Or, the opposite, if the camera tracks the foreground This is frequently used in cell animation
Linear perspective Parallel lines converge at a vanishing point One-point perspective One set of lines converging Gives a sense of immersion in the scene Background seems to pop out Two-point perspective Two sets of converging lines The object seems to pop out Three-point perspective Rarely used Very dramatic
Aerial perspective Surfaces get hazier as they recede into the distance
Relative size Apparent size decreases with distance So
Texture gradients Textures get finer as objects recede
Gestalt theory Theory of how we organize components of a percept into wholes Max Wertheimer, Kurt Koffka, Wolfgang Köhler Law of prägnanz (simplicity) We choose the simplest possible interpretation of a scene Formulated laws of grouping Disks or pacman?
Grouping by proximity
Grouping by similarity
Grouping by continuity
Grouping by closure
Grouping by common fate
Figure and ground We tend to separate the scene into One or more objects (“figure”), in front of or on top of A ground plane or background (“ground”) We use a number of tricks to do it Depth, grouping But it can be hard in degenerate cases
Attention Remember our fovea is only about 2° So we have to view scenes by scanning the fovea over the image Attention is driven both top-down by cognition Task Knowledge, expectations And bottom-up by Size Intensity Local contrast Motion Odd-man-out As well as recency (don’t look if we’ve just looked there)
Attention and scale space Attention starts at coarse (large) scales And narrows to finer scales
Attention and intensity All things being equal, our attention is driven to intense stimuli Bright lights Saturated colors Rapid motion
Attention and color Warm, bright colors draw attention Cooler, darker colors tend to recede into the background Claude Monet, Sunrise, 1872
Attention and contrast But one of the biggest drivers of attention is contrast We focus on unusual elements of the scene Elements that are different from their neighbors
Attention follows curves
Attention follows perspective and depth
Top-down attention Attention is also driven by expectations If I see something face-like I look for the facial features Or vice-versa And by task
Change blindness We only really see what the fovea is fixating If anything changes, we don’t notice unless we see motion Oh, and we don’t see motion while our eye is moving … Continuity in films Pretty woman Neisser et al’s demo Neisser et al’s demo Simons and Chabris’ demo SupplyPattern Work area Adapted from Ballard et al.
Shape recognition We don’t really know how this works It’s really hard … And there are a lot of high level semantic effects