1 Computational Vision CSCI 363, Fall 2012 Lecture 16 Stereopsis

2 Random Dot Stereogram

3 Transformed

4 Linear Systems Linear functions: F(x 1 + x 2 ) = F(x 1 ) + F(x 2 ) F(ax) = aF(x) Linear systems are nice to work with because you can predict (or compute) the responses of the system relatively easily. For example, if you double the input, the output doubles. Fourier Transforms are linear operations. (The Fourier transform of the sum of two images is the sum of the Fourier transforms of each image). Gabor filters are linear filters. Neurons are not linear.

5 Threshold and Saturation Threshold non-linearity: Neurons do not respond until the input reaches a minimum level (threshold). Saturation non-linearity: Neurons have a maximum firing rate. The response saturates after they reach this maximum. Response Input strength ThresholdSaturation Linear response region

6 Phase and Half-wave Rectification Phase non-linearity: Complex cells are insensitive to the phase (position) of a grating within the receptive field. Complex cells do not sum inputs within the receptive field. Half-wave Rectification: Cortical cells have a low spontaneous firing rate. There cannot be as large a negative response as a positive response. The bottom half of the waveform is clipped off. Response time This can be alleviated with pairs of matched cells that are 180 deg out of phase with one another. The difference in responses acts like a linear response.

7 Lateral Inhibition There is evidence that a spatial frequency channel is inhibited by other channels tuned to nearby frequencies. (Also true for orientation tuning). This is accomplished by lateral inhibitory connections within the cortex, known as lateral inhibition. This can cause interesting effects, such as repulsion of perceived orientation when 2 lines of similar orientation are shown close together. If you adapt 1 spatial frequency, there is an increased sensitivity at other nearby frequencies. Inhibitory interactions can help to make tuning curves narrower.

8 Random Dot Stereogram

9 Binocular Stereo The image in each of our two eyes is slightly different. Images in the plane of fixation fall on corresponding locations on the retina. Images in front of the plane of fixation are shifted outward on each retina. They have crossed disparity. Images behind the plane of fixation are shifted inward on the retina. They have uncrossed disparity.

10 Crossed and uncrossed disparity crossed (positive) disparity uncrossed (negative) disparity plane of fixation 1 2

11 Stereo processing To determine depth from stereo disparity: 1)Extract the "features" from the left and right images 2)For each feature in the left image, find the corresponding feature in the right image. 3)Measure the disparity between the two images of the feature. 4)Use the disparity to compute the 3D location of the feature.

12 The Correspondence problem How do you determine which features from one image match features in the other image? (This problem is known as the correspondence problem). This could be accomplished if each image has well defined shapes or colors that can be matched. Problem: Random dot stereograms. Left ImageRight ImageMaking a stereogram

13 Random Dot Stereogram

14 Problem with Random Dot Stereograms In 1980's Bela Julesz developed the random dot stereogram. The stereogram consists of 2 fields of random dots, identical except for a region in one of the images in which the dots are shifted by a small amount. When one image is viewed by the left eye and the other by the right eye, the shifted region is seen at a different depth. No cues such as color, shape, texture, shading, etc. to use for matching. How do you know which dot from left image matches which dot from the right image?

15 Using Constraints to Solve the Problem To solve the correspondence problem, we need to make some assumptions (constraints) about how the matching is accomplished. Constraints used by many computer vision stereo algorithms: 1)Uniqueness: Each point has at most one match in the other image. 2)Similarity: Each feature matches a similar feature in the other image (i.e. you cannot match a white dot with a black dot). 3)Continuity: Disparity tends to vary slowly across a surface. (Note: this is violated at depth edges). 4)Epipolar constraint: Given a point in the image of one eye, the matching point in the image for the other eye must lie along a single line.

16 The epipolar constraint Feature in left image Possible matches in right image

17 It matters where you look If the observer is fixating a point along a horizontal plane through the middle of the eyes, the possible positions of a matching point in the other image lie along a horizontal line. If the observer is looking upward or downward, the line will be tilted. Most stereo algorithms for machine vision assume the epipolar lines are horizontal. For biological systems, the stereo computation must take into account where the eyes are looking (e.g. upward or downward).