CS332 Visual Processing Department of Computer Science Wellesley College Binocular Stereo Vision Region-based stereo matching algorithms Properties of human stereo processing

Solving the stereo correspondence problem 1-2

1-3 (1) sum of absolute differences Σ | p left – p right | (2) normalized correlation (p left – p left ) (p right – p right ) patch σ p left σ p right Σ patch Measuring goodness of match between patches (1/n) Optional: divide by n = number of pixels in patch

Region-based stereo matching algorithm for each row r for each column c let p left be a square patch centered on (r,c) in the left image initialize best match score m best to ∞ initialize best disparity d best for each disparity d from – d range to + d range let p right be a square patch centered on (r,c+d) in the right image compute the match score m between p left and p right (sum of absolute differences) if (m < m best ), assign m best = m and d best = d record d best in the disparity map at (r,c) 1-4 (normalized correlation) How are the assumptions used??

1-5 leftright The real world works against us sometimes…

Example: Region-based stereo matching, using filtered images and sum-of-absolute differences 1-6 (from Carolyn Kim, 2013)

1-7 Properties of human stereo processing Use features for stereo matching whose position and disparity can be measured very precisely Stereoacuity is only a few seconds of visual angle difference in depth  0.01 cm at a viewing distance of 30 cm

1-8 Properties of human stereo processing Matching features must appear similar in the left and right images For example, we can’t fuse a left stereo image with a negative of the right image…

1-9 Properties of human stereo processing Only “fuse” objects within a limited range of depth around the fixation distance Vergence eye movements are needed to fuse objects over larger range of depths

1-10 Properties of human stereo vision We can only tolerate small amounts of vertical disparity at a single eye position Vertical eye movements are needed to handle large vertical disparities

1-11 Properties of human stereo processing In the early stages of visual processing, the image is analyzed at multiple spatial scales… Stereo information at multiple scales can be processed independently

1-12 Neural mechanisms for stereo processing G. Poggio & colleagues: complex cells in area V1 of primate visual cortex are selective for stereo disparity neurons that are selective for a larger disparity range have larger receptive fields zero disparity: at fixation distance near: in front of point of fixation far: behind point of fixation

1-13 In summary, some key points… Image features used for matching: simple, precise locations, multiple scales, similar between left/right images At single fixation position, match features over a limited range of horizontal & vertical disparity Eye movements used to match features over larger range of disparity Neural mechanisms selective for particular ranges of stereo disparity

1-14 Matching features for the MPG stereo algorithm zero-crossings of convolutions with  2 G operators of different size L M S rough disparities over large range accurate disparities over small range

1-15 large w left large w right small w left small w right correct match outside search range at small scale

1-16 large w left right small w left right correct match now inside search range at small scale vergence eye movements!

1-17 Stereo images (Tsukuba, CMU)

1-18 Zero-crossings for stereo matching - + ……

1-19 Simplified MPG algorithm, Part 1 To determine initial correspondence: (1) Find zero-crossings using a  2 G operator with central positive width w (2) For each horizontal slice: (2.1) Find the nearest neighbors in the right image for each zero-crossing fragment in the left image (2.2) Fine the nearest neighbors in the left image for each zero-crossing fragment in the right image (2.3) For each pair of zero-crossing fragments that are closest neighbors of one another, let the right fragment be separated by δ initial from the left. Determine whether δ initial is within the matching tolerance, m. If so, consider the zero-crossing fragments matched with disparity δ initial m = w/2

1-20 Simplified MPG algorithm, Part 2 To determine final correspondence: (1) Find zero-crossings using a  2 G operator with reduced width w/2 (2) For each horizontal slice: (2.1) For each zero-crossing in the left image: (2.1.1) Determine the nearest zero-crossing fragment in the left image that matched when the  2 G operator width was w (2.1.2) Offset the zero-crossing fragment by a distance δ initial, the disparity of the nearest matching zero-crossing fragment found at the lower resolution with operator width w (2.2) Find the nearest neighbors in the right image for each zero- crossing fragment in the left image (2.3) Fine the nearest neighbors in the left image for each zero- crossing fragment in the right image (2.4) For each pair of zero-crossing fragments that are closest neighbors of one another, let the right fragment be separated by δ new from the left. Determine whether δ new is within the reduced matching tolerance, m/2. If so, consider the zero-crossing fragments matched with disparity δ final = δ new + δ initial

1-21 Coarse-scale zero-crossings: Use coarse-scale disparities to guide fine-scale matching: Ignore coarse-scale disparities: w = 8 m = 4 w = 4 m = 2 w = 4 m = 2