Binocular Stereo Vision Stereo viewing geometry and the stereo correspondence problem

Constraints on stereo correspondence Uniqueness each feature in the left image matches with only one feature in the right (and vice versa…) Similarity matching features appear “similar” in the two images Continuity nearby image features have similar simple version: matching features have similar disparities Epipolar constraint vertical positions, but…

Marr-Poggio cooperative stereo algorithm  matches corresponding black & white dots in the left & right images  stereo disparities computed by a parallel network of simple computing elements  iterative: state of network changes with each iteration, to resolve matching ambiguity disparity map emerges over time

Input x 1 1 y left right (epipolar constraint)

3D cooperative network: initial state C(x,y,d) = 1 if L(x,y) = R(x+d,y) C(x,y,d) = 0 otherwise (similarity constraint) 3D cooperative network: initial state x C(x,y,d) +3 +2 +1 -1 -2 -3 y d +3 +2 +1 -1 -2 -3 1 1 1 1 1 left right

Updating the network For each iteration, the state of every element in the 3D network is updated: (1) S(x,y,d) = Ct(x,y,d) + E – ε I (2) Ct+1(x,y,d) = 1 if S(x,y,d) ≥  = 0 if S(x,y,d) <  “inhibition” evidence for other disparities at location (x,y) support for disparity d at location (x,y) “excitation” neighborhood support for disparity d at location (x,y) current state of network (0 or 1) threshold new, updated state at time t+1

3D cooperative network x y d C(x,y,d) = 0 or 1 left right 1 1 1 1 1 +3 +2 +1 -1 -2 -3 y d +3 +2 +1 -1 -2 -3 1 1 1 C(x,y,d) = 0 or 1 1 1 left right

Enforcing the continuity constraint x 1 disparity = +2 y E = sum of 1’s in a neighborhood around (x,y) E = 10 S(x,y,d) = Ct(x,y,d) + E – ε I

3D cooperative network C(x,y,d) x y d 1 1 1 +3 +2 +1 -1 -2 -3 +3 +2 +1 -1 -2 -3 y d +3 +2 +1 -1 -2 -3 1 1 1

Enforcing the uniqueness constraint x +3 +2 +1 -1 -2 -3 1 I = total number of alternative disparities d I = 3 S(x,y,d) = Ct(x,y,d) + E – ε I

x x 1 1 y y x right left 1 initial state: C0(x,y,d) d = -1 y E =

row 3: Y = 3 I = left right x d initial state: C0(x,y,d) 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 1 left right x 1 2 3 4 5 6 7 +2 +1 -1 -2 1 Y = 3 d initial state: C0(x,y,d) I = S(x,y,d) = Ct(x,y,d) + E – ε I Ct+1(x,y,d) = (ε = 2) ( = 3)