Computer Vision : CISC 4/689 Adaptation from: Prof. James M. Rehg, G.Tech

Computer Vision : CISC 4/689 Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923

Computer Vision : CISC 4/689 Teesta suspension bridge-Darjeeling, India

Computer Vision : CISC 4/689 Mark Twain at Pool Table", no date, UCR Museum of Photography

Computer Vision : CISC 4/689 Woman getting eye exam during immigration procedure at Ellis Island, c , UCR Museum of Phography

Computer Vision : CISC 4/689 Stereo results Ground truthScene –Data from University of Tsukuba (Seitz)

Computer Vision : CISC 4/689 Results with window correlation Window-based matching (best window size) Ground truth (Seitz)

Computer Vision : CISC 4/689 Results with better method State of the art method Boykov et al., Fast Approximate Energy Minimization via Graph Cuts,Fast Approximate Energy Minimization via Graph Cuts International Conference on Computer Vision, September Ground truth (Seitz)

Computer Vision : CISC 4/689 Stereo Constraint Color constancy –The color of any world points remains constant from image to image –This assumption is true under Lambertian Model –In practice, given photometric camera calibration and typical scenes, color constancy holds well enough for most stereo algorithms.

Computer Vision : CISC 4/689 Stereo Constraint Epipolar geometry –The epipolar geometry is the fundamental constraint in stereo. –Rectification aligns epipolar lines with scanlines Epipolar plane Epipolar line for p Epipolar line for p’

Computer Vision : CISC 4/689 Stereo Constraint Uniqueness and Continuity –Proposed by Marr&Poggio. –Each item from each image may be assigned at most one disparity value, ” and the “ disparity ” varies smoothly almost everywhere.

Computer Vision : CISC 4/689 Problems with Window-based matching Disparity within the window must be constant. Bias the results towards frontal-parallel surfaces. Blur across depth discontinuities. Perform poorly in textureless regions. Erroneous results in occluded regions

Computer Vision : CISC 4/689 Cooperative Stereo Algorithm Based on two basic assumption by Marr and Poggio: –Uniqueness: at most a single unique match exists for each pixel. –Continuous: disparity values are generally continuous, i.e., smooth within a local neighborhood.

Computer Vision : CISC 4/689 Disparity Space Image (DSI) The 3D disparity space has dimensions row r column c and disparity d. Each element (r, c, d) of the disparity space projects to the pixel (r, c) in the left image and to the (r, c + d) in the right image DSI represents the confidence or likelihood of a particular match.

Computer Vision : CISC 4/689 Illustration of DSI (r, c) slices for different d (c, d) slice for r = 151

Computer Vision : CISC 4/689 Definition Match value assigned to element (r, c, d) at iteration n Initial values computed from SSD or NCC Inhibition area for element (r, c, d) Local support area for element (r, c, d)

Computer Vision : CISC 4/689 Illustration of Inhibitory and Support Regions If d is disparity for (r,c), d -1 is for (r,c-1) And d+1 for (r,c+1), etc. Vertical inhibition is self explanatory. Diagonal basically means, r,c+1in left image cannot map to r,c+d in right image, i.e, it cannot have disparity of d-1. Similarly, r,c-1 in left image cannot map to r,c+d so it cannot have disparity of d+1. This satisfies uniqueness constraint.

Computer Vision : CISC 4/689 Iterative Updating DSI Questions to ponder: i)What is typical alpha? ii)Do we include inhibition region in numerator of Rn?

Computer Vision : CISC 4/689 Explicit Detection of Occlusion Identify occlusions by examining the magnitude of the converged values in conjunction with the uniqueness constrain

Computer Vision : CISC 4/689 Summary of Cooperative Stereo Prepare a 3D array, (r, c, d): (r, c) for each pixel in the reference image and d for the range of disparity. Set initial match values using a function of image intensities, such as normalized correlation or SSD. Iteratively update match values using (4) until the match values converge. For each pixel (r, c), find the element (r, c, d) with the maximum match value. If the maximum match value is higher than a threshold, output the disparity d, otherwise, declare a occlusion.

Computer Vision : CISC 4/689 Ordering Constraint If an object a is left on an object b in the left image then object a will also appear to the left of object b in the right image Ordering constraint……and its failure

Computer Vision : CISC 4/689 Stereo Correspondences …… Left scanlineRight scanline Match intensities sequentially between two scanlines

Computer Vision : CISC 4/689 Stereo Correspondences …… Left scanlineRight scanline Match Left occlusionRight occlusion

Computer Vision : CISC 4/689 Search Over Correspondences Three cases: –Sequential – cost of match –Left occluded – cost of no match –Right occluded – cost of no match Left scanline Right scanline Left Occluded Pixels Right occluded Pixels

Computer Vision : CISC 4/689 Standard 3-move Dynamic Programming for Stereo Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint Left Occluded Pixels Left scanline Right occluded Pixels Right scanline Start End

Computer Vision : CISC 4/689 Dynamic Programming Efficient algorithm for solving sequential decision (optimal path) problems … How many paths through this trellis?

Computer Vision : CISC 4/689 Dynamic Programming Suppose cost can be decomposed into stages: States:

Computer Vision : CISC 4/689 Dynamic Programming Principle of Optimality for an n-stage assignment problem

Computer Vision : CISC 4/689 Dynamic Programming

Computer Vision : CISC 4/689 Stereo Matching with Dynamic Programming Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal

Computer Vision : CISC 4/689 Stereo Matching with Dynamic Programming Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal

Computer Vision : CISC 4/689 Stereo Matching with Dynamic Programming Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal

Computer Vision : CISC 4/689 Stereo Matching with Dynamic Programming Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal

Computer Vision : CISC 4/689 Stereo Matching with Dynamic Programming Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal

Computer Vision : CISC 4/689 Stereo Matching with Dynamic Programming Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal

Computer Vision : CISC 4/689 Stereo Matching with Dynamic Programming Scan across grid computing optimal cost for each node given its upper-left neighbors. Backtrack from the terminal to get the optimal path. For each (i,j), look for what is Optimal to get to: would it be From (i-1,j-1), or (i,j-1), or (i-1,j-1)? Assign large values for left occlusion and right occlusion. Occluded Pixels Left scanline Dis-occluded Pixels Right scanline Terminal

Computer Vision : CISC 4/689 Stereo Matching with Dynamic Programming Pseudo-code describing how to calculate the optimal match

Computer Vision : CISC 4/689 Stereo Matching with Dynamic Programming Pseudo-code describing how to reconstruct the optimal path

Computer Vision : CISC 4/689 Results Local errors may be propagated along a scan-line and no inter scan-line consistency is enforced.

Computer Vision : CISC 4/689 Assumption Behind Segmentation-based Stereo Depth discontinuity tend to correlate well with color edges Disparity variation within a segment is small Approximating the scene with piece-wise planar surfaces

Computer Vision : CISC 4/689 Segmentation-based stereo Plane equation is fitted in each segment based on initial disparity estimation obtained SSD or Correlation Global matching criteria: if a depth map is good, warping the reference image to the other view according to this depth will render an image that matches the real view Optimization by iterative neighborhood depth hypothesizing

Computer Vision : CISC 4/689 Result

Computer Vision : CISC 4/689 Another Result