Computer Vision Stereo Vision

Bahadir K. Gunturk2 Pinhole Camera

Bahadir K. Gunturk3 Review: Perspective Projection

Bahadir K. Gunturk4 Stereo scene point optical center image plane p p’ p

Bahadir K. Gunturk5 Stereo Constraints X1X1 Y1Y1 Z1Z1 O1O1 Image plane Focal plane M p p’ Y2Y2 X2X2 Z2Z2 O2O2 Epipolar Line Epipole

Bahadir K. Gunturk6 A Simple Stereo System Z w =0 LEFT CAMERA Left image: reference Right image: target RIGHT CAMERA Elevation Z w disparity Depth Z baseline

Bahadir K. Gunturk7 Stereo View Left View Right View Disparity

Bahadir K. Gunturk8 Stereo Disparity The separation between two matching objects is called the stereo disparity.

Bahadir K. Gunturk9 Parallel Cameras OlOlOlOl OrOrOrOr P plplplpl prprprpr T Z xlxlxlxl xrxrxrxr f T is the stereo baseline Disparity:

Bahadir K. Gunturk10 Correlation Approach For Each point (x l, y l ) in the left image, define a window centered at the point (x l, y l ) LEFT IMAGE

Bahadir K. Gunturk11 Correlation Approach … search its corresponding point within a search region in the right image (x l, y l ) RIGHT IMAGE

Bahadir K. Gunturk12 Correlation Approach … the disparity (dx, dy) is the displacement when the correlation is maximum (x l, y l )dx(x r, y r ) RIGHT IMAGE

Bahadir K. Gunturk13 Maximize Cross correlation Minimize Sum of Squared Differences Comparing Windows =?f g

Bahadir K. Gunturk14 Feature-based correspondence Features most commonly used:  Corners Similarity measured in terms of:  surrounding gray values (SSD, Cross-correlation)  location  Edges, Lines Similarity measured in terms of:  orientation  contrast  coordinates of edge or line’s midpoint  length of line

Bahadir K. Gunturk15 Feature-based Approach For each feature in the left image… LEFT IMAGE corner line structure

Bahadir K. Gunturk16 Feature-based Approach Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum RIGHT IMAGE corner line structure

Bahadir K. Gunturk17 Correspondence Difficulties Why is the correspondence problem difficult?  Some points in each image will have no corresponding points in the other image. (1) the cameras might have different fields of view. (2) due to occlusion. A stereo system must be able to determine the image parts that should not be matched.

Bahadir K. Gunturk18 Structured Light Structured lighting  Feature-based methods are not applicable when the objects have smooth surfaces (i.e., sparse disparity maps make surface reconstruction difficult).  Patterns of light are projected onto the surface of objects, creating interesting points even in regions which would be otherwise smooth.  Finding and matching such points is simplified by knowing the geometry of the projected patterns.

Bahadir K. Gunturk19 Stereo results Ground truthScene  Data from University of Tsukuba (Seitz)

Bahadir K. Gunturk20 Results with window correlation Estimated depth of fieldGround truth (Seitz)

Bahadir K. Gunturk21 Results with better method A state of the art method Boykov et al., Fast Approximate Energy Minimization via Graph Cuts,Fast Approximate Energy Minimization via Graph Cuts International Conference on Computer Vision, September Ground truth (Seitz)

Bahadir K. Gunturk22 Other constraints It is possible to put some constraints. For example: smoothness. (Disparity usually doesn’t change too quickly.)

Bahadir K. Gunturk23 Parameters of a Stereo System Intrinsic Parameters  Characterize the transformation from camera to pixel coordinate systems of each camera  Focal length, image center, aspect ratio Extrinsic parameters  Describe the relative position and orientation of the two cameras  Rotation matrix R and translation vector T p l p r P OlOl OrOr XlXl XrXr PlPl PrPr flfl frfr ZlZl YlYl ZrZr YrYr R, T

Bahadir K. Gunturk24 Applications courtesy of Sportvision First-down line

Bahadir K. Gunturk25 Applications Virtual advertising courtesy of Princeton Video Image