Vision Sensors for Stereo and Motion Joshua Gluckman Polytechnic University

Stereo Vision depth map

Stereo With Mirrors [ Gluckman and Nayar (CVPR 99)]

Why Use Mirrors? Identical system response –Better stereo matching –Faster stereo matching

Why Use Mirrors? Identical system response –Better stereo matching –Faster stereo matching Data acquisition –No synchronization –Data Storage

Stereo Systems Using Mirrors Teoh and Zhang `84 Goshtasby and Gruver `93 Inaba `93 Mathieu and Devernay `95 Mitsumoto `92 Zhang and Tsui `98

Geometry and Calibration

Background – Relative Orientation C C` p p` R,t – 6 parameters

Background – Epipolar Geometry C C` p p` e e`

Background – Epipolar Geometry C C` p p` e e` Epipolar geometry – 7 parameters 4 3

Background – Epipolar Geometry C C` p p` e e` Epipolar geometry – 7 parameters 4 3

One Mirror – Relative Orientation camera virtual camera mirror

One Mirror – Relative Orientation camera 3 parameters virtual camera

One Mirror – Relative Orientation camera 3 parameters virtual camera

One Mirror – Epipolar Geometry 2 parameters – location of epipole

Two Mirrors – Relative Orientation camera virtual camera D

Two Mirrors – Relative Orientation camera virtual camera 1 1  D 2 D DDDDD  

Two Mirrors – Relative Orientation camera virtual camera  5 parameters

Two Mirrors – Epipolar Geometry V V` p p` e e` 4 6 parameters 2

Two Mirrors – Epipolar Geometry epipole e` p` p epipole e image of the axis m

Two Mirrors – Epipolar Geometry epipole e` p1`p1` p1p1 epipole e image of the axis m p2p2 p3p3 p4p4 p2`p2` p3`p3` p4`p4`

(a) (b) (c) (d)

Calibration Parameters Two Cameras6 (rigid transform)7 One Mirror3 (reflection transform)2 Two Mirrors5 (screw transform)6 Three+ Mirrors6 (rigid transform)7 Relative orientationEpipolar geometry

Mirror Stereo Systems

Real Time Stereo System Calibrate Get Images Rectify Matching Depth Map

Rectification of Stereo Images Perspective transformations

Why Rectify Stereo Images? Fast stereo matching O(hw 2 s)  O(hw 2 ) Removes differences in rotation and scale

Not All Rectification Transforms Are the Same

Rectification – Previous Methods Ayache and Hansen `88 Faugeras `93 Robert et al. `93 Hartley `98 Loop and Zhang `99 Roy et al. `97 Pollefeys et al. `99 3D methods – need calibration Non-perspective transformations 2D methods – rectify from epipolar geometry

The Bad Effects of Resampling the Images Creation of new pixels causes –Blurs the texture –Additional computation Loss of pixels –Loss of information –Aliasing [Gluckman and Nayar CVPR ’01]

Measuring the Effects of Resampling determinant of the Jacobian change in local area

Measuring the Effects of Resampling determinant of the Jacobian change in local area

Measuring the Effects of Resampling determinant of the Jacobian change in local area

Change In Aspect Ratio Preserves Local Area pixels lost pixels created

Skew Preserves Local Area aliasing

Minimizing the Effects of Resampling P and P’ must be rectifying transformation No change in aspect ratio and skew change in local area

The Class of Rectifying Transformations Rotation and translation Fundamental matrix e e ee

The Class of Rectifying Transformations e e

e e e e

e e e e 6 parameters

The Class of Rectifying Transformations e e e e 2 parameters no skew maintain aspect ratio

The Class of Rectifying Transformations 2 parameters scale perspective distortion

Finding the Best Rectifying Transform Find p 1 and p 8 that minimize  change in local area

Finding the Best Rectifying Transform Find p 1 and p 8 that minimize  change in local area  is quadratic in p 1 so the optimal scale can be found as a function of p 8  is a 16 th degree rational polynomial in p 8

Finding the Best Rectifying Transform The minimum of  is between 0 and f 5  1 and  2 are symmetric convex polynomials  1 has a minimum at p 8 = 0  2 has a minimum at p 8 = f 5 1  2 

Finding the Best Rectifying Transform  1 and  2 depend on the location of epipoles epipoles at the same distance 1  2 

Finding the Best Rectifying Transform  1 and  2 depend on the location of epipoles epipoles at a distance of 3 and 10 1  2 

Rectifying While Minimizing Resampling Effects Step 1: Rotate and translate the epipolar geometry

Rectifying While Minimizing Resampling Effects Step 1: Rotate and translate the epipolar geometry Step 2: Find p 1 and p 8 that minimize 

Rectifying While Minimizing Resampling Effects Step 1: Rotate and translate the epipolar geometry Step 2: Find p 1 and p 8 that minimize  Step 3: Construct P and P’

Rectifying While Minimizing Resampling Effects Step 1: Rotate and translate the epipolar geometry Step 2: Find p 1 and p 8 that minimize  Step 3: Construct P and P’ Step 4: Rectify the images using the perspective transformations

Rectification

Rectification and Stereo Matching

Rectified Stereo Using Mirrors Not rectified Rectified [Gluckman and Nayar CVPR ’00]

When Is a Stereo System Rectified? No relative rotation between stereo cameras Direction of translation along the scan lines (x-axis) Identical intrinsic parameters (focal length)

Rectified Stereo Sensors left virtual camera right virtual camera D

left virtual camera right virtual camera Rectified Stereo Sensors

What Constraints Must Be Satisfied?

How Many Reflections? Even number of reflections Odd number of reflections

Example: Four mirrors Won’t Work

What Constraints Must Be Satisfied?

Single Mirror Rectified Stereo

camera virtual camera b

Three Mirror Rectified Stereo

n 1, n 2, n 3 and x-axis must be coplanar One constraint on the angles One constraint on the distances 4 constraints

A Three Mirror Solution 9 d.o.f. – 4 constraints = 5 parameter family of solutions

Sensor Size 9 d.o.f. – 4 constraints = 5 parameter family of solutions

Optimized Solutions

Rectified Stereo Sensors Mirrors Mirror

Rectified Images and Depth Maps

Misplacement of the Camera Mirrors Mirror

Misplacement of the Camera Mirrors Mirror Invariant to misplacement of camera center

Misplacement of the Camera Mirrors Mirror Insensitive to tilt of optical axis

Misplacement of the Camera Mirrors Mirror Dependent on pan of optical axis

Split Shot Stereo Camera Nikon Coolpix camera mirror attachment

Image Sensors for Motion Computation

Camera Motion motion rotation, translation, depth

[ Anadan and Avidan (ECCV 00)] [e,e’]             y x y x

[Gluckman and Nayar ICCV ’98][Aloimonos et al]

Future Work

Split Shot Stereo Camera Nikon Coolpix camera mirror attachment

Split Shot Stereo Camera

The Class of Rectifying Transformations e e` p 1 changes the distance and p 8 changes the tilt of the rectifying plane Rectification projects the images onto a plane parallel to the camera centers

SensingPre-processingComputation