1. Vision Sensors for Stereo and Motion Joshua Gluckman Polytechnic University

2. Stereo Vision depth map

3. Stereo With Mirrors [ Gluckman and Nayar (CVPR 99)]

4. Why Use Mirrors? Identical system response –Better stereo matching –Faster stereo matching

5. Why Use Mirrors? Identical system response –Better stereo matching –Faster stereo matching Data acquisition –No synchronization –Data Storage

6. Stereo Systems Using Mirrors Teoh and Zhang `84 Goshtasby and Gruver `93 Inaba `93 Mathieu and Devernay `95 Mitsumoto `92 Zhang and Tsui `98

7. Geometry and Calibration

8. Background – Relative Orientation C C` p p` R,t – 6 parameters

9. Background – Epipolar Geometry C C` p p` e e`

10. Background – Epipolar Geometry C C` p p` e e` Epipolar geometry – 7 parameters 4 3

11. Background – Epipolar Geometry C C` p p` e e` Epipolar geometry – 7 parameters 4 3

12. One Mirror – Relative Orientation camera virtual camera mirror

13. One Mirror – Relative Orientation camera 3 parameters virtual camera

14. One Mirror – Relative Orientation camera 3 parameters virtual camera

15. One Mirror – Epipolar Geometry 2 parameters – location of epipole

17. Two Mirrors – Relative Orientation camera virtual camera D

18. Two Mirrors – Relative Orientation camera virtual camera 1 1  D 2 D 212 1 1 DDDDD  

19. Two Mirrors – Relative Orientation camera virtual camera  5 parameters

20. Two Mirrors – Epipolar Geometry V V` p p` e e` 4 6 parameters 2

21. Two Mirrors – Epipolar Geometry epipole e` p` p epipole e image of the axis m

22. Two Mirrors – Epipolar Geometry epipole e` p1`p1` p1p1 epipole e image of the axis m p2p2 p3p3 p4p4 p2`p2` p3`p3` p4`p4`

23. (a) (b) (c) (d)

24. Calibration Parameters Two Cameras6 (rigid transform)7 One Mirror3 (reflection transform)2 Two Mirrors5 (screw transform)6 Three+ Mirrors6 (rigid transform)7 Relative orientationEpipolar geometry

25. Mirror Stereo Systems

27. Real Time Stereo System Calibrate Get Images Rectify Matching Depth Map

28. Rectification of Stereo Images Perspective transformations

29. Why Rectify Stereo Images? Fast stereo matching O(hw 2 s)  O(hw 2 ) Removes differences in rotation and scale

30. Not All Rectification Transforms Are the Same

31. Rectification – Previous Methods Ayache and Hansen `88 Faugeras `93 Robert et al. `93 Hartley `98 Loop and Zhang `99 Roy et al. `97 Pollefeys et al. `99 3D methods – need calibration Non-perspective transformations 2D methods – rectify from epipolar geometry

32. The Bad Effects of Resampling the Images Creation of new pixels causes –Blurs the texture –Additional computation Loss of pixels –Loss of information –Aliasing [Gluckman and Nayar CVPR ’01]

33. Measuring the Effects of Resampling determinant of the Jacobian change in local area

34. Measuring the Effects of Resampling determinant of the Jacobian change in local area

35. Measuring the Effects of Resampling determinant of the Jacobian change in local area

36. Change In Aspect Ratio Preserves Local Area pixels lost pixels created

37. Skew Preserves Local Area aliasing

38. Minimizing the Effects of Resampling P and P’ must be rectifying transformation No change in aspect ratio and skew change in local area

39. The Class of Rectifying Transformations Rotation and translation Fundamental matrix e e ee

40. The Class of Rectifying Transformations e e

41. e e e e

42. e e e e 6 parameters

43. The Class of Rectifying Transformations e e e e 2 parameters no skew maintain aspect ratio

44. The Class of Rectifying Transformations 2 parameters scale perspective distortion

45. Finding the Best Rectifying Transform Find p 1 and p 8 that minimize  change in local area

46. Finding the Best Rectifying Transform Find p 1 and p 8 that minimize  change in local area  is quadratic in p 1 so the optimal scale can be found as a function of p 8  is a 16 th degree rational polynomial in p 8

47. Finding the Best Rectifying Transform The minimum of  is between 0 and f 5  1 and  2 are symmetric convex polynomials  1 has a minimum at p 8 = 0  2 has a minimum at p 8 = f 5 1  2 

48. Finding the Best Rectifying Transform  1 and  2 depend on the location of epipoles epipoles at the same distance 1  2 

49. Finding the Best Rectifying Transform  1 and  2 depend on the location of epipoles epipoles at a distance of 3 and 10 1  2 

50. Rectifying While Minimizing Resampling Effects Step 1: Rotate and translate the epipolar geometry

51. Rectifying While Minimizing Resampling Effects Step 1: Rotate and translate the epipolar geometry Step 2: Find p 1 and p 8 that minimize 

52. Rectifying While Minimizing Resampling Effects Step 1: Rotate and translate the epipolar geometry Step 2: Find p 1 and p 8 that minimize  Step 3: Construct P and P’

53. Rectifying While Minimizing Resampling Effects Step 1: Rotate and translate the epipolar geometry Step 2: Find p 1 and p 8 that minimize  Step 3: Construct P and P’ Step 4: Rectify the images using the perspective transformations

55. Rectification

56. Rectification and Stereo Matching

57. Rectified Stereo Using Mirrors Not rectified Rectified [Gluckman and Nayar CVPR ’00]

58. When Is a Stereo System Rectified? No relative rotation between stereo cameras Direction of translation along the scan lines (x-axis) Identical intrinsic parameters (focal length)

59. Rectified Stereo Sensors left virtual camera 1 2 3 4 5 right virtual camera D

60. left virtual camera 1 2 3 4 5 right virtual camera Rectified Stereo Sensors

61. What Constraints Must Be Satisfied?

62. How Many Reflections? Even number of reflections Odd number of reflections

63. Example: Four mirrors Won’t Work

64. What Constraints Must Be Satisfied?

65. Single Mirror Rectified Stereo

66. camera virtual camera b

67. Three Mirror Rectified Stereo

68. n 1, n 2, n 3 and x-axis must be coplanar One constraint on the angles One constraint on the distances 4 constraints

69. A Three Mirror Solution 9 d.o.f. – 4 constraints = 5 parameter family of solutions

70. Sensor Size 9 d.o.f. – 4 constraints = 5 parameter family of solutions

71. Optimized Solutions

72. Rectified Stereo Sensors Mirrors Mirror

73. Rectified Images and Depth Maps

74. Misplacement of the Camera Mirrors Mirror

75. Misplacement of the Camera Mirrors Mirror Invariant to misplacement of camera center

76. Misplacement of the Camera Mirrors Mirror Insensitive to tilt of optical axis

77. Misplacement of the Camera Mirrors Mirror Dependent on pan of optical axis

78. Split Shot Stereo Camera Nikon Coolpix camera mirror attachment

79. Image Sensors for Motion Computation

80. Camera Motion motion rotation, translation, depth

82. [ Anadan and Avidan (ECCV 00)] [e,e’]             y x y x

83. [Gluckman and Nayar ICCV ’98][Aloimonos et al]

90. Future Work

91. Split Shot Stereo Camera Nikon Coolpix camera mirror attachment

92. Split Shot Stereo Camera

102. The Class of Rectifying Transformations e e` p 1 changes the distance and p 8 changes the tilt of the rectifying plane Rectification projects the images onto a plane parallel to the camera centers

103. SensingPre-processingComputation