1. Binocular Stereo #1

2. Topics 1. Principle 2. binocular stereo basic equation 3. epipolar line 4. features and strategies for matching

3. Binocular stereo single image is ambiguous A another image taken from a different direction gives the unique 3D point a’ a”

4. Epipolar line Epipolar plane Epipolar line constraints Corresponding points lie on the Epipolar lines Epipolar line constratints Base line One image point Possible line of sight

5. Epipolar geometry (multiple points) C1C1 C2C2 e1e1 e2e2 Epipoles: intersections of baseline with image planes projection of the optical center in another image the vanishing points of camera motion direction

6. Examples of epipolar geometry

9. Characteristics of epipolar line rectification

10. Basic binocular stereo equation A physical point focal length right image point z left image point base line length right image plane left image plane World coordinate system left image center right image center

11. Camera Model Pinhole camera

12. Camera Model geometry (X, Y, Z) Image plane X Y -Z x y (x, y) f : focal length Perspective projection View point (Optical center) (sX, sY, sZ)

13. Basic binocular stereo equation z=-2df/(x”-x’) x”-x’: disparity 2d : base line length x” x’ -z f d d z d + xd - x

14. Classic algorithms for binocular Stereo Marr-Poggio Marr-Poggio-Grimson Nishihara-Poggio Lucas-Kanade Ohta-Kanade Matthie-Kanade Okutomi-Kanade Baker Hannah Moravec Barnard-Thompson MIT group CMU group Stanford group

15. Features for matching a. brightness b. edges c. edge intervals d. interest points 10 11 12 11 15 16

16. a. relaxation b. coarse to fine c. dynamic programming local optimam Strategies for matching global optimam 10 10 10 10 5 10 10 10 10 10 5 10 10 10 10

17. Main purpose of development simulate human stereo map making navigation Marr-Poggio Marr-Poggio-Grimson Nishihara-Poggio Lucas-Kanade Ohta-Kanade Matthie-Kanade Okutomi-Kanade Baker Hannah Moravec Barnard-Thompson

18. Features for matching points(random dots) edges intervals brightness(gradient) intervals brightness edges interest points Marr-Poggio Marr-Poggio-Grimson Nishihara-Poggio Lucas-Kanade Ohta-Kanade Matthie-Kanade Okutomi-Kanade Baker Hannah Moravec Barnard-Thompson

19. Strategies for matching relaxation coarse to fine relaxation dynamic programming Relaxation (Kalman filter) relaxation dynamic programming coarse to fine relaxation Marr-Poggio Marr-Poggio-Grimson Nishihara-Poggio Lucas-Kanade Ohta-Kanade Matthie-Kanade Okutomi-Kanade Baker Hannah Moravec Barnard-Thompson

20. Summary 1.binocular stereo takes two images of 3D point from two different positions and determines its 3D coordinate system. 2. Epipolar line 2D matching ↓ 1D matching 3. Features for matching ---brightness,edges,edge interval,interest point 4. Strategies for matching ---relaxation,coarse to fine,dynamic programming 5. Read B&B pp.88-93 Horn pp.299-303

21. Binocular Stereo #2

22. Topics case study area-based stereo Marr-poggio stereo simulate human visual system Ohta-Kanade stereo aerial image analysis Moravec stereo navigation

23. Classification of stereo method 1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force (not a strategy ???) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness

24. Area-based stereo 1. method b c bcbc 2. problem a. trade-off of window size and resolution b. dull peak b c

25. 1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force (not a strategy ???) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness Area-based stereo

26. Marr-Poggio Stereo(`76) Simulating human visual system (random dot stereo gram) Marr,Poggio “Coopertive computation of stereo disparity” Science 194,283-287

27. Input : random dot stereo left image random dot shift the catch pat right image we can see the height different between the central and peripheral area

28. Constraints –Epipolar line constraint –Uniqueness constraint »each point in a image has only one depth value O.K. No. –Continuity constraint »each point is almost sure to have a depth value near the values of neighbors O.K. No.

29. Uniqueness constraint prohibits two or more matching points on one horizontal or vertical lines continuity constraint attracts more matching on a diagonal line ABCABC D E F A B C ABCABC (E-A) (E-B) (E-C) prohibit attract (D-A) (E-B) (F-C) Same depth

30. nn+1 relaxation 10 10 10 10 5 10 10 10 10 10 5 10 10 10 10

32. 1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force (not a strategy ???) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness simulate the human visual system (MIT) Marr-Poggio Stereo(`76)

33. Ohta-Kanade Stereo(`85) Map making Ohta,Kanade “Stereo by intra- and inter-scanline search using dynamic programming”,IEEE Trans.,Vol. PAMI-7,No.2,pp.139-14

34. now matching become 1D to 1D yet, N line * M L * M R (512 * 100 * 100 * 10 m sec = 15 hours) L1 L2 L3 L4 L5 L6 R1 R2 R3 R4 R5 R6 L R disparity

35. Path Search u Matching problem can be considered as a path search problem u define a cost at each candidate of path segment based some ad-hoc function 10 100 100

36. Dynamic programming We can formalize the path finding problem as the following iterative formula optimum cost to K cost between M and K 3 0 21 Optimum costs are known

37. stereo pair edges

38. pathdisparity depth

39. stereo pair edges depth

40. 1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force (not a strategy ???) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness aerial image analysis (CMU) Ohta-Kanade Stereo(`85) Brightness of interval

41. Moravec Stereo(`79) navigation Moravec “Visual mapping by a robot rover” Proc 6th IJCAI,pp.598-600 (1979)

42. Moravec’s cart Slide stereo Motion stereo

43. Slider stereo (9 eyes stereo) u 9 C 2 = 36 stereo pairs!!! u each stereo has an uncertainty measure u uncertainty = 1 / base-line u each stereo has a confidence measure long base line large uncertainty

46. Coarse to fine expand matching

47. σ estimated distance σ:uncertainty measure area:confidence measure 9 C 2 = 36 curves Interest point

48. 1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force (not a strategy ???) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness navigation (Stanford) Moravec Stereo(`81) interest point

49. Summary 1. Two images from two different positions give depth information 2. Epipolar line and plane 3. Basic equation Z=-2df/(x”-x’) x”-x’: disparity 2d : base line length 4. case study area-based stereo Marr-poggio stereo simulate human visual system Ohta-Kanade stereo aerial image analysis Moravec stereo navigation 5. Read Horn pp.299-303

50. F matrix

51. Camera Model Pinhole camera

52. Camera Model geometry (X, Y, Z) Image plane X Y -Z x y (x, y) f : focal length Perspective projection View point (Optical center) (sX, sY, sZ)

53. Camera Model Perspective projection formularization Perspective projection (Non-linear) Affine projection (Linear) Projection matrix

54. Affine Camera Models General formularization OrthographicPerspective Affine camera

55. Affine Cameras perspectiveorthographic Focal length Distance from camera

56. Intrinsic parameters Image plane : an ideal image CCD : an actual picture Not equal ! CCD elements

57. Intrinsic parameters y An ideal image on the Image plane x u v θ An actual picture u0u0 v0v0 (x, y) (u, v)

58. Intrinsic parameters e.g. perspective projection Intrinsic matrix Projection matrix (normalized)

59. Extrinsic parameters Y X Z

60. Y X Z

62. R : rotation matrixt : translation vector

63. Summary (intrinsic & extrinsic parameters) Y X Z (X,Y,Z) World coordinate R, t (u, v) picture Camera coordinate World coordinate

64. Summary (intrinsic & extrinsic parameters) Y X Z (X,Y,Z) World coordinate R, t (u, v) picture 3 × 4 matrix

65. Epipolar geometry C1C1 C2C2 R Essential matrix : E

66. Essential & Fundamental matrix Image planes (ideal) Pictures (actual) Fundamental matrix : F Image 1 Image 2

67. F matrix (u, v, 1)(u’, v’, 1) F & (u, v) known Calculate the epipolar line picture 1picture 2

68. Computing F matrix (Linear solution)

69. Corner detector Extract interest points in each images x y Harris corner detector

70. Matching or

71. Computing F matrix (Linear solution) Suppose we found 8 pairs of corresponding points ·····

72. Computing F matrix (Singularity constraint) Epipolar pencil by linear solution (due to noise and error)

73. Computing F matrix (Singularity constraint) Singular value decomposition (SVD) Without noise, σ 3 must be 0 modification

74. Computing F matrix (Singularity constraint)

75. Summary u Pinhole camera and Affine camera u Intrinsic and extrinsic camera parameter u Epipolar geometry u Fundamental matrix