Binocular Stereo #1

Topics 1. Principle 2. binocular stereo basic equation 3. epipolar line 4. features and strategies for matching

Binocular stereo single image is ambiguous A another image taken from a different direction gives the unique 3D point a’ a”

Epipolar line Epipolar plane Epipolar line constraints Corresponding points lie on the Epipolar lines Epipolar line constratints Base line One image point Possible line of sight

Epipolar geometry (multiple points) C1C1 C2C2 e1e1 e2e2 Epipoles: intersections of baseline with image planes projection of the optical center in another image the vanishing points of camera motion direction

Examples of epipolar geometry

Characteristics of epipolar line rectification

Basic binocular stereo equation A physical point focal length right image point z left image point base line length right image plane left image plane World coordinate system left image center right image center

Camera Model Pinhole camera

Camera Model geometry (X, Y, Z) Image plane X Y -Z x y (x, y) f : focal length Perspective projection View point (Optical center) (sX, sY, sZ)

Basic binocular stereo equation z=-2df/(x”-x’) x”-x’: disparity 2d : base line length x” x’ -z f d d z d + xd - x

Classic algorithms for binocular Stereo Marr-Poggio Marr-Poggio-Grimson Nishihara-Poggio Lucas-Kanade Ohta-Kanade Matthie-Kanade Okutomi-Kanade Baker Hannah Moravec Barnard-Thompson MIT group CMU group Stanford group

Features for matching a. brightness b. edges c. edge intervals d. interest points

a. relaxation b. coarse to fine c. dynamic programming local optimam Strategies for matching global optimam

Main purpose of development simulate human stereo map making navigation Marr-Poggio Marr-Poggio-Grimson Nishihara-Poggio Lucas-Kanade Ohta-Kanade Matthie-Kanade Okutomi-Kanade Baker Hannah Moravec Barnard-Thompson

Features for matching points(random dots) edges intervals brightness(gradient) intervals brightness edges interest points Marr-Poggio Marr-Poggio-Grimson Nishihara-Poggio Lucas-Kanade Ohta-Kanade Matthie-Kanade Okutomi-Kanade Baker Hannah Moravec Barnard-Thompson

Strategies for matching relaxation coarse to fine relaxation dynamic programming Relaxation (Kalman filter) relaxation dynamic programming coarse to fine relaxation Marr-Poggio Marr-Poggio-Grimson Nishihara-Poggio Lucas-Kanade Ohta-Kanade Matthie-Kanade Okutomi-Kanade Baker Hannah Moravec Barnard-Thompson

Summary 1.binocular stereo takes two images of 3D point from two different positions and determines its 3D coordinate system. 2. Epipolar line 2D matching ↓ 1D matching 3. Features for matching ---brightness,edges,edge interval,interest point 4. Strategies for matching ---relaxation,coarse to fine,dynamic programming 5. Read B&B pp Horn pp

Binocular Stereo #2

Topics case study area-based stereo Marr-poggio stereo simulate human visual system Ohta-Kanade stereo aerial image analysis Moravec stereo navigation

Classification of stereo method 1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force (not a strategy ???) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness

Area-based stereo 1. method b c bcbc 2. problem a. trade-off of window size and resolution b. dull peak b c

1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force (not a strategy ???) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness Area-based stereo

Marr-Poggio Stereo(`76) Simulating human visual system (random dot stereo gram) Marr,Poggio “Coopertive computation of stereo disparity” Science 194,

Input : random dot stereo left image random dot shift the catch pat right image we can see the height different between the central and peripheral area

Constraints –Epipolar line constraint –Uniqueness constraint »each point in a image has only one depth value O.K. No. –Continuity constraint »each point is almost sure to have a depth value near the values of neighbors O.K. No.

Uniqueness constraint prohibits two or more matching points on one horizontal or vertical lines continuity constraint attracts more matching on a diagonal line ABCABC D E F A B C ABCABC (E-A) (E-B) (E-C) prohibit attract (D-A) (E-B) (F-C) Same depth

nn+1 relaxation

1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force (not a strategy ???) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness simulate the human visual system (MIT) Marr-Poggio Stereo(`76)

Ohta-Kanade Stereo(`85) Map making Ohta,Kanade “Stereo by intra- and inter-scanline search using dynamic programming”,IEEE Trans.,Vol. PAMI-7,No.2,pp

now matching become 1D to 1D yet, N line * M L * M R (512 * 100 * 100 * 10 m sec = 15 hours) L1 L2 L3 L4 L5 L6 R1 R2 R3 R4 R5 R6 L R disparity

Path Search u Matching problem can be considered as a path search problem u define a cost at each candidate of path segment based some ad-hoc function

Dynamic programming We can formalize the path finding problem as the following iterative formula optimum cost to K cost between M and K Optimum costs are known

stereo pair edges

pathdisparity depth

stereo pair edges depth

1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force (not a strategy ???) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness aerial image analysis (CMU) Ohta-Kanade Stereo(`85) Brightness of interval

Moravec Stereo(`79) navigation Moravec “Visual mapping by a robot rover” Proc 6th IJCAI,pp (1979)

Moravec’s cart Slide stereo Motion stereo

Slider stereo (9 eyes stereo) u 9 C 2 = 36 stereo pairs!!! u each stereo has an uncertainty measure u uncertainty = 1 / base-line u each stereo has a confidence measure long base line large uncertainty

Coarse to fine expand matching

σ estimated distance σ:uncertainty measure area:confidence measure 9 C 2 = 36 curves Interest point

1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force (not a strategy ???) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness navigation (Stanford) Moravec Stereo(`81) interest point

Summary 1. Two images from two different positions give depth information 2. Epipolar line and plane 3. Basic equation Z=-2df/(x”-x’) x”-x’: disparity 2d : base line length 4. case study area-based stereo Marr-poggio stereo simulate human visual system Ohta-Kanade stereo aerial image analysis Moravec stereo navigation 5. Read Horn pp

F matrix

Camera Model Pinhole camera

Camera Model geometry (X, Y, Z) Image plane X Y -Z x y (x, y) f : focal length Perspective projection View point (Optical center) (sX, sY, sZ)

Camera Model Perspective projection formularization Perspective projection (Non-linear) Affine projection (Linear) Projection matrix

Affine Camera Models General formularization OrthographicPerspective Affine camera

Affine Cameras perspectiveorthographic Focal length Distance from camera

Intrinsic parameters Image plane : an ideal image CCD : an actual picture Not equal ! CCD elements

Intrinsic parameters y An ideal image on the Image plane x u v θ An actual picture u0u0 v0v0 (x, y) (u, v)

Intrinsic parameters e.g. perspective projection Intrinsic matrix Projection matrix (normalized)

Extrinsic parameters Y X Z

Y X Z

R : rotation matrixt : translation vector

Summary (intrinsic & extrinsic parameters) Y X Z (X,Y,Z) World coordinate R, t (u, v) picture Camera coordinate World coordinate

Summary (intrinsic & extrinsic parameters) Y X Z (X,Y,Z) World coordinate R, t (u, v) picture 3 × 4 matrix

Epipolar geometry C1C1 C2C2 R Essential matrix : E

Essential & Fundamental matrix Image planes (ideal) Pictures (actual) Fundamental matrix : F Image 1 Image 2

F matrix (u, v, 1)(u’, v’, 1) F & (u, v) known Calculate the epipolar line picture 1picture 2

Computing F matrix (Linear solution)

Corner detector Extract interest points in each images x y Harris corner detector

Matching or

Computing F matrix (Linear solution) Suppose we found 8 pairs of corresponding points ·····

Computing F matrix (Singularity constraint) Epipolar pencil by linear solution (due to noise and error)

Computing F matrix (Singularity constraint) Singular value decomposition (SVD) Without noise, σ 3 must be 0 modification

Computing F matrix (Singularity constraint)

Summary u Pinhole camera and Affine camera u Intrinsic and extrinsic camera parameter u Epipolar geometry u Fundamental matrix