CS 395/495-25: Spring 2004 IBMR: The Fundamental Matrix The Fundamental Matrix Combines Two Cameras Jack Tumblin

Slides:



Advertisements
Similar presentations
Image Rectification for Stereo Vision
Advertisements

Epipolar Geometry.
Lecture 11: Two-view geometry
1 A camera is modeled as a map from a space pt (X,Y,Z) to a pixel (u,v) by ‘homogeneous coordinates’ have been used to ‘treat’ translations ‘multiplicatively’
3D reconstruction.
Jan-Michael Frahm, Enrique Dunn Spring 2012
Two-view geometry.
Lecture 8: Stereo.
Epipolar Geometry class 11 Multiple View Geometry Comp Marc Pollefeys.
Camera calibration and epipolar geometry
Camera Models A camera is a mapping between the 3D world and a 2D image The principal camera of interest is central projection.
Computer Vision : CISC 4/689
1 Basic geometric concepts to understand Affine, Euclidean geometries (inhomogeneous coordinates) projective geometry (homogeneous coordinates) plane at.
Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.
Geometry of Images Pinhole camera, projection A taste of projective geometry Two view geometry:  Homography  Epipolar geometry, the essential matrix.
CS 395/495-25: Spring 2003 IBMR: Week 6B R 3  R 2 and P 3  P 2 The Camera Matrix Jack Tumblin
Epipolar Geometry Class 7 Read notes Feature tracking run iterative L-K warp & upsample Tracking Good features Multi-scale Transl. Affine.
Epipolar geometry. (i)Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the corresponding point.
Structure from motion. Multiple-view geometry questions Scene geometry (structure): Given 2D point matches in two or more images, where are the corresponding.
Uncalibrated Geometry & Stratification Sastry and Yang
Epipolar Geometry and the Fundamental Matrix F
Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.
Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.
CS 395/495-26: Spring 2003 IBMR: Week 1B 2-D Projective Geometry Jack Tumblin
Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.
Cameras and Projections Dan Witzner Hansen Course web page:
Previously Two view geometry: epipolar geometry Stereo vision: 3D reconstruction epipolar lines Baseline O O’ epipolar plane.
Lecture 20: Two-view geometry CS6670: Computer Vision Noah Snavely.
CS 395/495-25: Spring 2003 IBMR: Week 8A Chapter 8... Epipolar Geometry: Two Cameras: Jack Tumblin
CS 395/495-25: Spring 2003 IBMR: Week 7A The Camera Matrix Continued... Jack Tumblin
Lec 21: Fundamental Matrix
CS 558 C OMPUTER V ISION Lecture IX: Dimensionality Reduction.
3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective.
Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates.
776 Computer Vision Jan-Michael Frahm, Enrique Dunn Spring 2013.
Automatic Camera Calibration
Computer vision: models, learning and inference
Lecture 11 Stereo Reconstruction I Lecture 11 Stereo Reconstruction I Mata kuliah: T Computer Vision Tahun: 2010.
COMP 175: Computer Graphics March 24, 2015
Multi-view geometry.
Epipolar geometry The fundamental matrix and the tensor
Geometric Models & Camera Calibration
Lecture 04 22/11/2011 Shai Avidan הבהרה : החומר המחייב הוא החומר הנלמד בכיתה ולא זה המופיע / לא מופיע במצגת.
3D Sensing and Reconstruction Readings: Ch 12: , Ch 13: , Perspective Geometry Camera Model Stereo Triangulation 3D Reconstruction by.
Structure from Motion Course web page: vision.cis.udel.edu/~cv April 25, 2003  Lecture 26.
Stereo Course web page: vision.cis.udel.edu/~cv April 11, 2003  Lecture 21.
1 Formation et Analyse d’Images Session 7 Daniela Hall 25 November 2004.
Announcements Project 3 due Thursday by 11:59pm Demos on Friday; signup on CMS Prelim to be distributed in class Friday, due Wednesday by the beginning.
Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics CS329 Amnon Shashua.
Two-view geometry. Epipolar Plane – plane containing baseline (1D family) Epipoles = intersections of baseline with image planes = projections of the.
Feature Matching. Feature Space Outlier Rejection.
Computer vision: models, learning and inference M Ahad Multiple Cameras
MASKS © 2004 Invitation to 3D vision Uncalibrated Camera Chapter 6 Reconstruction from Two Uncalibrated Views Modified by L A Rønningen Oct 2008.
Reconstruction from Two Calibrated Views Two-View Geometry
Uncalibrated reconstruction Calibration with a rig Uncalibrated epipolar geometry Ambiguities in image formation Stratified reconstruction Autocalibration.
Lec 26: Fundamental Matrix CS4670 / 5670: Computer Vision Kavita Bala.
Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates.
55:148 Digital Image Processing Chapter 11 3D Vision, Geometry
Two-view geometry Computer Vision Spring 2018, Lecture 10
Epipolar geometry.
Epipolar Geometry class 11
Epipolar geometry continued
Two-view geometry.
Computer Graphics Recitation 12.
Uncalibrated Geometry & Stratification
Reconstruction.
Two-view geometry Epipolar geometry F-matrix comp. 3D reconstruction
Two-view geometry.
Two-view geometry.
Multi-view geometry.
Presentation transcript:

CS 395/495-25: Spring 2004 IBMR: The Fundamental Matrix The Fundamental Matrix Combines Two Cameras Jack Tumblin

Basic Camera P 0 : P 3  P 2 ( or camera R 3 ) Basic Camera P 0 is a 3x4 matrix:Basic Camera P 0 is a 3x4 matrix: Non-square pixels? change scaling (  x,  y )Non-square pixels? change scaling (  x,  y ) Parallelogram pixels? set nonzero skew sParallelogram pixels? set nonzero skew s K matrix: “(internal) camera calib. matrix” K matrix: “(internal) camera calib. matrix” xyz xcxcycyczczc11xcxcycyczczc111 = P 0 X = x P 0 X = x y y’ f z C ycycycyc xcxcxcxc zczczczc p  x f s p x 0 0  y f p y K (3x3 submatrix) [K 0] = P 0 [K 0] = P 0

ONE Camera Matrix P links P 3  P 2 : Basic camera:Basic camera: x = P 0 X where P 0 = [K | 0] = World-space camera:World-space camera: translate X (origin) to camera location C, then rotate: x = PX = (P 0 ·R·T) X = K·R·[ I C] X Rewrite as:Rewrite as: P = K[R -RC] Redundant notation: P = [M p 4 ] M = RK p 4 = -K · R · CRedundant notation: P = [M p 4 ] M = RK p 4 = -K · R · C Output: x 2D Camera Image Input: X 3D World Space z y x X (world space) zczczczc ycycycyc xcxcxcxc C x (camera space) ~ ~  x f s p x 0 0  y f p y ~ ~

ONE Complete Camera Matrix P K matrix: “internal camera calib. matrix”K matrix: “internal camera calib. matrix” R·T matrix: “external camera calib. matrix”R·T matrix: “external camera calib. matrix” –T matrix: Translate world to the cam. origin –R matrix: 3D rotate world to fit cam. axes Combine: write (P 0 ·R·T·X = x or K[R -RC] ·X = x orP·X = x orP·X = x Output: x 2D Camera Image Input: X 3D World Space zyx X (world space) zczczczc ycycycyc xcxcxcxc C x (camera space) ~C ~~C

TWO Cameras: Epipolar Geometry Basic idea: Given ONLY images from 2 cameras C, C’Given ONLY images from 2 cameras C, C’ Different views of same objects X, but we don’t know world-space points X.Different views of same objects X, but we don’t know world-space points X. If we choose an x, how can we find x’ ?If we choose an x, how can we find x’ ? How are x, x’ linked?How are x, x’ linked? C C’ x’ x X

TWO Cameras: Epipolar Geometry Basic idea: 2 cameras located at C, C’ in world space.2 cameras located at C, C’ in world space. Find ‘baseline’ through camera centers C, C’Find ‘baseline’ through camera centers C, C’ Baseline hits image planes at ‘epipoles’Baseline hits image planes at ‘epipoles’ Notice baseline and X form a plane... (OTHER planes thru baseline for other X...) baseline C C’ x’ x X

X TWO Cameras: Epipolar Geometry Basic idea: 2 cameras located at C, C’ in world space.2 cameras located at C, C’ in world space. Find ‘baseline’ through camera centers C, C’Find ‘baseline’ through camera centers C, C’ Baseline hits image planes at ‘epipoles’Baseline hits image planes at ‘epipoles’ Family of planes thru baseline are all the ‘epipolar planes’Family of planes thru baseline are all the ‘epipolar planes’ Image of planes = lines = ‘epipolar lines’Image of planes = lines = ‘epipolar lines’ Lines intersect at epipolar points in both images.Lines intersect at epipolar points in both images. C C’ baseline x’ x

TWO Cameras: Epipolar Geometry Summary: Connect cameras C, C’ with a baseline, whichConnect cameras C, C’ with a baseline, which hits image planes at epipoles e, e’. Chose any world pt X, then  everything is coplanar! epipolar plane  includes image points x, x’, and these connect to epipoles e,e’ by epipolar lines L, L’Chose any world pt X, then  everything is coplanar! epipolar plane  includes image points x, x’, and these connect to epipoles e,e’ by epipolar lines L, L’ baseline C C’ epipolar line L epipolar line L’ epipolar plane  X x’ e’ x e

Epipolar Geometry Useful properties: Every image point x maps to an epipolar line L’ also:Every image point x maps to an epipolar line L’ also: –Epipoles e,e’ = each camera’s view of the other –All epipolar lines L’ pass through epipole e’ –Epipolar Line L’ is (image of the C  X ray...) –Epipolar Line L’ links (image of C) to (image of X) baseline C C’ epipolar line L epipolar line L’ epipolar plane  X x’ e’ x e

Fundamental Matrix: Fx = L’ One Matrix Summarizes ALL of Epipolar Geometry! Fundamental Matrix F: 3x3, rank 2. Fundamental Matrix F: 3x3, rank 2. Maps image point x to image point x’: x’ T F x = 0Maps image point x to image point x’: x’ T F x = 0 but F is only Rank 2 – given only x, F cannot find x’ for you!! Maps image point x to epipolar line L’: F x = L’Maps image point x to epipolar line L’: F x = L’ baseline C C’ epipolar line L epipolar line L’ X x’ e’ x e epipolar plane 

Fundamental Matrix: Fx = L’ One Matrix Summarizes ALL of Epipolar Geometry! Fundamental Matrix F: 3x3, rank 2. Fundamental Matrix F: 3x3, rank 2. Maps image point x to image point x’: (x’ T F) x = 0Maps image point x to image point x’: (x’ T F) x = 0 (We know x is on line L, thus (L T ) x = 0, so (We know x is on line L, thus (L T ) x = 0, so Maps image point x to epipolar line L’: F x = L’Maps image point x to epipolar line L’: F x = L’ baseline C C’ epipolar line L epipolar line L’ X x’ e’ x e epipolar plane 

Fundamental Matrix: Fx = L’ 2) What is F if we DON’T know the cameras P, P’, but we DO know some corresp. point pairs (x, x’)?2) What is F if we DON’T know the cameras P, P’, but we DO know some corresp. point pairs (x, x’)? –F finds epipolar line L’ from point x : Fx = L’ –(Recall that if (any) point x’ is on line a L’, then x’ T L’= 0) –Substitute Fx for L’:x’ T F x = 0 AHA! we can find F using DLT-like method! see Chap. 10 baseline C C’ epipolar line L epipolar line L’ X x’ e’ x epipolar plane  X( )

Derive the Fundamental Matrix: –Recall Pseudo-Inverse: P + = P T ( P P T ) -1 and P P + = I (pg148) –Write world-space ray X  C as: X( ) = P + x + C –Other camera’s image of the ray is its epipolar line L’: L’( ) = P’X( ) = P’P + x + P’C. –But P’C = e’; it is the epipole of the other camera, so L’( ) = P’P + x + e’ baseline C C’ epipolar line L epipolar line L’ X x’ e’ x epipolar plane  X( )

Derive the Fundamental Matrix: –But P’C = e’; it is the epipole of the other camera, so L’( ) = P’P + x + e’. –Rewrite epipolar line L’ using points P’P + x and e’ –(Recall: find line between two P 2 points with cross product: L= x 1  x 2 ) –To get L’ = e’  P ’ P + x or L’ = F x baseline C C’ epipolar line L epipolar line L’ X x’ e’ x epipolar plane  X( )

Derive the Fundamental Matrix: –But P’C = e’; it is the epipole of the other camera, so L’( ) = P’P + x + e’. –Rewrite epipolar line L’ using points P’P + x and e’ –(Recall: find line between two P 2 points with cross product: L= x 1  x 2 ) –To get L’ = (e’)  (P ’ P + x) or L’ = F x THUS F = [e’]  P ’ P + baseline C C’ epipolar line L epipolar line L’ X x’ e’ x epipolar plane  X( ) Hunh? What Is This?!?!

Derive the Fundamental Matrix: F = [e’]  P ’ P + But what’s this? A NEW TRICK: F = [e’]  P ’ P + But what’s this? A NEW TRICK: –Cross Product written as matrix multiply (Zisserman pg. 554) a  b =  = = = [a]  · b –So write: a  b = -b  a = [a]  ·b = (a T ·[b]  ) T baseline C C’ epipolar line L epipolar line L’ X x’ e’ x epipolar plane  X( ) a1a1a2a2a3a3a1a1a2a2a3a3 b1b1b2b2b3b3b1b1b2b2b3b3 a 2 b 3 – a 3 b 2 a 3 b 1 – a 1 b 3 a 1 b 2 – a 2 b 1 0 -a 3 a 2 a 3 0 -a 1 -a 2 a a 3 a 2 a 3 0 -a 1 -a 2 a 1 0 b1b1b2b2b3b3b1b1b2b2b3b3 a ‘skew symmetric’ matrix

Fundamental Matrix: Fx = L’ Given a point pair (x,x’) Matrix F is unique!Given a point pair (x,x’) Matrix F is unique! Cool! still works even for different cameras!Cool! still works even for different cameras! F tied DIRECTLY to corresp. point pairs (x, x’):F tied DIRECTLY to corresp. point pairs (x, x’): –F finds epipolar line L’ from point x : Fx = L’ –(Recall that if (any) point x’ is on line a L’, then x’ T L’= 0) –Substitute Fx for L’:x’ T F x = 0 baseline C C’ epipolar line L epipolar line L’ X x’ e’ x epipolar plane  X( ) F = [e’]  P ’ P +

Fundamental Matrix: Fx = L’ 1) How do we find F? If we know the camera matrices P and P’ (we almost never do), we showed (Zisserman pg 224) F = [e’]  P ’ P + F = [e’]  P ’ P + baseline C C’ epipolar line L epipolar line L’ X x’ e’ x epipolar plane  X( ) (Recall: P + = P T (PP T ) -1, the pseudo-inverse) ( Recall: a  b = -b  a = [a]  ·b = (a T ·[b]  ) T )

Fundamental Matrix: Fx = L’ 2) What is F if we DON’T know the cameras P, P’, but we DO know some corresp. point pairs (x, x’)?2) What is F if we DON’T know the cameras P, P’, but we DO know some corresp. point pairs (x, x’)? –F finds epipolar line L’ from point x : Fx = L’ –(Recall that if (any) point x’ is on line a L’, then x’ T L’= 0) –Substitute Fx for L’:x’ T F x = 0 AHA! we can find F using DLT-like method! see Chap. 10 baseline C C’ epipolar line L epipolar line L’ X x’ e’ x epipolar plane  X( )

Fundamental Matrix Summary F is 3x3 matrix, maps P 2  P 2, rank 2, 7-DOF If world space pt X  image space pts. x and x’ then x’ T F x = 0If world space pt X  image space pts. x and x’ then x’ T F x = 0 Every image pt has epipolar line in the other image: Fx = L’F T x’ = LEvery image pt has epipolar line in the other image: Fx = L’F T x’ = L Baseline pierces image planes at epipoles e, e’ :Baseline pierces image planes at epipoles e, e’ : Fe = 0F T e’ = 0 Fe = 0F T e’ = 0 (pg. 226)

Fundamental Matrix Summary F is 3x3 matrix, maps P 2  P 2, rank 2, 7-DOF Given camera matrices P, P’, find F matrix by:Given camera matrices P, P’, find F matrix by: F = [e’]  P’ P + (recall: e’ is image of C: e’ = P’C) F is unaffected by any proj. transforms done on BOTH cameras (PH, P’H) has same F matrix as (P, P’) for any full-rank HF is unaffected by any proj. transforms done on BOTH cameras (PH, P’H) has same F matrix as (P, P’) for any full-rank H (e.g. F measures camera C vs. Camera C’ only, no matter where you put them) (pg. 226)

Fundamental Matrix Uses Special case: camera translate only (no rotations) Camera matrices are P= K[ I | 0 ], P’ = K[ I | t ]Camera matrices are P= K[ I | 0 ], P’ = K[ I | t ] –where K is internal calib., t is 3D translation vector F matrix simplifies to F = [e’] F matrix simplifies to F = [e’]  Epipolar lines are all parallel to direction tEpipolar lines are all parallel to direction t x,x’ displacement depends only on t & 3D depth z:x,x’ displacement depends only on t & 3D depth z: x’ = x + (K t )(1/z) txtxtytytztztxtxtytytztz

Fundamental Matrix Uses Can we find a camera matrix K from motion + fundamental F? Let one camera position define the world’s coords: P = P 0 = K[ I | 0 ], and define the other as P’ = [M | m] = K[R - RC]Let one camera position define the world’s coords: P = P 0 = K[ I | 0 ], and define the other as P’ = [M | m] = K[R | - RC] where K is internal calib., R is rotation C is world-space position F matrix simplifies to F = [ m ]  MF matrix simplifies to F = [ m ]  M If we know how we moved the camera (R, C matrices) then find F by correspondence, and then solve for K. (pg 237)If we know how we moved the camera (R, C matrices) then find F by correspondence, and then solve for K. (pg 237) ~ ~ ~

Fundamental Matrix Uses General movement? Recall: rotations don’t change image contentRecall: rotations don’t change image content (camera rotate  a P 3 homography matrix H) ANY cameras, ANY movements can then be warped to remove rotations, THUSANY cameras, ANY movements can then be warped to remove rotations, THUS Can ALWAYS get parallel epipolar lines!Can ALWAYS get parallel epipolar lines! –Easier to find correspondences –Easier to find depth values z –‘Parallel Epipolar Lines’==‘Rectified Image Pair’

Fundamental Matrix Properties Why bother with F? Can find it from image pt. correspondences onlyCan find it from image pt. correspondences only Works even for mismatched cameras (example: 100-year time-lapse of Eiffel tower)Works even for mismatched cameras (example: 100-year time-lapse of Eiffel tower) Choose your own world-space coordinate system.Choose your own world-space coordinate system. SVD lets us recover P, P’ camera matrices from FSVD lets us recover P, P’ camera matrices from F –(4-way ambiguity; what is front/back of C and C’?) pg 240 –BUT WE DON’T NEED TO! Complete 2-camera mapping from world  imageComplete 2-camera mapping from world  image –2 images + corresponding point pairs (x i,x’ i )  F –Let camera coords == 3D world coords, then (x i,x’ i )  X i (pg. 226)

Correspondence Problem: Where Computer Vision, IBMR part ways: Few Images: (image pairs, trios…) FIND Fundamental Matrix + FIND Corresponding point pairs (x,x’) ************Versus************** Many Images: GATHER enough for P2-only work: ignore or simplify the correspondence? Many Images: GATHER enough for P2-only work: ignore or simplify the correspondence?

Conclusions P 2, P 3 matrix forms give elegant, principled notation for ALL image geometryP 2, P 3 matrix forms give elegant, principled notation for ALL image geometry –Cameras, lights, points, lines, planes, conics, quadrics, twisted cubics, … –Matrix form makes everything reversible: 3D from (2D)*! –We can find shapes from images, BUT ONLY if we have enough reliable point correspondences… And THAT’s what we usually CAN’T find!

END

Another way to describe P3 lines! Given world-space camera position C =[a b c 1] andGiven world-space camera position C =[a b c 1] and a world-space direction D (found from a pixel x d, as shown on prev. slide),a world-space direction D (found from a pixel x d, as shown on prev. slide), even though D is a ‘point at infinity’, we can use it to make a parametric line in world space: line(t) = C + Dt =[a b c 1] + [d e f 0]teven though D is a ‘point at infinity’, we can use it to make a parametric line in world space: line(t) = C + Dt =[a b c 1] + [d e f 0]t Note that ‘t’ is position along the line, and it’s in world- space coords!Note that ‘t’ is position along the line, and it’s in world- space coords! If you have two cameras C0, C1, and found two corresponding image points p0 and p1, you can use method on previous slide to find a world-space direction D0, D1 for each image point, then make two parameterized lines L0(t0), L1(t1) and solve for their world-space intersection point X. THAT’S how you can do triangulation!.If you have two cameras C0, C1, and found two corresponding image points p0 and p1, you can use method on previous slide to find a world-space direction D0, D1 for each image point, then make two parameterized lines L0(t0), L1(t1) and solve for their world-space intersection point X. THAT’S how you can do triangulation!.