1. CS 395/495-25: Spring 2004 IBMR: The Fundamental Matrix The Fundamental Matrix Combines Two Cameras Jack Tumblin jet@cs.northwestern.edu

2. Basic Camera P 0 : P 3  P 2 ( or camera R 3 ) Basic Camera P 0 is a 3x4 matrix:Basic Camera P 0 is a 3x4 matrix: Non-square pixels? change scaling (  x,  y )Non-square pixels? change scaling (  x,  y ) Parallelogram pixels? set nonzero skew sParallelogram pixels? set nonzero skew s K matrix: “(internal) camera calib. matrix” K matrix: “(internal) camera calib. matrix” xyz xcxcycyczczc11xcxcycyczczc111 = P 0 X = x P 0 X = x y y’ f z C ycycycyc xcxcxcxc zczczczc p  x f s p x 0 0  y f p y 0 0 0 1 0 K (3x3 submatrix) [K 0] = P 0 [K 0] = P 0

3. ONE Camera Matrix P links P 3  P 2 : Basic camera:Basic camera: x = P 0 X where P 0 = [K | 0] = World-space camera:World-space camera: translate X (origin) to camera location C, then rotate: x = PX = (P 0 ·R·T) X = K·R·[ I C] X Rewrite as:Rewrite as: P = K[R -RC] Redundant notation: P = [M p 4 ] M = RK p 4 = -K · R · CRedundant notation: P = [M p 4 ] M = RK p 4 = -K · R · C Output: x 2D Camera Image Input: X 3D World Space z y x X (world space) zczczczc ycycycyc xcxcxcxc C x (camera space) ~ ~  x f s p x 0 0  y f p y 0 0 0 1 0 ~ ~

4. ONE Complete Camera Matrix P K matrix: “internal camera calib. matrix”K matrix: “internal camera calib. matrix” R·T matrix: “external camera calib. matrix”R·T matrix: “external camera calib. matrix” –T matrix: Translate world to the cam. origin –R matrix: 3D rotate world to fit cam. axes Combine: write (P 0 ·R·T·X = x or K[R -RC] ·X = x orP·X = x orP·X = x Output: x 2D Camera Image Input: X 3D World Space zyx X (world space) zczczczc ycycycyc xcxcxcxc C x (camera space) ~C ~~C

5. TWO Cameras: Epipolar Geometry Basic idea: Given ONLY images from 2 cameras C, C’Given ONLY images from 2 cameras C, C’ Different views of same objects X, but we don’t know world-space points X.Different views of same objects X, but we don’t know world-space points X. If we choose an x, how can we find x’ ?If we choose an x, how can we find x’ ? How are x, x’ linked?How are x, x’ linked? C C’ x’ x X

6. TWO Cameras: Epipolar Geometry Basic idea: 2 cameras located at C, C’ in world space.2 cameras located at C, C’ in world space. Find ‘baseline’ through camera centers C, C’Find ‘baseline’ through camera centers C, C’ Baseline hits image planes at ‘epipoles’Baseline hits image planes at ‘epipoles’ Notice baseline and X form a plane... (OTHER planes thru baseline for other X...) baseline C C’ x’ x X

7. X TWO Cameras: Epipolar Geometry Basic idea: 2 cameras located at C, C’ in world space.2 cameras located at C, C’ in world space. Find ‘baseline’ through camera centers C, C’Find ‘baseline’ through camera centers C, C’ Baseline hits image planes at ‘epipoles’Baseline hits image planes at ‘epipoles’ Family of planes thru baseline are all the ‘epipolar planes’Family of planes thru baseline are all the ‘epipolar planes’ Image of planes = lines = ‘epipolar lines’Image of planes = lines = ‘epipolar lines’ Lines intersect at epipolar points in both images.Lines intersect at epipolar points in both images. C C’ baseline x’ x

8. TWO Cameras: Epipolar Geometry Summary: Connect cameras C, C’ with a baseline, whichConnect cameras C, C’ with a baseline, which hits image planes at epipoles e, e’. Chose any world pt X, then  everything is coplanar! epipolar plane  includes image points x, x’, and these connect to epipoles e,e’ by epipolar lines L, L’Chose any world pt X, then  everything is coplanar! epipolar plane  includes image points x, x’, and these connect to epipoles e,e’ by epipolar lines L, L’ baseline C C’ epipolar line L epipolar line L’ epipolar plane  X x’ e’ x e

9. Epipolar Geometry Useful properties: Every image point x maps to an epipolar line L’ also:Every image point x maps to an epipolar line L’ also: –Epipoles e,e’ = each camera’s view of the other –All epipolar lines L’ pass through epipole e’ –Epipolar Line L’ is (image of the C  X ray...) –Epipolar Line L’ links (image of C) to (image of X) baseline C C’ epipolar line L epipolar line L’ epipolar plane  X x’ e’ x e

10. Fundamental Matrix: Fx = L’ One Matrix Summarizes ALL of Epipolar Geometry! Fundamental Matrix F: 3x3, rank 2. Fundamental Matrix F: 3x3, rank 2. Maps image point x to image point x’: x’ T F x = 0Maps image point x to image point x’: x’ T F x = 0 but F is only Rank 2 – given only x, F cannot find x’ for you!! Maps image point x to epipolar line L’: F x = L’Maps image point x to epipolar line L’: F x = L’ baseline C C’ epipolar line L epipolar line L’ X x’ e’ x e epipolar plane 

11. Fundamental Matrix: Fx = L’ One Matrix Summarizes ALL of Epipolar Geometry! Fundamental Matrix F: 3x3, rank 2. Fundamental Matrix F: 3x3, rank 2. Maps image point x to image point x’: (x’ T F) x = 0Maps image point x to image point x’: (x’ T F) x = 0 (We know x is on line L, thus (L T ) x = 0, so (We know x is on line L, thus (L T ) x = 0, so Maps image point x to epipolar line L’: F x = L’Maps image point x to epipolar line L’: F x = L’ baseline C C’ epipolar line L epipolar line L’ X x’ e’ x e epipolar plane 

12. Fundamental Matrix: Fx = L’ 2) What is F if we DON’T know the cameras P, P’, but we DO know some corresp. point pairs (x, x’)?2) What is F if we DON’T know the cameras P, P’, but we DO know some corresp. point pairs (x, x’)? –F finds epipolar line L’ from point x : Fx = L’ –(Recall that if (any) point x’ is on line a L’, then x’ T L’= 0) –Substitute Fx for L’:x’ T F x = 0 AHA! we can find F using DLT-like method! see Chap. 10 baseline C C’ epipolar line L epipolar line L’ X x’ e’ x epipolar plane  X( )

13. Derive the Fundamental Matrix: –Recall Pseudo-Inverse: P + = P T ( P P T ) -1 and P P + = I (pg148) –Write world-space ray X  C as: X( ) = P + x + C –Other camera’s image of the ray is its epipolar line L’: L’( ) = P’X( ) = P’P + x + P’C. –But P’C = e’; it is the epipole of the other camera, so L’( ) = P’P + x + e’ baseline C C’ epipolar line L epipolar line L’ X x’ e’ x epipolar plane  X( )

14. Derive the Fundamental Matrix: –But P’C = e’; it is the epipole of the other camera, so L’( ) = P’P + x + e’. –Rewrite epipolar line L’ using points P’P + x and e’ –(Recall: find line between two P 2 points with cross product: L= x 1  x 2 ) –To get L’ = e’  P ’ P + x or L’ = F x baseline C C’ epipolar line L epipolar line L’ X x’ e’ x epipolar plane  X( )

15. Derive the Fundamental Matrix: –But P’C = e’; it is the epipole of the other camera, so L’( ) = P’P + x + e’. –Rewrite epipolar line L’ using points P’P + x and e’ –(Recall: find line between two P 2 points with cross product: L= x 1  x 2 ) –To get L’ = (e’)  (P ’ P + x) or L’ = F x THUS F = [e’]  P ’ P + baseline C C’ epipolar line L epipolar line L’ X x’ e’ x epipolar plane  X( ) Hunh? What Is This?!?!

16. Derive the Fundamental Matrix: F = [e’]  P ’ P + But what’s this? A NEW TRICK: F = [e’]  P ’ P + But what’s this? A NEW TRICK: –Cross Product written as matrix multiply (Zisserman pg. 554) a  b =  = = = [a]  · b –So write: a  b = -b  a = [a]  ·b = (a T ·[b]  ) T baseline C C’ epipolar line L epipolar line L’ X x’ e’ x epipolar plane  X( ) a1a1a2a2a3a3a1a1a2a2a3a3 b1b1b2b2b3b3b1b1b2b2b3b3 a 2 b 3 – a 3 b 2 a 3 b 1 – a 1 b 3 a 1 b 2 – a 2 b 1 0 -a 3 a 2 a 3 0 -a 1 -a 2 a 1 0 0 -a 3 a 2 a 3 0 -a 1 -a 2 a 1 0 b1b1b2b2b3b3b1b1b2b2b3b3 a ‘skew symmetric’ matrix

17. Fundamental Matrix: Fx = L’ Given a point pair (x,x’) Matrix F is unique!Given a point pair (x,x’) Matrix F is unique! Cool! still works even for different cameras!Cool! still works even for different cameras! F tied DIRECTLY to corresp. point pairs (x, x’):F tied DIRECTLY to corresp. point pairs (x, x’): –F finds epipolar line L’ from point x : Fx = L’ –(Recall that if (any) point x’ is on line a L’, then x’ T L’= 0) –Substitute Fx for L’:x’ T F x = 0 baseline C C’ epipolar line L epipolar line L’ X x’ e’ x epipolar plane  X( ) F = [e’]  P ’ P +

18. Fundamental Matrix: Fx = L’ 1) How do we find F? If we know the camera matrices P and P’ (we almost never do), we showed (Zisserman pg 224) F = [e’]  P ’ P + F = [e’]  P ’ P + baseline C C’ epipolar line L epipolar line L’ X x’ e’ x epipolar plane  X( ) (Recall: P + = P T (PP T ) -1, the pseudo-inverse) ( Recall: a  b = -b  a = [a]  ·b = (a T ·[b]  ) T )

19. Fundamental Matrix: Fx = L’ 2) What is F if we DON’T know the cameras P, P’, but we DO know some corresp. point pairs (x, x’)?2) What is F if we DON’T know the cameras P, P’, but we DO know some corresp. point pairs (x, x’)? –F finds epipolar line L’ from point x : Fx = L’ –(Recall that if (any) point x’ is on line a L’, then x’ T L’= 0) –Substitute Fx for L’:x’ T F x = 0 AHA! we can find F using DLT-like method! see Chap. 10 baseline C C’ epipolar line L epipolar line L’ X x’ e’ x epipolar plane  X( )

20. Fundamental Matrix Summary F is 3x3 matrix, maps P 2  P 2, rank 2, 7-DOF If world space pt X  image space pts. x and x’ then x’ T F x = 0If world space pt X  image space pts. x and x’ then x’ T F x = 0 Every image pt has epipolar line in the other image: Fx = L’F T x’ = LEvery image pt has epipolar line in the other image: Fx = L’F T x’ = L Baseline pierces image planes at epipoles e, e’ :Baseline pierces image planes at epipoles e, e’ : Fe = 0F T e’ = 0 Fe = 0F T e’ = 0 (pg. 226)

21. Fundamental Matrix Summary F is 3x3 matrix, maps P 2  P 2, rank 2, 7-DOF Given camera matrices P, P’, find F matrix by:Given camera matrices P, P’, find F matrix by: F = [e’]  P’ P + (recall: e’ is image of C: e’ = P’C) F is unaffected by any proj. transforms done on BOTH cameras (PH, P’H) has same F matrix as (P, P’) for any full-rank HF is unaffected by any proj. transforms done on BOTH cameras (PH, P’H) has same F matrix as (P, P’) for any full-rank H (e.g. F measures camera C vs. Camera C’ only, no matter where you put them) (pg. 226)

22. Fundamental Matrix Uses Special case: camera translate only (no rotations) Camera matrices are P= K[ I | 0 ], P’ = K[ I | t ]Camera matrices are P= K[ I | 0 ], P’ = K[ I | t ] –where K is internal calib., t is 3D translation vector F matrix simplifies to F = [e’] F matrix simplifies to F = [e’]  Epipolar lines are all parallel to direction tEpipolar lines are all parallel to direction t x,x’ displacement depends only on t & 3D depth z:x,x’ displacement depends only on t & 3D depth z: x’ = x + (K t )(1/z) txtxtytytztztxtxtytytztz

23. Fundamental Matrix Uses Can we find a camera matrix K from motion + fundamental F? Let one camera position define the world’s coords: P = P 0 = K[ I | 0 ], and define the other as P’ = [M | m] = K[R - RC]Let one camera position define the world’s coords: P = P 0 = K[ I | 0 ], and define the other as P’ = [M | m] = K[R | - RC] where K is internal calib., R is rotation C is world-space position F matrix simplifies to F = [ m ]  MF matrix simplifies to F = [ m ]  M If we know how we moved the camera (R, C matrices) then find F by correspondence, and then solve for K. (pg 237)If we know how we moved the camera (R, C matrices) then find F by correspondence, and then solve for K. (pg 237) ~ ~ ~

24. Fundamental Matrix Uses General movement? Recall: rotations don’t change image contentRecall: rotations don’t change image content (camera rotate  a P 3 homography matrix H) ANY cameras, ANY movements can then be warped to remove rotations, THUSANY cameras, ANY movements can then be warped to remove rotations, THUS Can ALWAYS get parallel epipolar lines!Can ALWAYS get parallel epipolar lines! –Easier to find correspondences –Easier to find depth values z –‘Parallel Epipolar Lines’==‘Rectified Image Pair’

25. Fundamental Matrix Properties Why bother with F? Can find it from image pt. correspondences onlyCan find it from image pt. correspondences only Works even for mismatched cameras (example: 100-year time-lapse of Eiffel tower)Works even for mismatched cameras (example: 100-year time-lapse of Eiffel tower) Choose your own world-space coordinate system.Choose your own world-space coordinate system. SVD lets us recover P, P’ camera matrices from FSVD lets us recover P, P’ camera matrices from F –(4-way ambiguity; what is front/back of C and C’?) pg 240 –BUT WE DON’T NEED TO! Complete 2-camera mapping from world  imageComplete 2-camera mapping from world  image –2 images + corresponding point pairs (x i,x’ i )  F –Let camera coords == 3D world coords, then (x i,x’ i )  X i (pg. 226)

26. Correspondence Problem: Where Computer Vision, IBMR part ways: Few Images: (image pairs, trios…) FIND Fundamental Matrix + FIND Corresponding point pairs (x,x’) ************Versus************** Many Images: GATHER enough for P2-only work: ignore or simplify the correspondence? Many Images: GATHER enough for P2-only work: ignore or simplify the correspondence?

27. Conclusions P 2, P 3 matrix forms give elegant, principled notation for ALL image geometryP 2, P 3 matrix forms give elegant, principled notation for ALL image geometry –Cameras, lights, points, lines, planes, conics, quadrics, twisted cubics, … –Matrix form makes everything reversible: 3D from (2D)*! –We can find shapes from images, BUT ONLY if we have enough reliable point correspondences… And THAT’s what we usually CAN’T find!

28. END

29. Another way to describe P3 lines! Given world-space camera position C =[a b c 1] andGiven world-space camera position C =[a b c 1] and a world-space direction D (found from a pixel x d, as shown on prev. slide),a world-space direction D (found from a pixel x d, as shown on prev. slide), even though D is a ‘point at infinity’, we can use it to make a parametric line in world space: line(t) = C + Dt =[a b c 1] + [d e f 0]teven though D is a ‘point at infinity’, we can use it to make a parametric line in world space: line(t) = C + Dt =[a b c 1] + [d e f 0]t Note that ‘t’ is position along the line, and it’s in world- space coords!Note that ‘t’ is position along the line, and it’s in world- space coords! If you have two cameras C0, C1, and found two corresponding image points p0 and p1, you can use method on previous slide to find a world-space direction D0, D1 for each image point, then make two parameterized lines L0(t0), L1(t1) and solve for their world-space intersection point X. THAT’S how you can do triangulation!.If you have two cameras C0, C1, and found two corresponding image points p0 and p1, you can use method on previous slide to find a world-space direction D0, D1 for each image point, then make two parameterized lines L0(t0), L1(t1) and solve for their world-space intersection point X. THAT’S how you can do triangulation!.