Projective cameras The end of projective SFM Euclidean upgrades Line geometry Purely projective cameras Présentations mercredi 26 mai de 9h30 à midi, salle U/V Planches : – –

An Affine Trick..Algebraic Scene Reconstruction Method

From Affine to Vectorial Structure Idea: pick one of the points (or their center of mass) as the origin.

What if we could factorize D? (Tomasi and Kanade, 1992) Affine SFM is solved! Singular Value Decomposition We can take

Back to perspective: Euclidean motion from E (Longuet-Higgins, 1981) Given F computed from n > 7 point correspondences, and its SVD F= UWV T, compute E=U diag(1,1,0) V T. There are two solutions t’ = u 3 and t’’ = -t’ to E T t=0. Define R’ = UWV T and R” = UW T V T where (It is easy to check R’ and R” are rotations.) Then [t £ ’]R’ = -E and [t £ ’]R” = E. Similar reasoning for t”. Four solutions. Only two of them place the reconstructed points in front of the cameras.

A different view of the fundamental matrix Projective ambiguity ! M’Q=[Id 0] MQ=[A b]. Hence: zp = [A b] P and z’p’ = [Id 0] P, with P=(x,y,z,1) T. This can be rewritten as: zp = ( A [Id 0] + [0 b] ) P = z’Ap’ + b. Or: z (b £ p) = z’ (b £ Ap’). Finally: p T Fp’ = 0 with F = [b £ ] A.

Projective motion from the fundamental matrix Given F computed from n > 7 point correspondences, compute b as the solution of F T b=0 with |b| 2 =1. Note that: [a £ ] 2 = aa T - |a| 2 Id for any a. Thus, if A 0 = - [b £ ] F, [b £ ] A 0 = - [b £ ] 2 F = - bb T F + |b| 2 F = F. The general solution is M = [A b] with A = A 0 + (  b | b |  b).

Two-view projective reconstruction. Mean relative error: 3.0%

Bundle adjustment Use nonlinear least-squares to minimize:

Bundle adjustment. Mean relative error: 0.2%

Projective SFM from multiple images z 11 p 11 … z 1n p 1n … … … z m1 p m1 … z mn p mn M1…MmM1…Mm P_1 … P_n =, D = MP If the z ij ’s are known, can be done via SVD. In principle the z ij ’s can be found pairwise from F (Triggs 96). Alternative, eliminate z ij from the minimization of E=|D-MP| 2 This reduces the problem to the minimization of E =  ij |p ij £ M i P j | 2 under the constraints |M i | 2 =|P j | 2 =1 with |p ij | 2 =1. Bilinear problem.

Bilinear projective reconstruction. Mean relative error: 0.2%

From uncalibrated to calibrated cameras Weak-perspective camera: Calibrated camera: Problem: what is Q ? Note: Absolute scale cannot be recovered. The Euclidean shape (defined up to an arbitrary similitude) is recovered.

Reconstruction Results (Tomasi and Kanade, 1992) Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi and T. Kanade, Proc. IEEE Workshop on Visual Motion (1991).  1991 IEEE.

What is some parameters are known? Weak-perspective camera: Zero skew: Problem: what is Q ? 0 Self calibration!

П1П1 Chasles’ absolute conic: x 1 2 +x 2 2 +x 3 2 = 0, x 4 = 0. Kruppa (1913); Maybank & Faugeras (1992) Triggs (1997); Pollefeys et al. (1998,2002)  ,  u 0, v 0 The absolute quadric u 0 = v 0 = 0 The absolute quadratic complex  2 =  2,  = 0 u0u0 v0v0 k l f x’ ¼ P ( H H -1 ) x H = [ X y ]

Relation between K, , and  *

x c ξ r y c Purely projective cameras

x c ξ r y x c ξ

x y   = x Ç y = = = (u ; v) s y – x u x £ y v The join of two points and Plücker coordinates (Euclidean version for) u O v Note: u. v = 0

x y  The join of two points and Plücker coordinates (projective version) u O v Note: u. v = 0  = x Ç y = uvuv [ ]

An inner product for lines  = (u ; v) !  * = (v ; u)  = (s ; t) ! (  |  ) =  *.  = .  * = u. t + v. s   (  |  ) = 0 Note: (  |  )= 2 u. v = 0

line screw P5P5 the Klein quadric Interpreting Plűcker coordinates

p2p2 Duality x p1p1 p3p3 x. p k = 0 x * = { p | x. p = 0 }

 = p Æ q = ( p Ç q ) * The meet of two planes p q 

   = x * Æ y * = ( x Ç y ) * Line duality x*x* y*y*   x y

The joint of a line and a point  x p =  Ç x p = [  Ç ] x where [  Ç ] = [u £ ] v -v T 0 When if  Ç x equal to 0?

The meet of a line and a plane  p x =  Æ p x = [  Æ ] p where [  Æ ] = [   Ç ] When if  Æ p equal to 0?

    1  1  1  2  2  2  3  3  3  4  4  4  5  5  5  6  6  6 Coplanar lines and Line bundles

» = u 1 » 1 + u 2 » 2 + u 3 » 3 Line bundles c x 33 11 22 

» = u 1 » 1 + u 2 » 2 + u 3 » 3 y = u 1 y 1 + u 2 y 2 + u 3 y 3 y2y2 c r x y y1y1 33 11 22  y3y3 Line bundles

» = X u, where X 2 R 6 £ 3, u 2 R 3 y = Y u, where Y 2 R 4 £ 3, u 2 R 3 y2y2 c r x y y1y1 33 11 22 y3y3 Line bundles 

u = Y z y y = Y z y [(c Ç x) Æ r] y2y2 c r x y y1y1 33 11 22 y3y3 Line bundles  Note:

u = Y z y y = Y z y [(c Ç x) Æ r] y2y2 c r x y y1y1 33 11 22 y3y3 Line bundles  Note: (c Ç x) Æ r = [c x – x c ] r TT

u = Y z y y = Y z y [(c Ç x) Æ r] = P x when z = c y2y2 c r x y y1y1 33 11 22 y3y3 Line bundles 

c r x y  u ¼ P x  ¼ P * p ¼ P T   ¼ P T y Perspective projection z  p’p’

Epipolar Geometry Epipolar Plane Epipoles Epipolar Lines Baseline

The fundamental matrix revisited ( »   | » 2 ) = 0 y 1 T F y 2 = 0 y 1 11 22  1 ¼ P 1 T y 1  2 ¼ P 2 T y 2 y 2

Trinocular Epipolar Constraints

    1  1  1  2  2  2  3  3  3  4  4  4  5  5  5  6  6  6 The trifocal tensor revisited T ( y 1, y 2, y 3 ) = 0

The trifocal tensor revisited D i ( » 1, » 2, » 3 ) = 0 or T i (u 1, u 2, u 3 ) = 0, for i = 1,2,3,4  1  1  1  2  2  2  3  3  3  4  4  4  5  5  5  6  6  6 δ η φ x (Ponce et al., CVPR’05)

c r x y  y ¼ P x  ¼ P * p ¼ P T   ¼ P T y Perspective projection z  p’p’

c r p  ’’ x’x’ p’p’ p’ ¼ X T p  ’ ¼ X *  x’  ¼ X T  ’  ¼ X T p’ x’ ¼ X*   Dual perspective projection

П1П1 Chasles’ absolute conic: x x x 3 2 = 0, x 4 = 0. The absolute quadratic complex:  T diag(Id,0)  = | u | 2 = 0.

e p T = H p p T e x = H -1 p x Coordinate changes --- Metric upgrades Planes: Points: Lines: e  = p  x’ ¼ P ( H H -1 ) x H = [ X y ]

Perspective projection c r x x ’  c r x x ’  x’ ¼ P x  ’ ¼ P *  p’ ¼ P T  ’  ¼ P T x’ x’ ¼ P x  ¼ P *  p’ ¼ P T   ¼ P T The AQC general equation:  T  = 0, with  = X *T X * Proposition:  T  ’ ¼ û ¢ û’ Proposition : P  P T ¼  ’ p  y ’  ’ y ’  ’ Proposition : P  * P T ¼  * Triggs (1997); Pollefeys et al. (1998) e p T = H p p T e x = H -1 p x e  = p 

Relation between K, , and  *

2480 points tracked in 196 images Non-linear, 7 images Non-linear, 20 images Non-linear, 196 images Linear, 20 images

Canon XL1 digital camcorder, 480 £ 720 pixel 2 (Ponce & McHenry, 2004) Projective structure from motion : Mahamud, Hebert, Omori & Ponce (2001)

Quantitative comparaison with Pollefeys et al. (1998, 2002) (synthetic cube with 30cm edges, corrupted by Gaussian noise)

What is a camera? (Ponce, CVPR’09) x c ξ r y x

x c ξ r y c

x c ξ r y x c ξ

x c ξ r y x c ξ ξ

x c ξ r y x ξ r y Linear family of lines x ξ x c ξ ξ ξ

Lines linearly dependent on 2 or 3 lines (Veblen & Young, 1910) Then go on recursively for general linear dependence © H. Havlicek, VUT

What a camera is Definition: A camera is a two-parameter linear family of lines – that is, a degenerate regulus, or a non-degenerate linear congruence.

Rank-3 families: Reguli Line fields ≡ epipolar plane images (Bolles, Baker, Marimont, 1987) Line bundles

Rank-4 (nondegenerate) families: Linear congruences Figures © H. Havlicek, VUT

x ξ y r x y r ξ Hyperbolic linear congruences Crossed-slit cameras (Zomet et al., 2003) Linear pushbroom cameras (Gupta & Hartley, 1997)

© E. Molzcan © Leica Hyperbolic linear congruences

© T. Pajdla, CTU Elliptic linear congruences Linear oblique cameras (Pajdla, 2002) Bilinear cameras (Yu & McMillan, 2004) Stereo panoramas / cyclographs (Seitz & Kim, 2002)

Parabolic linear congruences Pencil cameras (Yu & McMillam, 2004) Axial cameras (Sturm, 2005)

Plücker coordinates and the Klein quadric line screw the Klein quadric  = x Ç y = uvuv [ ] x y  Note: u. v = 0 P5P5

Pencils of screws and linear congruences line s P5P5 the Klein quadric Reciprocal screws: (s | t) = 0 Screw ≈ linear complex: s ≈ { ± | ( s | ± ) = 0 }

line s P5P5 the Klein quadric t l Pencils of screws and linear congruences Reciprocal screws: (s | t) = 0 Screw ≈ linear complex: s ≈ { ± | ( s | ± ) = 0 } Pencil of screws: l = { ¸ s + ¹ t ; ¸, ¹ 2 R } The carrier of l is a linear congruence

P5P5 e h p Reciprocal screws: (s | t) = 0 Screw ≈ linear complex: s ≈ { ± | ( s | ± ) = 0 } Pencil of screws: l = { ¸ s + ¹ t ; ¸, ¹ 2 R } The carrier of l is a linear congruence Pencils of screws and linear congruences

x ±2±2 Hyperbolic linear congruences »

x »1»1 p1p1 ±1±1 ±2±2 p2p2 » = (x T [ p 1 p 2 T ]x) » 1 + (x T [ p 1 p 2 T ]x) » 2 + (x T [ p 1 p 2 T ]x) » 3 + (x T [ p 1 p 2 T ]x) » 4 »2»2 »3»3 »4»4 »

x »1»1 p1p1 ±1±1 ±2±2 p2p2 Hyperbolic linear congruences » = (y T [ p 1 p 2 T ] y) » 1 + (y T [ p 1 q 2 T ] y) » 2 + (y T [ q 1 p 2 T ] y) » 3 + (y T [ q 1 q 2 T ] y) » 4 y = u 1 y 1 + u 2 y 2 + u 3 y 3 = Y u »2»2 »3»3 »4»4 » y

x »1»1 p1p1 ±1±1 ±2±2 p2p2 Hyperbolic linear congruences » = (u T [ ¼ 1 ¼ 2 T ]u) » 1 + (u T [ ¼ 1 ½ 2 T ]u) » 2 + (u T [ ½ 1 ¼ 2 T ]u) » 3 + (u T [ ½ 1 ½ 2 T ]u) » 4 = X û, where X 2 R 6 £ 4 and û 2 R 4 »2»2 »3»3 »4»4 » y

x ξ ± a2a2 p1p1 z p2p2 p a1a1 Parabolic linear congruences ± s ° T » = X û, where X 2 R 6 £ 5 and û 2 R 5

Elliptic linear congruences x » y » = X û, where X 2 R 6 £ 4 and û 2 R 4

x »1»1 y1y1 »2»2 y2y2 Epipolar geometry ( » 1 | » 2 ) = 0 or û 1 T F û 2 = 0, where F = X 1 T X 2 2 R 4 £ 4 Feldman et al. (2003): 6 £ 6 F for crossed-slit cameras Gupta & Hartley (1997): 4 £ 4 F for linear pushbroom cameras

Trinocular geometry D i ( » 1, » 2, » 3 ) = 0 or T i (û 1, û 2, û 3 ) = 0, for i = 1,2,3,4  1  1  1  2  2  2  3  3  3  4  4  4  5  5  5  6  6  6 δ η φ x

The essential map (Oblique cameras, Pajdla, 2002) x ξ Ax x ! ξ = x Ç Ax B H P E Canonical forms of essential maps A ( = matrices with quadratic minimal poylnomial) (Batog, Goaoc, Ponce, 2009) Alternative geometric characterization of linear congruences

A new elliptic camera? (Batog, Goaoc, Ponce, 2010)