1. Day x 5-10 minute quiz SVD demo review of metric rectification algorithm 1.build a 2x3 matrix A from orthog line pairs A = [L1M1, L1M2+L2M1, L2M2; L1’M1’, L1’M2’+L2’M1’, L2’M2’] 2.find null vector s of A (using SVD) 3.decode s into S 4.Cholesky decompose S to find K: S = KK^t S and K are 2x2, so should be simple 5.metrically rectify image using affinity [K 0; 0 1] image should have been affinely rectified already

2. Camera from F Structure computation Alg 12.1, HZ318 7-point algorithm normalized 8-pt alg for F HZ 11.2 Fundamental matrix HZ 9.2 Epipolar geometry HZ 9.1

3. Linear constraints on F let x = (x,y,1) and x’ = (x’,y’,1) be a point correspondence between 2 images, and let F be the fundamental matrix of the images we know that the point pair satisfies x’ F x = 0 this generates a linear constraint on the entries of F = (f ij ): –x’xf11 + x’yf12 + x’f13 + y’xf21 +... + f33 = 0 –(x’x, x’y, x’, y’x, y’y, y’, x, y, 1) f = 0 –where f = (f11,12,...,f33) encodes F in row-major order each point correspondence (xi,xi’) yields a row (xi’xi, xi’yi, xi’, yi’xi, yi’yi, yi’, xi, yi, 1) collect n rows (x’x, x’y, x’, y’x, y’y, y’, x, y, 1) into nx9 matrix A F is found as the solution of Af = 0 HZ279

4. Solving Af=0 if data is exact and rank(A) is exactly 8, then f is found as the null vector of A –in a perfect world, Af=0 has a solution –recall: use SVD (and last column of V) to compute the null vector if data is noisy and rank(A) = 9, then f is found as a least squares solution –min ||Af|| –SVD, use last column of V –just like in the DLT algorithm HZ280

5. The singularity constraint force F to be singular –we know the correct F is singular (see above) –we need F to be singular, since the epipoles are computed using its null space –due to noise, the computed F will not be singular how? use another SVD! let F = UDV^t where D = diag(r,s,t) with r>=s>=t let F’ = UD’V^t where D’ = diag(r,s,0) –this is guaranteed to be singular since det(F’) = 0 HZ280-281 without with

6. Preconditioning recall preconditioning in DLT algorithm apply it in 8 point algorithm for F too before creating A (as in Af=0) from the point correspondences, normalize the points –centroid to origin translation + scaling to \sqrt 2 average distance –also called root mean square (RMS) scaling let x i  Tx i and x i ’  T’x i ’ be the transformation matrices denormalize via F  T’ t F T after finding F (i.e., after enforcing the singularity constraint) –shampoo? HZ282

7. Normalized 8 point algorithm basically 2 SVD’s, the first to find F and the second to refine it the fodder for the SVD’s is a matrix built from the point correspondences 1.SIFT: find a set of at least 8 point correspondences (xi,xi’) 2.precondition each pointset {xi} and {xi’} independently (with T and T’) 3.[solve for F as a null vector f] –build nx9 matrix A from x^t F x = 0 constraints, n>=8 –compute SVD of A: A = U 1 D V 1 ^t –f = row-major encoding of F = null vector of A = last column of V1 4.[impose singularity constraint] –compute SVD of F: F = U 2 D V 2 t –replace F by U 2 diag(d11,d22,0) V 2 t 5.postcondition: replace F by T’ t F T SPASSP (SIFT, precondition, A, SVD for f, SVD for singularity, postcondition) HZ282, Alg 11.1

8. 7-point algorithm F has 8dof so we need 8 constraints but the singularity constraint (det F = 0) can act as one of the 8 constraints hence, only need 7 constraints from point correspondences build 7x9 matrix A (as above) the 2D null space of Af=0 is spanned by the last 2 columns of V in the SVD of A –let v1 and v2 be the last 2 columns of V –the vector family f(t) = (1-t)v1 + tv2 yields a matrix family F(t) = (1-t)F1 + tF2 now solve det (F(t)) = 0 for t since det(F(t)) is a cubic polynomial in t, it has 1 or 3 real roots hence, this algorithm yields 1 or 3 solutions for F HZ281

9. More than one solution for F degeneracy: 3 solutions if the camera centers and 3D points Xi (from which the point pairs (xi,xi’)) all lie on a ‘critical surface’ –critical surface can be two planes, cone, cylinder or hyperboloid of one sheet (N.B. always a ruled quadric) this is quite likely in the 7-point algorithm: 7 points and 2 camera centers always lie on some quadric, since 9 points define a quadric –the only question is whether the quadric is ruled or not (ellipsoid, hyperboloid of one sheet) beware of all points in the same plane, or identical camera centers –this will yield a 2d family of family of fundamental matrices! HZ296

10. Iterative optimization an even better approximation to F can be found using an iterative method, bootstrapping from the 7-point or 8-point solutions based on the idea of searching only for singular matrices that minimize ||Af|| –rather than separating out the singularity constraint in a later stage requires the epipole, which we do not know, so it leads to an iterative process –guess F, compute associated e, compute new F, iterate this goes beyond our scope here: we’ll talk about it in seminar techniques: 2 SVD’s (HZ595 minimizations) and Levenberg-Marquardt algorithm (HZ600): Algorithm 11.2, HZ284 there is also a gold standard algorithm that involves even more computation: Algorithm 11.3, HZ285 HZ282-285; HZ595,600

11. Which algorithm is best? an algorithm for computing F may be evaluated by computing the distance of a point x i ’ from the computed epipolar line Fx i conclusions from HZ experiments (Figure 11.3): –normalized 8-pt algorithm performs just fine –use more than 8 points: error is reduced up to about 15 points, then starts leveling off –8-point with 15 points is better than more elaborate algorithms on 8 points add RANSAC estimation for added robustness (Alg 11.4) note that we are computing F with no user interaction HZ288-291

12. Adding RANSAC avoids outlier contamination SIFT will have plenty of outliers HZ290-291 without with

13. Don’t use line correspondences fascinating: line correspondences offer no information in 2 images! reason: 2 points backproject to 2 lines, which must intersect if the points are corresponding –this adds a constraint, since 2 lines in 3-space do not intersect in general however, 2 lines backproject to 2 planes, and 2 planes always intersect, so there is no added constraint line correspondences do become useful in 3 views: see trifocal tensors HZ294

14. But curve correspondences are useful shared tangency at epipolar lines can be leveraged to add a constraint –corresponding curves in the 2 images must be tangent to the corresponding epipolar lines see Exercises (ii) and (iii) at end of Chapter 11 tangency again: we will explore later HZ295, 309