1. Computing the Fundamental matrix Peter Praženica FMFI UK May 5, 2008

2. Contents Fundamental matrix Linear methods Basic algorithm Normalized Nonlinear methods The distance to epipolar lines The Gradient criterion and an interpretation as a distance The “optimal” (reprojected) method M-Estimator with the Tukey function Monte Carlo - Least-median-of-squares method Example of estimation with comparison

3. Fundamental matrix -The basic description of the projective geometry of two views - The computation of the Fundamental matrix is of utmost theoretical and practical importance (self-calibration method, recovering the 3D projective geometry) Stage for this chapter - Two views of a scene taken by two (perspective projective) cameras - n (respectively n’) points of interest have been detected in image 1 (respectively in image 2) - Goal: to establish a set of correspondences and to compute the estimate of the Fundamental matrix F that best explains those correspondences

4. The key idea: Each point correspondence (m, m') yields one equation, therefore with a sufficient number of point correspondences in general position we can determine F. No knowledge about the cameras or scene structure is necessary. Principle: Let and be two corresponding points. If the Fundamental matrix is, then the epipolar constraint can be rearranged as Where Equation 6.1 is linear and homogeneous in the 9 unknown coefficients of matrix F. Thus if we are given 8 matches, then we will be able, in general, to determine a unique solution for F, defined up to a scale factor. This approach is known as the eight-point algorithm. Linear methods (6.1)

5. In this case, we have more linear equations than unknowns, and, in the presence of noise, we look for the minimum of the error criterion which is equivalent to finding the minimum of (6.3) Where 2 solutions 1) closed-form solution via the linear equations 2) solution of subject to LM: Least-squares solution

6. Closed-form solution via the linear equations: One of the coefficients of F is set to 1, yielding a parameterization of F by eight values, which are the ratios of the eight other coefficients to the normalizing one. Let us assume for the sake of discussion that the normalizing coefficient is F33. We can rewrite (6.3) as where is formed with the first 8 columns of, and b is the ninth column of. The solution is known to be if the matrix is invertible. The second method solves the classical problem with The solution is the eigenvector associated with the smallest eigenvalue of.

7. Linear methods: Discussion The advantage of the eight-point algorithm is that it leads to a noniterative computation method which is easy to implement using any linear algebra numerical package; however, we have found that it is quite sensitive to noise, even with numerous data points. The constraint det(F) = 0 is not satisfied The quantity that is minimized in (6.2) does not have a good geometric interpretation, one could say that one might be trying to minimize an irrelevant quantity In all of the linear estimation techniques that will be described in this section, we deal with projective coordinates of the pixels. In a typical image of dimensions 512 x 512, a typical image point will have projective coordinates whose orders of magnitude are given componentwise by the triple (250,250,1). The fact that the first two coordinates are more than one order of magnitude larger than the third one has the effect of causing poor numerical conditioning. Solution: normalizing the points in the two images so that their three projective coordinates are of roughly the same order.

8. LM: Enforcing the rank constraint Two-step approximate linear determination of parameters: The idea is to note that in Equation 5.10 of Proposition 5.12, if the epipoles are known, then the Fundamental matrix is a linear function of the coefficients (a, b, c, d) of the epipolar homography. 1. We determine the epipole e by solving the constrained minimization problem subject to, which yields e as the unit norm eigenvector of matrix corresponding to the smallest eigenvalue 2. Once the epipoles are known, we can use Equation 5.10 or, if the epipoles are not at infinity, Equation 5.14, and write the expression as a linear expression of the coefficients of the epipolar homography: where (respectively ) are the affine coordinates of (respectively ) This equation can be used to solve for the coefficients of the epipolar homography in a least-squares formulation, thereby completing the determination of the parameters of F.

9. LM: Enforcing the rank constraint Using the closest rank 2 matrix: The matrix F found by the eight-point algorithm is replaced by the singular matrix that minimizes the Frobenius norm llF - F'll. This step can be done using a singular value decomposition. If, with, then. Solving a cubic equation: One selects two components of F, say, without loss of generality, F32 and F33 and rewrites (6.3) as The vector h is made of the first 7 components of the vector f, the matrix is made of the first 7 columns of matrix, and the vectors W 8 and W 9 are the eighth and ninth column vectors of that matrix, respectively. The solution h is We have a linear family of solutions parametrized by F32 and F33. In that family of solutions, there are some that satisfy the condition det(F) = 0. They are obtained by expressing the determinant of the solutions as a function of F32 and F33. One obtains a homogeneous polynomial of degree 3 which must be zero. Solving for one of the two ratios F32 : F33 or F33 : F32 one obtains at least one real solution for the ratio and therefore at least one solution for the Fundamental matrix.

10. Nonlinear methods The distance to epipolar lines The idea is to use a non-quadratic error function and to minimize However, two images should play the symmetric roles It can be written using the fact that. It can be observed that this criterion does not depend on the scale factor used to compute F. The Gradient criterion and an interpretation as a distance Uses criterion

11. Nonlinear methods The “optimal” method The nonlinear method based on the minimization of distances between observed points and reprojected ones where f is a vector of parameters for F, {Mi} are the 3D coordinates of the points in space, and P(f), P’(f) are the perspective projection matrices in the first and second images, given the Fundamental matrix F(f). That is, we estimate not only F, but also the most likely true positions. Because of the epipolar constraint, there are 4 - 1 degrees of freedom for each point correspondence, so the total number of parameters is 3p + 7. The estimation of the Fundamental matrix parameters f can be decoupled from the estimation of the parameters Mi : Since the optimal parameters can be determined from the parameters f, we have to minimize only over f. At each iteration, we have to perform the projective reconstruction of p points. The problem can be rewritten as

12. M-Estimator Let be the residual of the i-th datum. For example, in the case of the linear methods, in the case of the nonlinear methods The standard least-squares method tries to minimize, which is unstable if there are outliers present in the data. The M-estimators try to reduce the effect of outliers by replacing the squared residuals by another function of the residuals, yielding the error function where p is a symmetric, positive function with a unique minimum at zero and is chosen to be growing slower than the square function. The M-estimator of f based on the function is a vector f which is a solution of the system of q equations (6.16) where the derivative is called the influence function. If we now define define a weight function then Equation 6.16 becomes

13. M-Estimator with the Tukey function otherwise where is some estimated standard deviation of errors and c = 4.6851 is the tuning constant. The corresponding weight function is otherwise

14. Monte Carlo - Least-median-of-squares method It estimates the parameters by solving the nonlinear minimization problem Given p point correspondences,we proceed through the following steps 1) A Monte Carlo type technique is used to draw m random subsamples of q = 7 different point correspondences 2) For each subsample, indexed by J, we use the technique described in Proposition 5.14 to compute the Fundamental matrix F J. We may have at most 3 solutions 3) Compute the median, denoted by M J, of the squared residuals 4) Retain the estimate F J for which M J is minimal among all m M J ’s

15. Example of estimation with comparison - Pair of calibrated stereo images - There are 241 point matches which are established automatically - Outliers have been discarded - The calibrated parameters of the cameras are not used, but the Fundamental matrix computed from these parameters serves as a ground truth Methodee’RMSCPU Calibrated5138.18-8875.851642.02-2528.910.99 Linear5.85%304.018124.039256.219230.3063.400.13s Normalization7.20%-3920.67678.718489.07-15393.50.890.15s Nonlinear0.92%8135.03-14048.31896.19-2917.110.870.38s Gradient0.92%8166.05-14104.11897.80-2920.120.870.40s M-estimator0.12%4528.94-7516.31581.19-2313.720.871.05s Reprojected0.92%8165.05-14102.31897.74-2920.010.8719.1s LMedS0.13%3919.126413.11500.21-2159.650.752.40s (+ each F is normalized by its Frobenius norm) e, e’ positions of 2 epipoles RMS = squared distances between points and their epipolar lines CPU = approximate CPU time in seconds when the program is run on a Sparc 20 workstation

16. Better comparison method 1)Choose randomly a point m in the first image 2)Draw epipolar line of m in the second image using F 1 3)If the epipolar line does not intersect the second image, then go to Step 1 4)Choose randomly a point m’ on the epipolar line. Note that m and m’ correspond to each other exactly with respect to F 1 5)Draw the epipolar line of m in the second image using F 2, and compute the distance, denoted by d’ 1, between point m’ and line F 2 m 6)Draw the epipolar line of m’ in the first image using F 2, and compute the distance, denoted by d 1, between point m and line F 2 Tm’ 7)Conduct the same procedure from Step 2 through Step 6, but reversing the roles of F 1 and F 2, and compute d 2 and d’ 2 8)Repeat Step 1 through Step 7 N times 9)Compute the average distance of d’s, which is the measure of difference between the two Fundamental matrices

17. New comparason LinearNormalizedNonlinearGradientM-EstimatorReprojectedLMeDS Calibrated116.45.972.66 2.272.661.33 Linear117.29115.97116.40115.51116.25115.91 Normalized4.134.125.274.115.89 Nonlinear0.011.190.011.86 Gradient1.190.001.86 M-Estimator1.201.03 Reprojected1.88 Linear method is very bad The linear method with prior data normalization gives quite a reasonable result The nonlinear method based on point-line distances and that based on gradient- weighted epipolar errors give very similar result to those obtained based on minimization of distances between observed points and reprojected ones. The latter should be avoided because it is too time consuming. M-Estimators or the LMedS method give still better results because they try to limit or eliminate the effect of poorly localized points. The epipolar geometry estimated by LMedS is closer to the one computed through stereo calibration.