Multiple-view Reconstruction from Points and Lines Yi Ma ECE Department, CSL, Beckman University of Illinois at Urbana-Champaign

Problem formulation Input: Corresponding images (of “features”) in multiple images. Output: Camera motion, camera calibration, object structure. Here is a picture illustrating the fundamental problem that we are interested in multiple view geometry. The scenario is that we are given multiple pictures or images of some 3D object and after establishing correspondence of certain geometric features such as a point, line or curve, we then try to use such information to recover the relative locations of the camera where these images are taken, as well as recover the 3D structure of the object. Yi Ma Tutorial ICRA 2003

Projection model – point features Homogeneous coordinates of a 3-D point Homogeneous coordinates of its 2-D image Projection of a 3-D point to an image plane Now let me quickly go through the basic mathematical model for a camera system. Here is the notation. We will use a four dimensional vector X for the homogeneous coordinates of a 3-D point p, its image on a pre-specified plane will be described also in homogeneous coordinate as a three dimensional vector x. If everything is normalized, then W and z can be chosen to be 1. We use a 3x4 matrix Pi to denote the transformation from the world frame to the camera frame. R may stand for rotation, T for translation. Then the image x and the world coordinate X of a point is related through the equation, where lambda is a scale associated to the depth of the 3D point relative to the camera center o. But in general the matrix Pi can be any 3x4 matrix, because the camera may add some unknown linear transformation on the image plane. Usually it is denoted by a 3x 3 matrix A(t). Yi Ma Tutorial ICRA 2003

Image of a line feature Homogeneous representation of a 3-D line Homogeneous representation of its 2-D co-image Projection of a 3-D line to an image plane First let us talk about line features. To describe a line in 3-D, we need to specify a base point on the line and a vector indicating the direction of the line. On the image plane we can use a three dimensional vector l to describe the image of a line L. More specifically, if x is the image of a point on this line, its inner product with l is 0. Yi Ma Tutorial ICRA 2003

Incidence relations among features Multiview constraints are nothing but incidence relations at play! Now consider multiple images of a simplest object, say a cube. All the constraints are incidence relations, are all of the same nature. Is there any way that we can express all the constraints in a unified way? Yes, there is. . . . Yi Ma Tutorial ICRA 2003

Traditional multifocal constraints For images of the same 3-D point : Let us first see how multiple view geometry is traditionally studied. In particular, what is the nature of multilinear constraints among multiple images. For given m images of a 3-D point p, we can rewrite the equations in a single matrix form as follows. Speculate this equation for a while, we may observe that both lambda and X are associated to the 3-D location of the point p, relative to different coordinate frames. To eliminate these 3-D structural parameters, we can form the following matrix N. From the above equation, it is straightforward to see that this matrix is rank deficient. Furthermore, if this matrix has an exact rank m+3, we can recover \lambda and X as its only kernel. Hence we can reduce our study to examine this matrix N. Notice that given a camera configuration specified by the matrices Pi’s, not any set of vectors x1, …, xm would make this matrix rank deficient. That means that images from the same 3-D point should satisfy certain relationship. One way to describe such relationship is by writing down the determinants of all m+4 by m+4 submatrices of N and these determinants must be zero. Turns out these equations can be reduced to equations involving images from 2, 3, or 4 views at a time, which leads to the traditional multilinear constraints for 2, 3, 4 views… (leading to the conventional approach) Multilinear constraints among 2, 3, 4-wise views Yi Ma Tutorial ICRA 2003

Rank conditions for point feature WLOG, choose camera frame 1 as the reference Multiple-View Matrix Lemma [Rank Condition for Point Features] My opinion on this approach now is that it is way too early to write down the rank deficiency condition on N in terms of the determinants of its submatrices. After a little linear algebraic manipulation on N, we may obtain a matrix H of this form whose rank is closely related to that of N. More specifically, rank(N) is equal to m+2 plus rank(M). This gives us two possible cases: one is more generic and the other degenerate. In any case, we know that if x1, …, xm are images of some 3-D point p, the matrix H must be rank deficient, or in other words, the two columns of H are linearly dependent. Let then and are linearly dependent. Yi Ma Tutorial ICRA 2003

Rank conditions vs. multifocal constraints These constraints are only necessary but NOT sufficient! However, there is NO further relationship among quadruple wise views. Quadrilinear constraints hence are redundant! Now let us examine the rank deficient M matrix and see what we get from it. First observe that the two columns of M are linearly dependent implies that the pair of vectors from each three rows must be linearly dependent. This in fact gives rise to the well-known epipolar constraints. One thing we notice from this derivation is that, epipolar constraints are only necessary but not sufficient for matrix H to be rank deficient. Yi Ma Tutorial ICRA 2003

Multiple-view matrix: line vs. point Point Features Line Features We may then derive a M matrix for a line feature, we call it M_l. Comparing with the M matrix for a point feature, now denoted as M_p, they both have rank 1. M_l is however a matrix of four columns. The linear dependency between any two rows of M_l gives rise to the trilinear constraints in terms of image lines, which is well-known in the literature. Geometric interpretation for the M_l matrix is no longer a sphere, but a circle. This circle gives a family of parallel lines which may give the same M_l matrix. The radius of the circle is the distance of these lines from the center of the reference camera frame and the normal of this circle is the direction of these lines. Yi Ma Tutorial ICRA 2003

Rank conditions: line vs. point Continue with constraints that the rank of the matrix M_l imposes on multiple images of a line feature. If rank(M) is 1, it means that all the planes through every camera center and image line intersect at a unique line in 3-D. If the rank is 0, it corresponds to the only degenerate case that all the planes are the same hence the 3-D line is determined only up to a plane on which all the camera centers must lie. Yi Ma Tutorial ICRA 2003

Global multiple-view analysis: examples The rank condition actually allows to you to do multiple view analysis globally. For example if you choose a multiple view matrix as following. Its rank can only be 1 or 2. Corresponding to each value, there is a generic picture of the configuration of the 3D features involved and the relative camera configuration. Yi Ma Tutorial ICRA 2003

A family of intersecting lines each can randomly take the image of any of the lines: What is the essential meaning of this rank 2 case? Here is an example explaining it. Consider a family of lines in 3D intersecting at one point p. You then randomly choose the image of any of the lines in the family in each view and form a multiple view matrix. Then this matrix in general has rank 2. We know before that if all the images chosen happen to correspond to the same 3D line, the rank of M is 1. Here, you don’t need to have exact correspondence among those lines, yet you still get some non-trivial constraints among their images… Nonlinear constraints among up to four views . . . Yi Ma Tutorial ICRA 2003

Universal rank condition Theorem [The Universal Rank Condition] for images of a point on a line: Multi-nonlinear constraints among 3, 4-wise images. Multi-linear constraints among 2, 3-wise images. Here comes my favorite slide: Consider multiple images of a point on a line. In order to express all the incidence constraints associated to these features and all their images, you can formally define a matrix M as following, where the components D_i and D_i^perp are up to you to choose and plug in. D_I’s are really the images of either the point or the line, D_I^perp’s are the coimages. Choices for D_I and D_j are independent if I is not equal to j. The interesting thing is no matter what you choose, the rank of the resulting matrix must be one of the two cases. The second case gives you all the multi-linear constraints, of course only up to 2 or 3 views; if you really want to know what is among four views, there are some nonlinear constraints. We will see a few examples and give you some intuitive ideas… Yi Ma Tutorial ICRA 2003

Instances with mixed features Examples: Case 1: a line reference Case 2: a point reference Here we take two instantionations of the multiple view matrix. All previously known constraints are the theorem’s instances. Degenerate configurations if and only if a drop of rank. Yi Ma Tutorial ICRA 2003

Generalization – restriction to a plane Homogeneous representation of a 3-D plane We have talked about points and lines, what about plane? Suppose now the point or the line feature belongs to some plane in 3D, say pi. We know in general such a plane can be expressed by the following equation. We may lump all the coefficients into a vector pi. Pi^1 is just the first three components and pi^2 is the d. Then the rank condition we had before must be modified somehow since now we have this extra restriction. It turns out all you need to do is to append an extra row to the multiple view matrix and then all the rank conditions remain exactly the same. Corollary [Coplanar Features] Rank conditions on the new extended remain exactly the same! Yi Ma Tutorial ICRA 2003

Generalization – restriction to a plane Given that a point and line features lie on a plane in 3-D space: GENERALIZATION – Multiple View Matrix: Coplanar Features In addition to previous constraints, it simultaneously gives homography: For example, the multiple view matrices for m images of a point or a line on such a plane are given by the following two matrices. According to the theorem, they both should have rank 1. What this extra row gives you is exactly the so called homography in computer vision literature. In addition to the homography, the multiple view matrix keeps all the constraints. Yi Ma Tutorial ICRA 2003

Multiple-view structure and motion recovery Given images of points: The third problem, also the most important one, is how to recover camera configuration from given m images of a set of n points. Using the rank deficiency condition on M, the problem becomes looking for T2, R2, …, Tm, Rm such that the two columns of M are linearly dependent. That is, there exists coefficients alpha^j’s such that the equations hold. I won’t get into the detail how to determine those coefficients alpha^j’s even without knowing all the camera motions. For now, I only tell you their values depend on a choice of coordinate frames. These frames could be either Euclidean, affine or projective. Assuming these coefficients are known, then finding the camera configuration simply becomes a problem of solving a linear equation. This equation have a unique solution if we have in general more than 6 points. Yi Ma Tutorial ICRA 2003

SVD based 4-step algorithm for SFM Here is the outline of this algorithm. The only thing I’d like to point out is that the algorithm is initialized by a two view algorithm due to the fact that non-trivial constraints for point features start with two views. Yi Ma Tutorial ICRA 2003

Utilizing all incidence relations Three edges intersect at each vertex. . . . Yi Ma Tutorial ICRA 2003

Example: simulations Yi Ma Tutorial ICRA 2003

Example: simulations Yi Ma Tutorial ICRA 2003

Example: experiments Errors in all right angles < 1o Yi Ma Tutorial ICRA 2003

Summary Incidence relations <=> rank conditions Rank conditions => multiple-view factorization Rank conditions implies all multi-focal constraints Rank conditions for points, lines, planes, and (symmetric) structures. Yi Ma Tutorial ICRA 2003