3D Reconstruction – Factorization Method Seong-Wook Joo KG-VISA 3/10/2004

Problem Setup P feature points (u p,v p ) from F frames Input: measurement matrix centroid = origin assumed for all frames Goal: Find motion and structure u 11 u 12    u 21 u 22    v 11 v 12    v 21 v 22    features frames

Camera model: orthographic camera i and j are unit vectors representing the x and y axis of the image plane in world coordinates. Camera matrix is essentially the rotation matrix with orthographic projection (no third row) No translation

Rotation matrix and shape matrix Measurement matrix W can be expressed as Where R, S represents rotation and shape

The Rank Theorem Since R is 2F  3 and S is 3  P, in the ideal case (without noise), W is at most of rank three. The rank theorem says the measurement matrix is highly redundant. In fact it resides in a 3-dimensional subspace.

The Factorization Algorithm SVD is used to decompose W into R and S. (Assuming 2F  P) Since the rank of W is at most 3, only the first three singular values (diagonal elements in  ) should be non-zero. But this does not hold in practice because of noise. Therefore the best rank-3 approximation W to the ideal W is obtained by taking the top 3 singular values.

Affine Reconstruction define so that e.g., However the decomposition is not unique. If Q is any invertible matrix, below is also a valid decomposition.

Euclidean Reconstruction Suppose the true R and S can be obtained by the linear transformation Q To find Q, We use the constraint that R consists of orthonormal vectors.

Shape Reconstruction Result

Extensions to Other Camera Models Affine camera –Scaled orthographic (weak perspective) Unknown scale factor f for each frame –Paraperspective camera matrix is still a 2x3 matrix (affine), with unknown offset m f and scale f for each frame –Same as orthographic case up to the affine reconstruction step –Use orthonormality of Rotation vectors to also solve for the additional unknowns Projective camera –Use depth( fp )-multiplied measurement matrix W –Depth estimation is another issue Reference –

Could we have used PCA? Measurement vectors A Px2F = [u 1 …u F v 1 …v F ] –Suppose we don’t know anything about camera geometry –Noisy measurements of unknown (hopefully linear) process We want –Invariant structure underlying the measurement data  shape –(variant) coefficients that gives a particular frame  motion PCA –Largest Eigenvectors of AA T : e 1, e 2, e 3  E AA T = E D E T –Comparing with the SVD A=W T =O 2 T  O 1 T AA T = O 2 T  O 1 T O 1  O 2 = O 2  2 O 2 T –E is essentially O 2, the “structure”

SVD output formats “Economy size” A mxn U mxn D nxn V T nxn m>n m<n Matlab default: D is the same size as A A mxn U mxm D mxn V T nxn A mxn U mxm D mxn V T nxn m>n m<n A mxn U mxm D mxm V T mxn (possible in theory, but Matlab doesn’t give this)