1. 3D Reconstruction – Factorization Method Seong-Wook Joo KG-VISA 3/10/2004

2. Problem Setup P feature points (u p,v p ) from F frames Input: measurement matrix centroid = origin assumed for all frames Goal: Find motion and structure u 11 u 12    u 21 u 22    v 11 v 12    v 21 v 22    features frames

3. Camera model: orthographic camera i and j are unit vectors representing the x and y axis of the image plane in world coordinates. Camera matrix is essentially the rotation matrix with orthographic projection (no third row) No translation

4. Rotation matrix and shape matrix Measurement matrix W can be expressed as Where R, S represents rotation and shape

5. The Rank Theorem Since R is 2F  3 and S is 3  P, in the ideal case (without noise), W is at most of rank three. The rank theorem says the measurement matrix is highly redundant. In fact it resides in a 3-dimensional subspace.

6. The Factorization Algorithm SVD is used to decompose W into R and S. (Assuming 2F  P) Since the rank of W is at most 3, only the first three singular values (diagonal elements in  ) should be non-zero. But this does not hold in practice because of noise. Therefore the best rank-3 approximation W to the ideal W is obtained by taking the top 3 singular values.

7. Affine Reconstruction define so that e.g., However the decomposition is not unique. If Q is any invertible matrix, below is also a valid decomposition.

8. Euclidean Reconstruction Suppose the true R and S can be obtained by the linear transformation Q To find Q, We use the constraint that R consists of orthonormal vectors.

9. Shape Reconstruction Result

10. Extensions to Other Camera Models Affine camera –Scaled orthographic (weak perspective) Unknown scale factor f for each frame –Paraperspective camera matrix is still a 2x3 matrix (affine), with unknown offset m f and scale f for each frame –Same as orthographic case up to the affine reconstruction step –Use orthonormality of Rotation vectors to also solve for the additional unknowns Projective camera –Use depth( fp )-multiplied measurement matrix W –Depth estimation is another issue Reference –http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/UESHIBA1/ueshiba.html

11. Could we have used PCA? Measurement vectors A Px2F = [u 1 …u F v 1 …v F ] –Suppose we don’t know anything about camera geometry –Noisy measurements of unknown (hopefully linear) process We want –Invariant structure underlying the measurement data  shape –(variant) coefficients that gives a particular frame  motion PCA –Largest Eigenvectors of AA T : e 1, e 2, e 3  E AA T = E D E T –Comparing with the SVD A=W T =O 2 T  O 1 T AA T = O 2 T  O 1 T O 1  O 2 = O 2  2 O 2 T –E is essentially O 2, the “structure”

12. SVD output formats “Economy size” A mxn U mxn D nxn V T nxn m>n m<n Matlab default: D is the same size as A A mxn U mxm D mxn V T nxn A mxn U mxm D mxn V T nxn m>n m<n A mxn U mxm D mxm V T mxn (possible in theory, but Matlab doesn’t give this)