Previously Two view geometry: epipolar geometry Stereo vision: 3D reconstruction epipolar lines Baseline O O’ epipolar plane

Today Orthographic projection Two views 3 Views: geometric interpretation >3 Views: factorization – simultaneous recovery of motion and shape

Orthographic Projection (Reminder) Parallel projection rays, orthogonal to image plane Focal center at infinity

Two Views Implies that Eliminating Z Since Therefore

Epipolar lines This is a linear equation

Further Simplification Select one point in first image and its corresponding point in the second image to be the origin of the two images In this coordinate frame translation is 0 Expression for epipolar lines:

Epipolar Line Recovery We need 4 corresponding points: 1 to eliminate translation 3 to determine the 4 components of R up to scale The rest of the components cannot be determined In particular, cannot be determined from, because these components are known only up to scale

Shape Recovery from Two Views Perspective:  Translation recovered up to scale  3D shape recovered up to scale  Recovery only if non-zero translation  No calibration – recovery up to a projective transformation (“projective shape”) Orthographic:  Rotation along epipolar line cannot be recovered  3D shape cannot be recovered  Recovery is possible up to an affine transformation (“affine shape”)  Recovery only if non-trivial rotation  Translation along line at infinity = rotation

Recovery from Three Views Under orthographic projection metric recovery is possible from three views Only rotation matters Rotation has three degrees of freedom Given an image, one rotation is in the image and two are out of plane rotations Ignoring the in-plane rotation we can associate the image with a point on the unit sphere

Recovery from Three Views Im1 Im2 Im3

Recovery from Three Views    a b c Im1 Im2 Im3

Recovery from Three Views a, b, c are unknown – rotation angles  are known – angles between epipolar lines Can we determine lengths from angles?    a b c

Recovery from Three Views In the plane the angles determine the sides of a triangle up to scale      

Recovery from Three Views On a sphere the sides are determined completely by the angles Therefore three views determine all the components of rotation (up to reflection) Sides (rotation angles) can be computed using the analogue of the cosine theorem on the sphere Once the rotation is known structure can be recovered    a b c

Factorization Simultaneous recovery of shape and motion Input:  A video sequence  Tracked feature points Assumptions:  Rigid scene  Orthographic projection  All tracked points appear in all frames Observation: tracked point locations satisfy linear relations that can be exploited for robust recovery

Singular Value Decomposition (SVD) Every real matrix can be decomposed to a product of three matrices: With   diagonal  U,V orthonormal U orthonormal basis to row space of M V orthonormal basis to column space of M

Singular Value Decomposition (SVD) or are called the singular values

Relation to Eigenvalue Problems

Singular Value Decomposition (SVD) Rank k least squares approximation of M Example: k=3  Take the 3 largest singular values:  Rank 3 approximation of M:

Factorization Goal: given p corresponding points in f frames, compute the 3-D location of each point and the transformation between the frames MeasurementsTransformationShape (3-D locations)

Factorization Step 1: eliminate translation Set the centroid of the points in each frame to be the origin Now

Factorization Constructing M :

Factorization Constructing M :

Factorization Constructing M :

Factorization Constructing M :

Factorization Constructing M :

Factorization Goal: given M, find S and T What should rank(M) be?

Factorization Goal: given M, find S and T Compute the SVD of M  rank(M) should be 3, since rank(T)=rank(S)=3  Noise cleaning: find the rank 3 approximation of M using the 3 largest singular values

So far: Define The decomposition can now be written as Factorization is not unique, since, A invertible Factorization

T should contain valid rotations 3f equations, 6 unknowns:  Each row is defined as  And should maintain Factorization

is 3x3, symmetric Linear system of equations in 6 unknowns Once B is recovered it can be factored to find A Solution is unique up to a global rotation

Input: tracked sequence Eliminate translation, produce M Use SVD to find the rank 3 approximation of M Factorization ambiguous, up to invertible matrix Find matrix A such that T contains valid rotations Solution is unique (up to a global rotation) Output: motion, shape Factorization (Algorithm)

Factorization Advantages  Simultaneous recovery of shape and motion  Simple algorithm, based on linear equations  Robust to noise Disadvantages  Orthographic projection  All points should appear in all frames (factorization with missing data is difficult)

Summary Shape and motion recovery under orthographic projection Two views:  Parallel epipolar lines  4 corresponding points are needed  Recovery of affine shape Three or more views  Metric recovery  Simultaneous recovery of shape and motion using SVD factorization