1. Structure from Motion

2. For now, static scene and moving cameraFor now, static scene and moving camera – Equivalently, rigidly moving scene and static camera Limiting case of stereo with many camerasLimiting case of stereo with many cameras Limiting case of multiview camera calibration with unknown targetLimiting case of multiview camera calibration with unknown target Given n points and N camera positions, have 2 nN equations and 3 n +6 N unknownsGiven n points and N camera positions, have 2 nN equations and 3 n +6 N unknowns

3. Approaches Obtaining point correspondencesObtaining point correspondences – Optical flow – Stereo methods: correlation, feature matching Solving for points and camera motionSolving for points and camera motion – Nonlinear minimization (bundle adjustment) – Various approximations…

4. OrthographicApproximation Simplest SFM case: camera approximated by orthographic projectionSimplest SFM case: camera approximated by orthographic projection PerspectiveOrthographic

5. Weak Perspective An orthographic assumption is sometimes well approximated by a telephoto lensAn orthographic assumption is sometimes well approximated by a telephoto lens Weak Perspective

6. Consequences of Orthographic Projection Scene can be recovered up to scaleScene can be recovered up to scale Translation perpendicular to image plane can never be recoveredTranslation perpendicular to image plane can never be recovered

7. Orthographic Structure from Motion Method due to Tomasi & Kanade, 1992Method due to Tomasi & Kanade, 1992 Assume n points in space p 1 … p nAssume n points in space p 1 … p n Observed at N points in time at image coordinates ( x ij, y ij )Observed at N points in time at image coordinates ( x ij, y ij ) – Feature tracking, optical flow, etc.

8. Orthographic Structure from Motion Write down matrix of dataWrite down matrix of data Points  Frames 

9. Orthographic Structure from Motion Step 1: find translationStep 1: find translation Translation parallel to viewing direction can not be obtainedTranslation parallel to viewing direction can not be obtained Translation perpendicular to viewing direction equals motion of average position of all pointsTranslation perpendicular to viewing direction equals motion of average position of all points

10. Orthographic Structure from Motion Subtract average of each rowSubtract average of each row

11. Orthographic Structure from Motion Step 2: try to find rotationStep 2: try to find rotation Rotation at each frame defines local coordinate axes,, andRotation at each frame defines local coordinate axes,, and ThenThen

12. Orthographic Structure from Motion So, can write where R is a “rotation” matrix and S is a “shape” matrixSo, can write where R is a “rotation” matrix and S is a “shape” matrix

13. Orthographic Structure from Motion Goal is to factorGoal is to factor Before we do, observe that rank ( ) = 3 (in ideal case with no noise)Before we do, observe that rank ( ) = 3 (in ideal case with no noise) Proof:Proof: – Rank of R is 3 unless no rotation – Rank of S is 3 iff have noncoplanar points – Product of 2 matrices of rank 3 has rank 3 With noise, rank ( ) might be > 3With noise, rank ( ) might be > 3

14. SVD Goal is to factor into R and SGoal is to factor into R and S Apply SVD:Apply SVD: But should have rank 3  all but 3 of the w i should be 0But should have rank 3  all but 3 of the w i should be 0 Extract the top 3 w i, together with the corresponding columns of U and VExtract the top 3 w i, together with the corresponding columns of U and V

15. Factoring for Orthographic Structure from Motion After extracting columns, U 3 has dimensions 2 N  3 (just what we wanted for R )After extracting columns, U 3 has dimensions 2 N  3 (just what we wanted for R ) W 3 V 3 T has dimensions 3  n (just what we wanted for S ) W 3 V 3 T has dimensions 3  n (just what we wanted for S ) So, let R * = U 3, S * = W 3 V 3 TSo, let R * = U 3, S * = W 3 V 3 T

16. Affine Structure from Motion The i and j entries of R * are not, in general, unit length and perpendicularThe i and j entries of R * are not, in general, unit length and perpendicular We have found motion (and therefore shape) up to an affine transformationWe have found motion (and therefore shape) up to an affine transformation This is the best we could do if we didn’t assume orthographic cameraThis is the best we could do if we didn’t assume orthographic camera

17. Ensuring Orthogonality Since can be factored as R * S *, it can also be factored as ( R * Q )( Q -1 S * ), for any QSince can be factored as R * S *, it can also be factored as ( R * Q )( Q -1 S * ), for any Q So, search for Q such that R = R * Q has the properties we wantSo, search for Q such that R = R * Q has the properties we want

18. Ensuring Orthogonality Want orWant or Let T = QQ TLet T = QQ T Equations for elements of T – solve by least squaresEquations for elements of T – solve by least squares Ambiguity – add constraintsAmbiguity – add constraints

19. Ensuring Orthogonality Have found T = QQ THave found T = QQ T Find Q by taking “square root” of TFind Q by taking “square root” of T – Cholesky decomposition if T is positive definite – General algorithms (e.g. sqrtm in Matlab)

20. Orthogonal Structure from Motion Let’s recap:Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation – Factor matrix using SVD – Write down equations for orthogonalization – Solve using least squares, square root At end, get matrix R = R * Q of camera positions and matrix S = Q -1 S * of 3D pointsAt end, get matrix R = R * Q of camera positions and matrix S = Q -1 S * of 3D points

21. Results Image sequenceImage sequence [Tomasi & Kanade]

22. Results Tracked featuresTracked features [Tomasi & Kanade]

23. Results Reconstructed shapeReconstructed shape [Tomasi & Kanade] Front view Top view

24. Orthographic  Perspective With orthographic or “weak perspective” can’t recover all informationWith orthographic or “weak perspective” can’t recover all information With full perspective, can recover more information (translation along optical axis)With full perspective, can recover more information (translation along optical axis) Result: can recover geometry and full motion up to global scale factorResult: can recover geometry and full motion up to global scale factor

25. Perspective SFM Methods Bundle adjustment (full nonlinear minimization)Bundle adjustment (full nonlinear minimization) Methods based on factorizationMethods based on factorization Methods based on fundamental matricesMethods based on fundamental matrices Methods based on vanishing pointsMethods based on vanishing points

26. Motion Field for Camera Motion Translation:Translation: Motion field lines converge (possibly at  )Motion field lines converge (possibly at  )

27. Motion Field for Camera Motion Rotation:Rotation: Motion field lines do not convergeMotion field lines do not converge

28. Motion Field for Camera Motion Combined rotation and translation: motion field lines have component that converges, and component that does notCombined rotation and translation: motion field lines have component that converges, and component that does not Algorithms can look for vanishing point, then determine component of motion around this pointAlgorithms can look for vanishing point, then determine component of motion around this point “Focus of expansion / contraction”“Focus of expansion / contraction” “Instantaneous epipole”“Instantaneous epipole”

29. Finding Instantaneous Epipole Observation: motion field due to translation depends on depth of pointsObservation: motion field due to translation depends on depth of points Motion field due to rotation does notMotion field due to rotation does not Idea: compute difference between motion of a point, motion of neighborsIdea: compute difference between motion of a point, motion of neighbors Differences should point towards instantaneous epipoleDifferences should point towards instantaneous epipole

30. SVD (Again!) Want to fit line to all  v (differences in optical flow) within some neighborhoodWant to fit line to all  v (differences in optical flow) within some neighborhood PCA on matrix of  vPCA on matrix of  v Equivalently, take eigenvector of A =  (  v)(  v) T corresponding to largest eigenvalueEquivalently, take eigenvector of A =  (  v)(  v) T corresponding to largest eigenvalue Gives direction of parallax l i in that patch, together with estimate of reliabilityGives direction of parallax l i in that patch, together with estimate of reliability

31. SFM Algorithm Compute optical flowCompute optical flow Find vanishing point (least squares solution)Find vanishing point (least squares solution) Find direction of translation from epipoleFind direction of translation from epipole Find perpendicular component of motionFind perpendicular component of motion Find velocity, axis of rotationFind velocity, axis of rotation Find depths of points (up to global scale)Find depths of points (up to global scale)