1. Structure from Motion with Non-linear Least Squares
David Bargeron
Noah Snavely

2. Structure from Motion Input: Output Applications
Projection of a set of 3D points onto a set of camera projection planes
Output
3D point locations
Camera motion
Applications
Object tracking
“Match Moves” in TV, movies

3. Structure from Motion
Solution: minimize the residual error of the projections of reconstructed 3D points
Algorithm: Levenberg-Marquardt
(with Conjugate Gradient under the hood)

4. Levenberg-Marquardt
Function to minimize:
Jacobian:
Hessian:

5. Levenberg-Marquardt Inverse Hessian: Steepest Descent: Choose c:
Modified Hessian:
Levenberg-Marquardt:
, so

6. Synthetic Example #1 Generated test data from a rotating sphere
Sphere shape
Vertex projections

7. Synthetic Example #1 Solved for 3D vertex positions
Parameters of transformation R(Rj p) + tj
(Seven global parameters, four parameters per frame)
Ran Levenberg-Marquardt for 35 iterations

8. Synthetic Example #1
Iteration 0

9. Synthetic Example #1
Iteration 1

10. Synthetic Example #1
Iteration 2

11. Synthetic Example #1
Iteration 3

12. Synthetic Example #1
Iteration 6

13. Synthetic Example #1
Iteration 7

14. Synthetic Example #1
Iteration 8

15. Synthetic Example #1
Iteration 10

16. Synthetic Example #1 RMS Projection Error vs. Iteration
RMS 3D Error vs. Iteration

17. Synthetic Example #2
Pig shape
Vertex projections

18. Synthetic Example #2
True pig
Reconstructed pig

19. Real Application – Match Move
First tracked points in an input video
Video
Tracked points
(Thanks to Li Zhang for the video)

20. Real Application – Match Move
Solved for 3D points, camera motion
Reconstructed points

21. Real Application – Match Move
Used camera motion to insert synthetic object
Results!