1. Stanford CS223B Computer Vision, Winter 2007 Lecture 8 Structure From Motion Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens Slide credit: Gary Bradski, Stanford SAIL

2. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Summary SFM n Problem –Determine feature locations (=structure) –Determine camera extrinsic (=motion) n Two Principal Solutions –Bundle adjustment (nonlinear least squares, local minima) –SVD (through orthographic approximation, affine geometry) n Correspondence –(RANSAC) –Expectation Maximization

3. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Structure From Motion camera features Recover: structure (feature locations), motion (camera extrinsics)

4. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 SFM = Holy Grail of 3D Reconstruction n Take movie of object n Reconstruct 3D model n Would be commercially highly viable live.com

5. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Structure From Motion (1) [Tomasi & Kanade 92]

6. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Structure From Motion (2) [Tomasi & Kanade 92]

7. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Structure From Motion (3) [Tomasi & Kanade 92]

8. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Structure From Motion (4a): Images Marc Pollefeys

9. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Structure From Motion (4b) Marc Pollefeys

10. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Structure From Motion (5) http://www.cs.unc.edu/Research/urbanscape

11. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Structure From Motion n Problem 1: –Given n points p ij =(x ij, y ij ) in m images –Reconstruct structure: 3-D locations P j =(x j, y j, z j ) –Reconstruct camera positions (extrinsics) M i =(A j, b j ) n Problem 2: –Establish correspondence: c(p ij )

12. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Structure From Motion camera features Recover: structure (feature locations), motion (camera extrinsics)

13. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Recovery Problems 1 image2+ images Location knowncalibrationstereo Location unknown SFM, stitching

14. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 SFM: General Formulation fZ X O -x

15. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 SFM: Bundle Adjustment fZ X O -x

16. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bundle Adjustment n SFM = Nonlinear Least Squares problem n Minimize through –Gradient Descent –Conjugate Gradient –Gauss-Newton –Levenberg Marquardt common method n Prone to local minima

17. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Count # Constraints vs #Unknowns n m camera poses n n points n 2mn point constraints n 6m+3n unknowns n Suggests: need 2mn  6m + 3n n But: Can we really recover all parameters???

18. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 How Many Parameters Can’t We Recover? 036781012nmnm Place Your Bet! We can recover all but… m = #camera poses n = # feature points

19. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Count # Constraints vs #Unknowns n m camera poses n n points n 2mn point constraints n 6m+3n unknowns n Suggests: need 2mn  6m + 3n n But: Can we really recover all parameters??? –Can’t recover origin, orientation (6 params) –Can’t recover scale (1 param) n Thus, we need 2mn  6m + 3n - 7

20. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Are we done? n No, bundle adjustment has many local minima.

21. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The “Trick Of The Day” Replace Perspective by Orthographic Geometry Replace Euclidean Geometry by Affine Geometry Solve SFM linearly via PCA (“closed” form, globally optimal) Post-Process to make solution Euclidean Post-Process to make solution perspective By Tomasi and Kanade, 1992

22. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Orthographic Camera Model Orthographic = Limit of Pinhole Model: Extrinsic Parameters Rotation Orthographic Projection

23. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Orthographic Projection Limit of Pinhole Model: Orthographic Projection

24. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Orthographic SFM Problem subject to

25. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Affine SFM Problem subject to drop the constraints

26. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Count # Constraints vs #Unknowns n m camera poses n n points n 2mn point constraints n 8m+3n unknowns n Suggests: need 2mn  8m + 3n n But: Can we really recover all parameters???

27. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 How Many Parameters Can’t We Recover? 036781012nmnm Place Your Bet! We can recover all but…

28. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Answer is (at least): 12 CAA ii  '

29. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Points for Solving Affine SFM Problem n m camera poses n n points n Need to have: 2mn  8m + 3n-12

30. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Affine SFM Fix coordinate system by making p i0 = P 0 =origin Proof: Rank Theorem: Q has rank 3

31. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Rank Theorem n elements 2m elements

32. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Singular Value Decomposition

33. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Affine Solution to Orthographic SFM Gives also the optimal affine reconstruction under noise

34. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Back To Orthographic Projection Find C for which constraints are met Search in 9-dim space (instead of 8m + 3n-12)

35. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Back To Projective Geometry Orthographic (in the limit) Projective

36. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Back To Projective Geometry fZ X O -x Optimize Using orthographic solution as starting point

37. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The “Trick Of The Day” Replace Perspective by Orthographic Geometry Replace Euclidean Geometry by Affine Geometry Solve SFM linearly via PCA (“closed” form, globally optimal) Post-Process to make solution Euclidean Post-Process to make solution perspective By Tomasi and Kanade, 1992

38. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Structure From Motion n Problem 1: –Given n points p ij =(x ij, y ij ) in m images –Reconstruct structure: 3-D locations P j =(x j, y j, z j ) –Reconstruct camera positions (extrinsics) M i =(A j, b j ) n Problem 2: –Establish correspondence: c(p ij )

39. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Correspondence Problem View 1View 3View 2

40. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence: Solution 1 n Track features (e.g., optical flow) n …but fails when images taken from widely different poses

41. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence: Solution 2 n Start with random solution A, b, P n Compute soft correspondence: p(c|A,b,P) n Plug soft correspondence into SFM n Reiterate See Dellaert/Seitz/Thorpe/Thrun, Machine Learning Journal, 2003

42. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Example

43. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Results: Cube

44. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Animation

45. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Tomasi’s Benchmark Problem

46. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Reconstruction with EM

47. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 3-D Structure

48. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Correspondence: Alternative Approach n Ransac [Fisher/Bolles] = Random sampling and consensus n Will be discussed Wednesday

49. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Summary SFM n Problem –Determine feature locations (=structure) –Determine camera extrinsic (=motion) n Two Principal Solutions –Bundle adjustment (nonlinear least squares, local minima) –SVD (through orthographic approximation, affine geometry) n Correspondence –(RANSAC) –Expectation Maximization