2. Stanford CS223B Computer Vision, Winter 2005 Lecture 11: Structure From Motion 2 Sebastian Thrun, Stanford Rick Szeliski, Microsoft Hendrik Dahlkamp and Dan Morris, Stanford

3. Sebastian Thrun Stanford University CS223B Computer Vision Overall Distribution

4. Sebastian Thrun Stanford University CS223B Computer Vision Question 1: Calibration

5. Sebastian Thrun Stanford University CS223B Computer Vision Question 1: Calibration n Calibration with planar unknown target n Unknown parameters 4 intrinsics 6K extrinsics (K = #images) 2M calibration target parameters (but can’t recover 3) 2KM constraints

6. Sebastian Thrun Stanford University CS223B Computer Vision Question 2: Perspective Geometry

7. Sebastian Thrun Stanford University CS223B Computer Vision Question 2: Perspective Geometry n Collinearity in 3D  2D (but not converse) n Order in 3D  2D (but not converse) n Equidistance: Not preserved! n Proof (collinearity in 2D):

8. Sebastian Thrun Stanford University CS223B Computer Vision Question 3: Stereopsis

9. Sebastian Thrun Stanford University CS223B Computer Vision Question 3: Stereopsis

10. Sebastian Thrun Stanford University CS223B Computer Vision Question 3: Stereopsis How does  Z scale with Z? – in approximation!!!

11. Sebastian Thrun Stanford University CS223B Computer Vision Question 4: True or False

12. Sebastian Thrun Stanford University CS223B Computer Vision Question 5: Build A System!

13. Sebastian Thrun Stanford University CS223B Computer Vision Question 5: Build A System! n Range: stereo or laser n Classification : template, optical flow?, SIFT? n Alternatively: segmentation, range discontinuities n Prediction: person and car n Robustness: normalize image, bring light source n (many other possibilities)

14. Stanford CS223B Computer Vision, Winter 2005 Lecture 11: Structure From Motion 2 Sebastian Thrun, Stanford Rick Szeliski, Microsoft Hendrik Dahlkamp and Dan Morris, Stanford

15. Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion (1) [Tomasi & Kanade 92]

16. Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion (2) [Tomasi & Kanade 92]

17. Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion (3) [Tomasi & Kanade 92]

18. Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion n Problem 1: –Given n points p ij =(x ij, y ij ) in m images –Reconstruct structure: 3-D locations P j =(x j, y j, z j ) –Reconstruct camera positions (extrinsics) M i =(A j, b j ) n Problem 2: –Establish correspondence: c(p ij )

19. Sebastian Thrun Stanford University CS223B Computer Vision The “Trick Of The Day” n Replace Euclidean Geometry by Affine Geometry n Solve SFM linearly (“closed” form) n Post-Process to make Euclidean n By Tomasi and Kanade, 1992

20. Sebastian Thrun Stanford University CS223B Computer Vision Orthographic Camera Model Limit of Pinhole Model: Extrinsic Parameters Rotation Orthographic Projection

21. Sebastian Thrun Stanford University CS223B Computer Vision Orthographic Projection Limit of Pinhole Model: Orthographic Projection

22. Sebastian Thrun Stanford University CS223B Computer Vision The Affine SFM Problem

23. Sebastian Thrun Stanford University CS223B Computer Vision Count # Constraints vs #Unknowns n m camera poses n n points n 2mn point constraints n 8m+3n unknowns n Suggests: need 2mn  8m + 3n n But: Can we really recover all parameters???

24. Sebastian Thrun Stanford University CS223B Computer Vision How Many Parameters Can’t We Recover? 036891012nmnm Place Your Bet! We can recover all but…

25. Sebastian Thrun Stanford University CS223B Computer Vision The Answer is (at least): 12

26. Sebastian Thrun Stanford University CS223B Computer Vision Points for Solving Affine SFM Problem n m camera poses n n points n Need to have: 2mn  8m + 3n-12

27. Sebastian Thrun Stanford University CS223B Computer Vision Affine SFM Fix coordinate system by making p 0 =origin Proof: Rank Theorem: Q has rank 3

28. Sebastian Thrun Stanford University CS223B Computer Vision The Rank Theorem n elements 2m elements

29. Sebastian Thrun Stanford University CS223B Computer Vision Tomasi/Kanade 1992 Singular Value Decomposition

30. Sebastian Thrun Stanford University CS223B Computer Vision Tomasi/Kanade 1992 Gives also the optimal affine reconstruction under noise

31. Sebastian Thrun Stanford University CS223B Computer Vision Back To Orthographic Projection Find C and d for which constraints are met

32. Sebastian Thrun Stanford University CS223B Computer Vision Back To Projective Geometry Orthographic (in the limit) Projective

33. Sebastian Thrun Stanford University CS223B Computer Vision The “Trick Of The Day” n Replace Euclidean Geometry by Affine Geometry n Solve SFM linearly (“closed” form) n Post-Process to make Euclidean n By Tomasi and Kanade, 1992

34. Sebastian Thrun Stanford University CS223B Computer Vision SFM With Projective Camera: See Rick Szeliski’s Lecture! Non-Linear Optimization Problem: Bundle Adjustment!

35. Sebastian Thrun Stanford University CS223B Computer Vision Structure From Motion n Problem 1: –Given n points p ij =(x ij, y ij ) in m images –Reconstruct structure: 3-D locations P j =(x j, y j, z j ) –Reconstruct camera positions (extrinsics) M i =(A j, b j ) n Problem 2: –Establish correspondence: c(p ij )

36. Sebastian Thrun Stanford University CS223B Computer Vision The Correspondence Problem View 1View 3View 2

37. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence: Solution 1 n Track features (e.g., optical flow) n …but fails when images taken from widely different poses

38. Sebastian Thrun Stanford University CS223B Computer Vision Correspondence: Solution 2 n Start with random solution A, b, P n Compute soft correspondence: p(c|A,b,P) n Plug soft correspondence into SFM n Reiterate n See Dellaert et al 2003, Machine Learning Journal

39. Sebastian Thrun Stanford University CS223B Computer Vision Example

40. Sebastian Thrun Stanford University CS223B Computer Vision Results: Cube

41. Sebastian Thrun Stanford University CS223B Computer Vision Animation

42. Sebastian Thrun Stanford University CS223B Computer Vision Tomasi’s Benchmark Problem

43. Sebastian Thrun Stanford University CS223B Computer Vision Reconstruction with EM

44. Sebastian Thrun Stanford University CS223B Computer Vision 3-D Structure

45. Sebastian Thrun Stanford University CS223B Computer Vision Summary SFM n Problem –Determine feature locations (=structure) –Determine camera extrinsic (=motion) –The name SFM is somewhat of a misdemeanor n Two Principal Solutions –Nonlinear optimization (local minima) –Linear (affine geometry) n Correspondence –RANSAC –Expectation Maximization