1. CSCE 641 Computer Graphics: Image-based Modeling (Cont.) Jinxiang Chai

2. Image-based Modeling and Rendering Images user input range scans Model Images Image based modeling Image- based rendering Geometry+ Images Geometry+ Materials Images + Depth Light field Panoroma Kinematics Dynamics Etc. Camera + geometry

3. Stereo reconstruction Given two or more images of the same scene or object, compute a representation of its shape knowncameraviewpoints

4. Stereo reconstruction Given two or more images of the same scene or object, compute a representation of its shape knowncameraviewpoints How to estimate camera parameters? - where is the camera? - where is it pointing? - what are internal parameters, e.g. focal length?

5. Spectrum of IBMR Images user input range scans Model Images Image based modeling Image- based rendering Geometry+ Images Geometry+ Materials Images + Depth Light field Panoroma Kinematics Dynamics Etc. Camera + geometry

6. Calibration from 2D motion Structure from motion (SFM) - track points over a sequence of images - estimate for 3D positions and camera positions - calibrate intrinsic camera parameters before hand Self-calibration: - solve for both intrinsic and extrinsic camera parameters

7. SFM = Holy Grail of 3D Reconstruction Take movie of object Reconstruct 3D model Would be commercially highly viable

8. How to Get Feature Correspondences Feature-based approach - good for images - feature detection (corners or sift features) - feature matching using RANSAC (epipolar line) Pixel-based approach - good for video sequences - patch based registration with lucas-kanade algorithm - register features across the entire sequence

9. Structure from Motion Two Principal Solutions Bundle adjustment (nonlinear optimization) Factorization (SVD, through orthographic approximation, affine geometry)

10. Projection Matrix Perspective projection: 2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters: KRTP

11. Nonlinear Approach for SFM What’s the difference between camera calibration and SFM?

12. Nonlinear Approach for SFM What’s the difference between camera calibration and SFM? - camera calibration: known 3D and 2D

13. Nonlinear Approach for SFM What’s the difference between camera calibration and SFM? - camera calibration: known 3D and 2D - SFM: unknown 3D and known 2D

14. Nonlinear Approach for SFM What’s the difference between camera calibration and SFM? - camera calibration: known 3D and 2D - SFM: unknown 3D and known 2D - what’s 3D-to-2D registration problem?

15. Nonlinear Approach for SFM What’s the difference between camera calibration and SFM? - camera calibration: known 3D and 2D - SFM: unknown 3D and known 2D - what’s 3D-to-2D registration problem?

16. SFM: Bundle Adjustment SFM = Nonlinear Least Squares problem Minimize through Gradient Descent Conjugate Gradient Gauss-Newton Levenberg Marquardt common method Prone to local minima

17. Count # Constraints vs #Unknowns M camera poses N points 2MN point constraints 6M+3N unknowns Suggests: need 2mn  6m + 3n But: Can we really recover all parameters???

18. Count # Constraints vs #Unknowns M camera poses N points 2MN point constraints 6M+3N unknowns (known intrinsic camera parameters) Suggests: need 2mn  6m + 3n But: Can we really recover all parameters??? Can’t recover origin, orientation (6 params) Can’t recover scale (1 param) Thus, we need 2mn  6m + 3n - 7

19. Are We Done? No, bundle adjustment has many local minima.

20. SFM Using Factorization Assume an othorgraphic camera Image World

21. SFM Using Factorization Assume othorgraphic camera Image World Subtract the mean

22. SFM Using Factorization Stack all the features from the same frame:

23. SFM Using Factorization Stack all the features from the same frame: Stack all the features from all the images: W

24. SFM Using Factorization Stack all the features from the same frame: Stack all the features from all the images: W

25. SFM Using Factorization Stack all the features from all the images: W Factorize the matrix into two matrix using SVD:

26. SFM Using Factorization Stack all the features from all the images: W Factorize the matrix into two matrix using SVD: Is the solution unique?

27. SFM Using Factorization Stack all the features from all the images: Factorize the matrix into two matrix using SVD: Is the solution unique? No!

28. SFM Using Factorization Stack all the features from all the images: W Factorize the matrix into two matrix using SVD: How to compute the matrix ?

29. SFM Using Factorization M is the stack of rotation matrix:

30. SFM Using Factorization M is the stack of rotation matrix: 10 10 10 10 Orthogonal constraints from rotation matrix

31. SFM Using Factorization M is the stack of rotation matrix: 10 10 10 10 Orthogonal constraints from rotation matrix

32. SFM Using Factorization 10 10 10 10 Orthogonal constraints from rotation matrices:

33. SFM Using Factorization 10 10 10 10 Orthogonal constraints from rotation matrices: QQ: symmetric 3 by 3 matrix

34. SFM Using Factorization 10 10 10 10 Orthogonal constraints from rotation matrices: How to compute QQ T ? least square solution - 4F linear constraints, 9 unknowns (6 independent due to symmetric matrix) QQ: symmetric 3 by 3 matrix

35. SFM Using Factorization 10 10 10 10 Orthogonal constraints from rotation matrices: How to compute QQ T ? least square solution - 4F linear constraints, 9 unknowns (6 independent due to symmetric matrix) How to compute Q from QQ T : SVD again: QQ: symmetric 3 by 3 matrix

36. SFM Using Factorization M is the stack of rotation matrix: 10 10 10 10 Orthogonal constraints from rotation matrix QQ T : symmetric 3 by 3 matrix Computing QQ T is easy: - 3F linear equations - 6 independent unknowns

37. SFM Using Factorization 1.Form the measurement matrix 2.Decompose the matrix into two matrices and using SVD 3.Compute the matrix Q with least square and SVD 4.Compute the rotation matrix and shape matrix: and

38. Weak-perspective Projection Factorization also works for weak-perspective projection (scaled orthographic projection): d z0z0

39. Factorization for Full-perspective Cameras [Han and Kanade]

40. SFM Summary Bundle adjustment (nonlinear optimization) - work with perspective camera model - work with incomplete data - prone to local minima Factorization: - closed-form solution for weak perspective camera - simple and efficient - usually need complete data - becomes complicated for full-perspective camera model Phil Torr’s structure from motion toolkit in matlab (click here)here Voodoo camera tracker (click here)here

41. All Together Video Click herehere - feature detection - feature matching (epipolar geometry) - structure from motion - stereo reconstruction - triangulation - texture mapping

42. SFM: Recent Development Large Scale SFM: Reconstructing the World from Internet Photos Project website: source code and data sets are available from http://www.cs.cornell.edu/projects/bigsfm/ http://www.cs.cornell.edu/projects/bigsfm/