1. CSCE 641 Computer Graphics: Image-based Modeling (Cont.) Jinxiang Chai

2. Image-based Modeling and Rendering Images user input range scans Model Images Image based modeling Image- based rendering Geometry+ Images Geometry+ Materials Images + Depth Light field Panoroma Kinematics Dynamics Etc. Camera + geometry

3. Stereo reconstruction Given two or more images of the same scene or object, compute a representation of its shape knowncameraviewpoints

4. Stereo reconstruction Given two or more images of the same scene or object, compute a representation of its shape knowncameraviewpoints How to estimate camera parameters? - where is the camera? - where is it pointing? - what are internal parameters, e.g. focal length?

5. Spectrum of IBMR Images user input range scans Model Images Image based modeling Image- based rendering Geometry+ Images Geometry+ Materials Images + Depth Light field Panoroma Kinematics Dynamics Etc. Camera + geometry

6. How can we estimate the camera parameters?

7. Camera calibration Augmented pin-hole camera - focal point, orientation - focal length, aspect ratio, center, lens distortion Known 3D Classical calibration - 3D 2D - correspondence Camera calibration online resources

8. Camera and calibration target

9. Classical camera calibration Known 3D coordinates and 2D coordinates - known 3D points on calibration targets - find corresponding 2D points in image using feature detection algorithm

10. Camera parameters u0u0 v0v0 100 -s y 0 sxsx а u v 1 Perspective proj. View trans. Viewport proj. Known 3D coords and 2D coords

11. Camera parameters u0u0 v0v0 100 -s y 0 sxsx а u v 1 Perspective proj. View trans. Viewport proj. Known 3D coords and 2D coords Intrinsic camera parameters (5 parameters) extrinsic camera parameters (6 parameters)

12. Camera matrix Fold intrinsic calibration matrix K and extrinsic pose parameters (R,t) together into a camera matrix M = K [R | t ] (put 1 in lower r.h. corner for 11 d.o.f.)

13. Camera matrix calibration Directly estimate 11 unknowns in the M matrix using known 3D points (X i,Y i,Z i ) and measured feature positions (u i,v i )

14. Camera matrix calibration Linear regression: Bring denominator over, solve set of (over-determined) linear equations. How?

15. Camera matrix calibration Linear regression: Bring denominator over, solve set of (over-determined) linear equations. How? Least squares (pseudo-inverse) - 11 unknowns (up to scale) - 2 equations per point (homogeneous coordinates) - 6 points are sufficient

16. Nonlinear camera calibration Perspective projection:

17. Nonlinear camera calibration Perspective projection: KRTP

18. Nonlinear camera calibration Perspective projection: 2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters: KRTP

19. Nonlinear camera calibration Perspective projection: 2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters: KRTP

20. Multiple calibration images Find camera parameters which satisfy the constraints from M images, N points: for j=1,…,M for i=1,…,N This can be formulated as a nonlinear optimization problem:

21. Multiple calibration images Find camera parameters which satisfy the constraints from M images, N points: for j=1,…,M for i=1,…,N This can be formulated as a nonlinear optimization problem: Solve the optimization using nonlinear optimization techniques: - Gauss-newton - Levenberg-MarquardtLevenberg-Marquardt

22. Nonlinear approach Advantages: can solve for more than one camera pose at a time fewer degrees of freedom than linear approach Standard technique in photogrammetry, computer vision, computer graphics - [Tsai 87] also estimates lens distortions (freeware @ CMU) http://www.cs.cmu.edu/afs/cs/project/cil/ftp/html/v-source.html Disadvantages: more complex update rules need a good initialization (recover K [R | t] from M)

23. How can we estimate the camera parameters?

24. Application: camera calibration for sports video [Farin et. Al] imagesCourt model

25. Calibration from 2D motion Structure from motion (SFM) - track points over a sequence of images - estimate for 3D positions and camera positions - calibrate intrinsic camera parameters before hand Self-calibration: - solve for both intrinsic and extrinsic camera parameters

26. SFM = Holy Grail of 3D Reconstruction Take movie of object Reconstruct 3D model Would be commercially highly viable

27. How to Get Feature Correspondences Feature-based approach - good for images - feature detection (corners) - feature matching using RANSAC (epipolar line) Pixel-based approach - good for video sequences - patch based registration with lucas-kanade algorithm - register features across the entire sequence

28. Structure from Motion Two Principal Solutions Bundle adjustment (nonlinear optimization) Factorization (SVD, through orthographic approximation, affine geometry)

29. Nonlinear Approach for SFM What’s the difference between camera calibration and SFM?

30. Nonlinear Approach for SFM What’s the difference between camera calibration and SFM? - camera calibration: known 3D and 2D

31. Nonlinear Approach for SFM What’s the difference between camera calibration and SFM? - camera calibration: known 3D and 2D - SFM: unknown 3D and known 2D

32. Nonlinear Approach for SFM What’s the difference between camera calibration and SFM? - camera calibration: known 3D and 2D - SFM: unknown 3D and known 2D - what’s 3D-to-2D registration problem?

33. Nonlinear Approach for SFM What’s the difference between camera calibration and SFM? - camera calibration: known 3D and 2D - SFM: unknown 3D and known 2D - what’s 3D-to-2D registration problem?

34. SFM: Bundle Adjustment SFM = Nonlinear Least Squares problem Minimize through Gradient Descent Conjugate Gradient Gauss-Newton Levenberg Marquardt common method Prone to local minima

35. Count # Constraints vs #Unknowns M camera poses N points 2MN point constraints 6M+3N unknowns Suggests: need 2mn  6m + 3n But: Can we really recover all parameters???

36. Count # Constraints vs #Unknowns M camera poses N points 2MN point constraints 6M+3N unknowns (known intrinsic camera parameters) Suggests: need 2mn  6m + 3n But: Can we really recover all parameters??? Can’t recover origin, orientation (6 params) Can’t recover scale (1 param) Thus, we need 2mn  6m + 3n - 7

37. Are We Done? No, bundle adjustment has many local minima.

38. SFM Using Factorization Assume an othorgraphic camera Image World

39. SFM Using Factorization Assume othorgraphic camera Image World Subtract the mean

40. SFM Using Factorization Stack all the features from the same frame:

41. SFM Using Factorization Stack all the features from the same frame: Stack all the features from all the images: W

42. SFM Using Factorization Stack all the features from the same frame: Stack all the features from all the images: W

43. SFM Using Factorization Stack all the features from all the images: W Factorize the matrix into two matrix using SVD:

44. SFM Using Factorization Stack all the features from all the images: Factorize the matrix into two matrix using SVD:

45. SFM Using Factorization Stack all the features from all the images: W Factorize the matrix into two matrix using SVD: How to compute the matrix ?

46. SFM Using Factorization M is the stack of rotation matrix:

47. SFM Using Factorization M is the stack of rotation matrix: 10 10 10 10 Orthogonal constraints from rotation matrix

48. SFM Using Factorization M is the stack of rotation matrix: 10 10 10 10 Orthogonal constraints from rotation matrix

49. SFM Using Factorization 10 10 10 10 Orthogonal constraints from rotation matrices:

50. SFM Using Factorization 10 10 10 10 Orthogonal constraints from rotation matrices: QQ: symmetric 3 by 3 matrix

51. SFM Using Factorization 10 10 10 10 Orthogonal constraints from rotation matrices: How to compute QQ T ? least square solution - 4F linear constraints, 9 unknowns (6 independent due to symmetric matrix) QQ: symmetric 3 by 3 matrix

52. SFM Using Factorization 10 10 10 10 Orthogonal constraints from rotation matrices: How to compute QQ T ? least square solution - 4F linear constraints, 9 unknowns (6 independent due to symmetric matrix) How to compute Q from QQ T : SVD again: QQ: symmetric 3 by 3 matrix

53. SFM Using Factorization M is the stack of rotation matrix: 10 10 10 10 Orthogonal constraints from rotation matrix QQ T : symmetric 3 by 3 matrix Computing QQ T is easy: - 3F linear equations - 6 independent unknowns

54. SFM Using Factorization 1.Form the measurement matrix 2.Decompose the matrix into two matrices and using SVD 3.Compute the matrix Q with least square and SVD 4.Compute the rotation matrix and shape matrix: and

55. Weak-perspective Projection Factorization also works for weak-perspective projection (scaled orthographic projection): d z0z0

56. Factorization for Full-perspective Cameras [Han and Kanade]

57. SFM for Deformable Objects For detail, click herehere

58. SFM Using Factorization Bundle adjustment (nonlinear optimization) - work with perspective camera model - work with incomplete data - prone to local minima Factorization: - closed-form solution for weak perspective camera - simple and efficient - usually need complete data - becomes complicated for full-perspective camera model Phil Torr’s structure from motion toolkit in matlab (click here)here Voodoo camera tracker (click here)here

59. All Together Video Click herehere - feature detection - feature matching (epipolar geometry) - structure from motion - stereo reconstruction - triangulation - texture mapping