1. Digital Camera and Computer Vision Laboratory Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R.O.C. Chapter 14 Analytic Photogrammetry Presented by 王夏果 and Dr. Fuh R94922103@ntu.edu.tw 0937384214

2. DC & CV Lab. NTU CSIE Analytic Photogrammetry Make inferences about : 3D position Orientation Length of the observed 3D object parts in a world reference frame from measurements of one or more 2D- perspective projections of a 3D object

3. DC & CV Lab. NTU CSIE Analytic Photogrammetry (cont.) These inference problems can be construed as nonlinear least-square problems Iteratively linearize the nonlinear functions from an initially given approximate solution

4. DC & CV Lab. NTU CSIE Photogrammetry Provide a collection of methods for determining the position and orientation of cameras and range sensors in the scene and relating camera positions and range measurements to scene coordinates GIS: Geographic Information System GPS: Global Positioning System

5. DC & CV Lab. NTU CSIE

6. DC & CV Lab. NTU CSIE Exterior Orientation Determine position and orientation of camera in absolute coordinate system from projections of calibration points in scene The exterior orientation of the camera is specified by all parameters of camera pose, such as perspectivity center position, optical axis direction.

7. DC & CV Lab. NTU CSIE Exterior Orientation (cont.) Exterior orientation specification: requires 3 rotation angles, 3 translations

8. DC & CV Lab. NTU CSIE

9. DC & CV Lab. NTU CSIE Interior Orientation Determine internal geometry of camera The interior orientation of camera is specified by all the parameters that determines the geometry of 3D rays from measured image coordinates

10. DC & CV Lab. NTU CSIE Interior Orientation (cont.) The parameters of interior orientation relate the geometry of ideal perspective projection to the physics of a camera. Parameters: camera constant, principal point, lens distortion, …

11. DC & CV Lab. NTU CSIE Interior Orientation (cont.) With interior and external orientation, we can complete specify the camera orientation.

12. DC & CV Lab. NTU CSIE Relative Orientation Determine relative position and orientation between 2 cameras from projections of calibration points in scene Calibrate relation between two cameras for stereo Relates coordinate systems of two cameras to each other, not knowing 3D points themselves, only their projections in image

13. DC & CV Lab. NTU CSIE Relative Orientation (cont.) Assume interior orientation of each camera known Specified by 5 parameters: 3 rotation angles, 2 translations

14. DC & CV Lab. NTU CSIE

15. DC & CV Lab. NTU CSIE Absolute Orientation Determine transformation between 2 coordinate systems or position and orientation of range sensor in absolute coordinate system from coordinates of calibration points Convert depth measurements in viewer- centered coordinates to absolute coordinate system for the scene

16. DC & CV Lab. NTU CSIE Absolute Orientation (cont.) Orientation of stereo model in world reference frame Determine scale, 3 translations, 3 rotations Recovery of relation between two coordinate system

17. DC & CV Lab. NTU CSIE

18. DC & CV Lab. NTU CSIE Symbol Definition

19. DC & CV Lab. NTU CSIE Rotation Matrix

20. DC & CV Lab. NTU CSIE Rotation Matrix (cont.)

21. DC & CV Lab. NTU CSIE Rotation Matrix (cont.)

22. DC & CV Lab. NTU CSIE World Frame to Camera Frame (x, y, z)’ in world frame represented by (p, q, s)’ in camera frame:

23. DC & CV Lab. NTU CSIE Pinhole Camera Projection Pinhole camera with image at distance f from camera lens, projection: where f is a camera constant, related to focal length of lens

24. DC & CV Lab. NTU CSIE Principal Point Origin of measurement image plane coordinate Represented by (u 0, v 0 )

25. DC & CV Lab. NTU CSIE Perspective Projection Equations Collinearity equation:

26. DC & CV Lab. NTU CSIE Perspective Projection Equations (cont.) Show that the relationship between the measured 2D-perspective projection coordinates and the 3D coordinates is a nonlinear function of u 0, v 0, x 0, y 0, z 0, ω, ψ, and κ

27. DC & CV Lab. NTU CSIE Take a Break

28. DC & CV Lab. NTU CSIE Nonlinear Least-Square Solutions Noise model:

29. DC & CV Lab. NTU CSIE Nonlinear Least-Square Solutions (cont.) Maximum likelihood solution: β 1, …, β M maximize Prob(α 1, …, α k | β 1, …, β M ) In other words, this solution minimizes least-squares criterion: where

30. DC & CV Lab. NTU CSIE First-Order Taylor Series Expansion First-order Taylor series expansion of g k taken around β t :

31. DC & CV Lab. NTU CSIE First-Order Taylor Series Expansion (cont.)

32. DC & CV Lab. NTU CSIE Exterior Orientation Problem Determine the unknown rotation and translation that put the camera reference frame in the world reference frame.

33. DC & CV Lab. NTU CSIE Exterior Orientation Problem (cont.)

34. DC & CV Lab. NTU CSIE One Camera Exterior Orientation Problem Known: (x n, y n, z n )’ and (u n, v n )’ (u n, v n )’ is the corresponding set of 2D- perspective projections, n = 1, …, N Unknown: (ω,ψ,κ) and (x 0, y 0, z 0 )’

35. DC & CV Lab. NTU CSIE Other Exterior Orientation Problem Camera calibration problem: unknown position of camera in object frame Object pose estimation problem: unknown object position in camera frame Spatial resection problem in photogrammetries: 3D positions from 2D orientation

36. DC & CV Lab. NTU CSIE Nonlinear Transformation For Exterior Orientation

37. DC & CV Lab. NTU CSIE Standard Solution By chain rule,

38. DC & CV Lab. NTU CSIE In matrix form,

39. DC & CV Lab. NTU CSIE

40. DC & CV Lab. NTU CSIE Standard Solution (cont.)

41. DC & CV Lab. NTU CSIE Auxiliary Solution Not iteratively adjust the angles directly Reorganize the calculation such that we iteratively adjust the three auxiliary parameters of a skew symmetric matrix associated with the rotation matrix Then, we determine the adjustment of the angles

42. DC & CV Lab. NTU CSIE

43. DC & CV Lab. NTU CSIE Quaternion Representation From any skew symmetric matrix, we can construct a rotation matrix R by choosing scalar d: R = (dI + S)(dI - S) -1 which guarantees that R’R = I

44. DC & CV Lab. NTU CSIE Quaternion Representation (cont.) Expanding the equation for R: parameters a, b, c, d can be constrained to satisfy a 2 + b 2 + c 2 + d 2 = 1

45. DC & CV Lab. NTU CSIE Quaternion Representation (cont.)

46. DC & CV Lab. NTU CSIE Take a Break

47. DC & CV Lab. NTU CSIE Relative Orientation The transformation from one camera station to another can be represented by a rotation and a translation The relation between the coordinates, r l and r r of a point P can be given by means of a rotation matrix and an offset vector

48. DC & CV Lab. NTU CSIE

49. DC & CV Lab. NTU CSIE Relative Orientation (cont.) Relative orientation is typically with the determination of the position and orientation of one photograph with respect to another, given a set of corresponding image points

50. DC & CV Lab. NTU CSIE Relative Orientation (cont.) Relative orientation specified by five parameters: (y R - y L ), (z R - z L ), (ω R - ω L ), (ψ R - ψ L ), (κ R - κ L ) Assumption: Camera interior orientation known Image positions expressed to identical scale and with respect to principal point

51. DC & CV Lab. NTU CSIE Standard Solution Let Q’ L and Q’ R be the rotation matrices with the exterior orientation of the left and the right image:

52. DC & CV Lab. NTU CSIE Standard Solution (cont.) f R : distance between right image plane and right lens f L : distance between left image plane and left lens From perspective collinearity equation

53. DC & CV Lab. NTU CSIE Standard Solution (cont.) Hence, where

54. DC & CV Lab. NTU CSIE Quaternion Solution Instead of determining the relative orientation of the right image with respect to the left image, we aligns a reference frame having its x-axis along the line from the left image lens to the right image lens

55. DC & CV Lab. NTU CSIE Quaternion Solution (cont.) The relative orientation is then determined by the angles (ω R, ψ R, κ R ), which rotate the right image into this reference frame, and the angles (ω L, ψ L, κ L ), which rotate the left image into this reference frame

56. DC & CV Lab. NTU CSIE Interior Orientation A camera is specified by: Camera constant f: distance between image plane and camera lens Principal point (u p, v p ): intersection of optic axis with image plane in measurement reference frame located on image plane Geometric distortion characteristics of the lens; assuming isotropic around the principal point

57. DC & CV Lab. NTU CSIE

58. DC & CV Lab. NTU CSIE Stereo Optical axes parallel to one another and perpendicular to baseline simple camera geometry for stereo photography

59. DC & CV Lab. NTU CSIE

60. DC & CV Lab. NTU CSIE

61. DC & CV Lab. NTU CSIE Stereo (cont.) Parallax: deplacement in perspective projection by position translation (x, y, z): 3D point position (u L, v L ): perspective projection on left image of stereo pair (u R, v R ): perspective projection on right image of stereo pair b x : baseline length in x-axis

62. DC & CV Lab. NTU CSIE Stereo (cont.)

63. DC & CV Lab. NTU CSIE Stereo (cont.)

64. DC & CV Lab. NTU CSIE

65. DC & CV Lab. NTU CSIE Stereo (cont.) Relation is close to being useless in real-world, because Observed perspective projections are subject to measurement errors so that v L ≠ v R for corresponding points Left and right camera frames may have slightly different orientations When two cameras used, almost always f R ≠ f L

66. DC & CV Lab. NTU CSIE Take a Break

67. DC & CV Lab. NTU CSIE Relationship Between Coordinate System The relationship between two coordinate systems is easy to find if we can measure the coordinates of a number of points in both systems

68. DC & CV Lab. NTU CSIE Relationship Between Coordinate System(cont.) It takes three measurements to tie two coordinate systems together uniquely A single measurement leaves three degrees of freedom motion A second measurement removes all but one degree of freedom Third measurement rigidly attaches two coordinate systems to each other

69. DC & CV Lab. NTU CSIE

70. DC & CV Lab. NTU CSIE 2D-2D Pose Detection Problem Determine from matched points more precise estimate of rotation matrix R and translation t such that y n = Rx n + t, n = 1, …, N Determine R and t that minimize weighted sum of residual errors:

71. DC & CV Lab. NTU CSIE 3D-3D Absolute Orientation We must determine rotation matrix R and translation vector t satisfying Constrained least-squares problem to minimize

72. DC & CV Lab. NTU CSIE 3D-3D Absolute Orientation (cont.) The least-square problem can be modeled by a mechanical system in which corresponding points in the two coordinate systems are attached to each other by means of springs The solution to the least-squares problem corresponds to the equilibrium position of the system, which minimizes the energy stored in the springs

73. DC & CV Lab. NTU CSIE

74. DC & CV Lab. NTU CSIE Robust M-Estimation Least-squares techniques are ideal when random data perturbations or measurement errors are Gaussian distribution We need some robust techniques for nonlinear regression

75. DC & CV Lab. NTU CSIE Robust M-Estimation (cont.) M-Estimator:

76. DC & CV Lab. NTU CSIE Robust M-Estimation (cont.) or

77. DC & CV Lab. NTU CSIE Robust M-Estimation (cont.) ρ: Symmetric Positive-defined function Has unique minimum at zero Chosen to be less increasing than square

78. DC & CV Lab. NTU CSIE Robust M-Estimation (cont.)

79. DC & CV Lab. NTU CSIE Error Propagation If we have the input parameter x 1, …, x N, and random errors Δx 1, …, Δx N, the quantity y depends on input parameters through known function f: y = f(x 1, …, x N ) will become y + Δy= f(x 1 +Δx 1, …, x N +Δx N )

80. DC & CV Lab. NTU CSIE Error Propagation Analysis Determines expected value and variance of y + Δy Known information about Δx 1, …, Δx N : mean and variance

81. DC & CV Lab. NTU CSIE Implicit Form A known function f has the form: f(x 1, …, x N, y) = 0 The quantities (x 1 +Δx 1, …, x N +Δx N ) are observed, and the quantity y + Δy is determined to satisfy f(x 1 +Δx 1, …, x N +Δx N, y + Δy ) =0

82. DC & CV Lab. NTU CSIE Implicit Form: General Case General case: y is not a scalar but a L × 1 vector β x 1, …, x N : are K N × 1 vectors representing true values x 1 +Δx 1, …, x K +Δx K : are K N × 1 vectors representing noisy observed values Δx 1, …, Δx K : random perturbations β: a L × 1 vector representing unknown true parameters

83. DC & CV Lab. NTU CSIE Implicit Form: General Case Noiseless model: With noisy observations, the idealized model:

84. DC & CV Lab. NTU CSIE Summary We have shown how to: Take a nonlinear least-squares problem Linearize it Solve by iteratively solving successive linearized least-squares problems