1. Advanced Computer Vision
Chapter 7
STRUCTURE FROM MOTION
Presented by
Prof. Chiou-Shann Fuh & Jia-Yau Shiau (蕭家堯)
Structure from Motion

2. Advanced Computer Vision
Structure from Motion

3. What Is Structure from Motion?
To be brief
A method for creating 3D models from 2D pictures of an object.
Structure from Motion

4. Example
Picture 1
Picture 2
Structure from Motion

5. Example (cont). 3D model created from the two images
Structure from Motion

6. What Is Structure from Motion?
To be brief
A method for creating 3D models from 2D pictures of an object.
To be formal
An automatic recovery of camera motion and scene structure from two or more images.
A self calibration technique and called automatic camera tracking or match-moving.
Structure from Motion

7. What Is Structure from Motion?
Let
K, R, t denote the parameters of camera
X denotes the 3D position of the object
x denotes the 2D position of the object
Then
Triangulation: estimate X with known x, K, R, t
Camera Calibration: estimate K, R, t with known x, X
Structure from Motion: estimate K, R, t, X with known x
Structure from Motion

8. What Is Structure from Motion?
2D feature tracking
3D estimation
optimization
geometry fitting
Essential matrix
Triangulation
Bundle adjust
Structure from Motion

9. Agenda Camera Parameters Triangulation Two-frame Structure from Motion
Factorization
Bundle Adjustment
Constrained Structure and Motion
Structure from Motion

10. Camera Parameters Pinhole camera model Canon EF 16-35mm f/2.8L II USM
簡單來說，給定camera的pose，計算物體在3D中的位子。
Canon EF 16-35mm
f/2.8L II USM
Pinhole camera model
Structure from Motion

11. Camera Parameters Structure from Motion
簡單來說，給定camera的pose，計算物體在3D中的位子。
Structure from Motion

12. Camera Parameters Intrinsic matric K R t Structure from MotionK R t
簡單來說，給定camera的pose，計算物體在3D中的位子。
Intrinsic matric
Structure from Motion

13. What Is Structure from Motion?
2D feature tracking
3D estimation
optimization
geometry fitting
Triangulation
Structure from Motion

14. 7.1Triangulation
A problem of estimating a point’s 3D location when it is seen from multiple cameras is known as triangulation.
It is a converse of pose estimation problem.
Given projection matrices, 3D points can be computed from their measured image positions in two or more views.
簡單來說，給定camera的pose，計算物體在3D中的位子。
Structure from Motion

15. 7.1Triangulation
Structure from Motion

16. 7.1Triangulation
Find the 3D point p that lies closest to all of the 3D rays corresponding to the 2D matching feature locations {xj} observed by cameras
{Pj = Kj [Rj | tj] }
tj = -Rjcj
cj is the jth camera center.
KRT 相機參數
Structure from Motion

17. 7.1Triangulation
Suppose camera j
Structure from Motion

18. 7.1Triangulation p c d
Goal: minimized the squared distance between p and q
Structure from Motion

19. Method 1 (cont.)
Structure from Motion

20. Method 1 (cont.) The squared distance between p and qj is
The optimal value for p, which lies closest to all of the rays, can be computed as a regular least square problem by summing over all the rj2 and finding the optimal value of p,
Structure from Motion

21. Method 2 (cont.)
Structure from Motion

22. Method 2 (cont.)
Structure from Motion

23. Example: Triangulation
Structure from Motion

24. What Is Structure from Motion?
2D feature tracking
3D estimation
optimization
geometry fitting
Structure from Motion

25. Epipolar Geometry
Structure from Motion

26. Epipolar Geometry
Epipolar plane
Structure from Motion

27. Epipolar Geometry
Epipolar line
對應點找
Structure from Motion

28. Epipolar Geometry Baseline Structure from Motion
所有Epipolar plane都會通過Baseline
Baseline
Structure from Motion

29. Epipolar Geometry Epipolar pole Structure from Motion
所有Epipolar line都會通過pole
Epipolar pole
Structure from Motion

30. Epipolar Geometry
Structure from Motion

31. Two-Frame Structure from Motion (cont).
由兩個不同的角度來看同一個物件，其所構成的投影幾何就稱為 Epipolar Geometry。 
Warning of Mathematic
Structure from Motion

32. Two-Frame Structure from Motion (cont).
Structure from Motion

33. Two-Frame Structure from Motion (cont).
Structure from Motion

34. Two-Frame Structure from Motion (cont).
Cross product (外積) 得到垂直向量
Dot product (內積) 垂直得到0
Structure from Motion

35. Two-Frame Structure from Motion (cont).
Cross product (外積) 得到垂直向量
Dot product (內積) 垂直得到0
Structure from Motion

36. Two-Frame Structure from Motion (cont).
Cross product with t
Cross product (外積) 得到垂直向量
Dot product (內積) 垂直得到0
Structure from Motion

37. Two-Frame Structure from Motion (cont).
Cross product (外積) 得到垂直向量
Dot product (內積) 垂直得到0
Structure from Motion

38. Two-Frame Structure from Motion (cont).
Two-Frame Structure from Motion (cont).
Fundamental matrix 包含相機參數
Structure from Motion

39. Two-Frame Structure from Motion (cont).
Structure from Motion

40. Two-Frame Structure from Motion (cont).
In the matrix, some entries are product of image measurement such as xi0yi1 and others are direct image measurements (even identity).
Given N>8 such equation, we can compute an estimate for the entire E.
Structure from Motion

41. Two-Frame Structure from Motion (cont).
Two-Frame Structure from Motion (cont).
Structure from Motion

42. Singular Value Decomposition (Brief)
Singular Value Decomposition (Brief)
Structure from Motion

43. Two-Frame Structure from Motion (cont).
If the measurements have noise, the terms that are product of measurement have their noise amplified by the other element in the product, which lead to poor scaling.
In order to deal with this, a suggestion is that the point coordinate should be translated and scaled so that their centroid lies at the origin, variance is unity; i.e.
Structure from Motion

44. Two-Frame Structure from Motion (cont).
such that
and
n= number of points.
Once the essential matrix has been computed from the transformed coordinates; the original essential matrix E can be recovered as
Structure from Motion

45. Two-Frame Structure from Motion (cont).
When the essential matrix has been recovered, the direction of the translation vector t can be estimated.
The absolute distance between two cameras can never be recovered from pure image measurement alone.
Ground control points in Photogrammetry:
knowledge about absolute camera, point positions or distances.
Required to establish the final scale, position and orientation.
Structure from Motion

46. Two-Frame Structure from Motion (cont).
Structure from Motion

47. Pure Translation
Figure 7.9: Pure translation camera motion results in visual motion where all the points move towards (or away from) a common focus of expansion (FOE).
擴展焦點（focus of expansion）
Structure from Motion

48. Focus of Expansion https://www.youtube.com/watch?v=GIUDAZLfYhY
Structure from Motion

49. Pure Translation (cont).
Known rotation:
The resulting essential matrix E is (in the noise-free case) skew symmetric and can estimate more directly by setting eij= -eji and eii = 0.
Two-point parallax now suffices to estimate the FOE.
Structure from Motion

50. Pure Translation (cont).
A more direct derivation of FOE estimates can be obtained by minimizing the triple product.
which is equivalent to finding null space for the set of equations
Structure from Motion

51. Pure Translation (cont).
In a situation where large number of points at infinity are available, (when the camera motion is small compared to distant objects, this suggests a strategy.
Pick a pair of points to estimate a rotation, hoping that both of the points lie at infinity (very far from camera).
Then compute FOE and check whether residual error is small and whether the motions towards or away from the epipoler (FOE) are all in the same direction.
Structure from Motion

52. Pure Rotation
This results in a degenerate estimate of the essential matrix E and the translation direction.
If we consider that the rotation matrix is known, the estimates for the FOE will be degenerate, since
and hence
is degenerate.
Structure from Motion

53. Example: Structure from Motion
CSE 576, Spring 2008
Structure from Motion

54. Self-calibration https://www.youtube.com/watch?v=p1kCR1i2nF0
Auto-calibration is developed for covering a projective reconstruction into a metric one, which is equivalent to recovering the unknown calibration matrix Kj associated with each image.
In the presence of additional information about scene, different methods can be applied.
If there are parallel lines in the scene, three or more vanishing points, which are the images of points at infinity, can be used to establish homography for the plane at infinity, from which focal length and rotation can be recovered.
Structure from Motion

55. Self-calibration (cont).
In the absence of external information:
consider all sets of camera matrices Pj = Kj[ Rj | tj ]
projecting world coordinates pi=(Xi,Yi,Zi,Wi) into screen coordinates xij ~ Pjpi.
Consider transforming the 3D scene {pi} through an arbitrary projective transformation yielding a new model consisting of points
Structure from Motion

56. Self-calibration (cont).
A technique that can recover the focal lengths (f0,f1) of both images from fundamental matrix F in a two-frame reconstruction.
Assume that camera has zero skew, a known aspect ratio, and known optical center.
Most cameras have square pixels and an optical center near middle of image and are likely to deviate from simple camera model due to radial distortion
Problem occurs when images have been cropped off-center.
Structure from Motion

57. Application: View Morphing
Application of basic two-frame structure from motion.
Also known as view interpolation.
Used to generate a smooth 3D animation from one view of a 3D scene to another.
To create such a transition: smoothly interpolate camera matrices, i.e., camera position, orientation, focal lengths. More effect is obtained by easing in and easing out camera parameters.
Structure from Motion

58. Application: View Morphing
Triangulate set of matched feature points in each image .
To generate in-between frames: establish full set of 3D correspondences or 3D models for each reference view.
As the 3D points are re-projected into their intermediate views, pixels can be mapped from their original source images to their new views using affine or projective mapping.
The final image then composited using linear blend of the two reference images as with usual morphing.
Structure from Motion

59. Factorization
When processing video sequences, we often get extended feature track from which it is possible to recover the structure and motion using a process called factorization.
Structure from Motion

60. Factorization (cont.)
Figure 7.10: 3D reconstruction of a rotating ping pong ball using factorization (Tomasi and Kanade 1992) : (a) sample image with tracked features overlaid; (b)sub-sampled feature motion stream ; (c) two views of the reconstructed 3D model.
Structure from Motion

61. Factorization (cont.)
A disadvantage is they require a complete set of tracks i.e., each point must be visible in each frame, in order for the factorization approach to work.
Structure from Motion

62. Factorization (cont.)
2x1
2x3
3x1
2x1
Structure from Motion

63. Factorization (cont.)
Warning of Mathematic
Structure from Motion

64. Factorization (cont.) shifting Suppose we have n feature in one image:
Suppose we have n feature in one image:
Structure from Motion

65. Factorization (cont.)
Now, we have m images:
Structure from Motion

66. Factorization (cont.) Structure from Motion
Factorization (cont.)
Structure from Motion

67. Perspective and Projective Factorization
Factorization disadvantage is that it cannot deal with perspective cameras.
Perform an initial affine (e.g.,orthographic) reconstruction and to then correct for the perspective effects in an iterative manner.
Structure from Motion

68. What Is Structure from Motion?
2D feature tracking
3D estimation
optimization
geometry fitting
Structure from Motion

69. Bundle Adjustment
The most accurate way to recover structure from motion is to perform robust nonlinear minimization of the measurement (re-projection) errors, which is known as photogrammetry (in computer vision) communities as bundle adjustment.
Our feature location measurement xij now depends only on the point (track index) i but also on the camera pose index j.
xij = f(pi,Rj,cj,Kj)
3D point positions pi are also updated simultaneously
Structure from Motion

70. Bundle Adjustment minimized the error between
xij = f(pi,Rj,cj,Kj) and real measurement
Structure from Motion

71. Levenberg-Marquardt Mix of Newton’s method and steepest descent method
Structure from Motion

72. Exploiting Sparsity
Large bundle adjustment problems, such as those involving 3D scenes from thousands of Internet photographs can require solving non-linear least square problems with millions of measurements
Structure from motion is bipartite problem in structure and motion.
Each feature point xij in a given image depends on one 3D point position pi and 3D camera pose (Rj,cj).
Structure from Motion

73. Exploiting Sparsity
Structure from Motion

74. Uncertainty and Ambiguity
Structure from motion involves the estimation of so many highly coupled parameters, often with no known “ground truth” components.
The estimates produces by structure from motion algorithm can often exhibit large amounts of uncertainty .
Example: bas-relief ambiguity, which makes it hard to simultaneously estimate 3D depth of scene and the amount of camera motion.
Structure from Motion

75. Reconstruction from Internet Photos
Widely used application of structure from motion: the reconstruction of 3D objects and scenes from video sequences and collection of images.
Before structure from motion comparison can begin, it is first necessary to establish sparse correspondences between different pairs of images and to then link such correspondences into feature track, which associates individual 2D image feature with global 3D points .
Structure from Motion

76. Reconstruction from Internet Photos (cont).
For the reconstruction process, it is important to select good pair of images and a significant amount of out-of-plane parallax to ensure that a stable reconstruction can be obtained .
Structure from Motion

77. Reconstruction from Internet Photos (cont).
Figure 7.15: Incremental structure from motion: Starting with an initial
two-frame reconstruction of Trevi Fountain, batches of images are
added using pose estimation, and their positions (along with 3D
model) are refined using bundle adjustment
Structure from Motion

78. Reconstruction from Internet Photos (cont).
Figure7.16: 3D reconstruction produced by the incremental structure from motion
algorithm. (a) cameras and point cloud from Trafalgar Square; (b) cameras
and points overlaid on an image from the Great Wall of China.; (c) overhead
view of reconstruction of Old Town Square in Prague registered to an
aerial photograph.
Structure from Motion

79. Constrained Structure and Motion
If the object of interest is rotating around a fixed but unknown axis, specialized techniques can be used to recover this motion.
In other situation, the camera itself may be moving in a fixed arc around some center of rotation.
Specialized capture steps, such as mobile stereo camera rings or moving vehicles equipped with multiple fixed cameras, can also take advantage of the knowledge that individual cameras are mostly fixed with respect to the capture rig.
Structure from Motion

80. Constrained Structure and Motion (cont).
Line-based technique:
Pairwise epipolar geometry cannot be recovered from line matches alone, even if the cameras are calibrated.
Consider projecting the set of lines in each image into a set of 3D planes in space. You can move the two cameras around into any configuration and still obtain a valid reconstruction for 3D lines.
Structure from Motion

81. Constrained Structure and Motion (cont).
When lines are visible in three or more views, the trifocal tensor can be used to transfer lines from one pair of image to another.
The trifocal tensor can also be computed on the basis line matches alone.
For triples of images, the trifocal tensor is used to verify that the lines are in geometric correspondence before evaluating the correlations between line segments.
Structure from Motion

82. Constrained Structure and Motion (cont).
Figure 7.18: Two images of toy house along with their matched 3D
line segments.
Structure from Motion

83. End of Today
End
Structure from Motion