视觉的三维运动理解 刘允才 上海交通大学 2002 年 11 月 16 日

Understanding 3D Motion from Images Yuncai Liu Shanghai Jiao Tong University November 16, 2002

Understanding motions and structures of 3D scene A basic problem of visual system and computer vision 3-D Motion Non-rigid object motion Articulated object motion Rigid object motion

3D rigid motion understanding Use 3D FeaturesUse 2D Features Monocular ImagesBinocular Images StereoStructured light PointsLinesCorners PointsLines Texture

Minimum Number of Correspondences for Motion Solution with Monocular Images CorrespondencesMin NumberDegeneration 3D-3D points3collinear 3D-3D lines2parallel 2D-3D points3 2D-3D lines3 2D-2D points5quadratic surface 2D-2D lines6 (3 frames) 2D corners1 CC + 2 PC

xo y o X Y P (X, Y) P (x, y, z) L l N P0P0 focus length = f r R Perspective Projection Point: Line: AX + BY + C = 0 is normal of projection plan !

3D motion expression Motion of a rigid object in 3D is usually expressed as a rotation around system origin followed by a translation. Let be a 3D point of an object at time t 1 be a 3D point of an object at time t 2

R be rotation, a 3x3 orthonormal matrix. be translation vector. Then, from time t 1 to t 2 :

Note: There are only 3 independent parameters in R.

Properties of rotation matrix

R is expressed by rotation axis and rotation angle. Let rotation axis be where: n 1 2 +n 2 2 +n 3 2 =1. Rotation is made by a rotating with an angle  around a rotation axis. 

Then the elements of R are:

Express R by 3 rotation angles.

Quaternion form of rotation: Quaternion is a four-element vector, which can be used to express a rotation: Let a rotation around axis by angle 

Quaternion product: A 3D vector can be expressed as a quaternion with scalar part being zero: Pure rotation in 3D: Express the rotation by quaternion:

Motion from 2D PC From 2D images to determine 3D motion At least 5-point correspondences over two- image view are required D translation can only be determined over a scale factor Degeneration case: 3D points are on a quadratic surface.

Assume: Single stationary camera Central projection model Rigid moving object Focus length f = 1, thus

Let 3D motion from of time to of (1) Where From equ (1) (2) Apply to both sides of equ (2) (3)

Apply to both sides of equ (3) (4) Let we define (5) The eq(4) can be rewritten as (6) Note: eq.(6) is a homogeneous scalar equation. is a matrix containing only motion parameters, 8 or more PCs can uniquely determine E, subject to:

R p T RpRp p’ P P’ RPRP o

After matrix E is found,translation can be solved: i.e. (7) can be determined from eq(7) subject to

Once is obtained, rotation R can be obtained by least-square method: (8) Or let

Note, 180 o reflection of motion is still a solution of equ (7) (homogeneous equation). In this case, object is moving behind the camera. To check for a real solution, we apply to both sides of equ (2). Therefore if z > 0, it must hold that Thus if, let

Motion from LC’s from two image frames of a single camera, 3D motion can never be solved over 3 frames, at least 6 LC’s are required motion models for a linear algorithm, 13 LC’s are needed.

Model A

Model B Relation between model A and B:

x o y o X Y L l P0P0 focus length = f N N’ R -1 L’ l’ For a pure rotation, and are collinear. In case of three frames, three collinear vectors collinear, forming a null parallelepiped, i.e.,

o o N’ L’ l l’ N For a pure translation, and lie in a plan that pass though origin and perpendicular to 3D line l. Thus, in three frames, for a general motion, the null parallelepiped condition still holds i.e.,

Now, let we consider model the case B: At time t 1 : (10) At time t 2 : (11)

At time t 3 : (12) From equ. (11) (13) Applying to both sides of equ. (13) and notice that

We get (14) In the same way (15) Eliminate from eqs (14) and (15), we obtain: (16) If we define (17)

F, G, H are 3x3 matrices. Then equ. (16) can be written in compact form: (18) Where Note: equ. (18) is a vector equation containing 3 linear homogeneous equations. And only two of them are linear independent.

Therefore, 13 LC’s over 3 frames are needed to linearly solve for F, G, and H. Let we define: We have After E is found, translation can be solved by: Subject to, let be

Similarly, we define Then And Subject to, let the solution be

In solving for R 12 and R 13, we rather reconstruct E and E ’ for consistence (remember E and E ’ were column by column). are chosen such that Rotations can then be solved by:

Remark: check revered rotations: if Next, we determine relative amplitudes of translations Let substitute them into eqs (17):

when m, n are solved, translations are: Build structures of 3D lines: Direction of 3D line : Position of : Choose sign to make

3D Line Check translation reflection. Evidently, So for and For and

If Else if

Motion from 2D Corners uFirst, the 3D structure of a corner is recovered easily from its image by introducing a new coordinate system; u Then, the rotation matrix R and translation vector T are computed from the recovered 3D corner correspondence; u Finally, it is concluded that one corner and two points correspondences over two views are sufficient to uniquely determine the motion.

1. Representations of 3D and image corners  3D corner:  Image corner: is the normal of the projecting plane of the edge line:

2. Recovering a 3D orthogonal corner from a single view Given the image corner of a 3D orthogonal corner: to recover the 3D structure of the corner: Introduce a new coordinate system such that

z x y o w v u k l p0p0 P0P0 L

Suppose that the edge line of a 3D corner has a slope in the coordinate then the direction of the 3D edge line in the coordinate system is The axis of the coordinate has coordinate in its own coordinate system However, in the, has the same direction with. So, (1)

Similarly, Write in a matrix form:

From Eq.(1), the direction of the correspondent edge line in the For orthogonal corners, (2) Substitute (2) into the above equations, and we can get three equations about the slope value

After and are found, the directions of the edge lines of the 3D corner can be easily computed Thus the 3D corner is reconstructed by Where is the 3D depth of the vertex of the corner, which cannot be determined from a single view.

3. Determining the rotation matrix R From the image corner correspondence, To reconstruct

The directions of the edge lines of the corner at time and are related by are orthogonal unit vectors Remark: since we get two sets of slope values, that means we recover two 3D corners from one image corner, so four rotation matrix can be computed. Therefore, additional information is needed to determine a unique solution.

4. Determining the translation vector T Since the rotation matrix R has been computed, we can eliminate the rotation motion from the whole motion. Then there is only the motion of translation. Intermediate time Time t1 Time t2

Following, we suppose there is only translation motion between time t1 and t2. Remark: It is impossible to uniquely determine a translation from a single corner correspondence over two views of images. The rank of the coefficient matrix of the equations for translation is always less than 3. Proof: A maximum of 4 equations can be derived from a single corner over two views of images: the three equations from edge lines, the other one from the vertex. Edge lines of the corner satisfy the equations : (3)

Equations (3) and (4) are four linear equations about the unknown T, but they are not independent. The rank of the coefficient matrix is only 2. Another image point not lying in any of the three edge lines is needed to determine the translation. the other equation for the vertex of the corner is: (4) where

5. Getting a unique solution Images over two frames Four rotation matrix R A corner correspondence One translation T for each R a corner and a nonsingular point correspondence Unique solution R and T Another nonsingular point correspondence

Uniqueness  If a 3D motion is a pure rotation, an orthogonal corner correspondence and a nonsingular point correspondence over two frames can uniquely determine the motion.  If a 3D motion is a pure translation, an orthogonal corner correspondence and a nonsingular point correspondence over two frames can uniquely determine the motion.  If a 3D motion is a rotation followed by a translation, an orthogonal corner correspondence and two point correspondences can uniquely determine the motion.

6. Motion estimation from a corner with known space angles The process is the same as that in the orthogonal corner case. The only difference is : Orthogonal corner Corner with angles:

Experiment Result: Rotation: ( , , ; 33 o 54”) Translation: ( )

Thanks