1. Two-View Motion Segmentation by Exploiting the Rank and Geometry of the Multibody Fundamental Matrix
René Vidal and Xiaodong Fan
Center for Imaging Science
Johns Hopkins University
Add d15.avi

2. Structure and motion recovery problem
Input: Corresponding points in multiple images
Output: camera motion, scene structure, camera calibration
Structure = 3D surface
Motion = camera position and orientation
Rene Vidal

3. The fundamental matrix
Input: point correspondences
Output: rigid motion
Rotation:
Translation:
Epipolar constraint
Estimate fundamental matrix F linearly
Eight-point algorithm
Project onto the essential manifold for noise reduction
Recover (R,T) from F
Write Sym(F_1,…., F_n).
Rene Vidal

4. Projection onto the essential manifold
Characterizing essential manifold
F Extend to multiple motions!
Projection – minimizing the Frobenious norm:
Rene Vidal

5. 3-D motion segmentation problem
Given a set of point correspondences in multiple views, determine
Number of motion models
Motion model: affine
Segmentation: model to which each pixel belongs
Mathematics of the problem depends on
Number of frames (2, 3, multiple)
Projection model (affine, perspective)
Motion model (affine, translational, planar motion, rigid motion)
3-D structure (planar or not)
Rene Vidal

6. Our approach to motion segmentation
Estimation of multiple motion models equivalent to estimation of one multibody motion model
Eliminate feature clustering: multiplication
Estimate a single multibody motion model: polynomial fitting
Segment multibody motion model: polynomial differentiation
chicken-and-egg
Rene Vidal

7. A unified approach to motion segmentation
Applies to most motion models in computer vision
All motion models can be segmented algebraically by
Fitting multibody model: real or complex polynomial to all data
Fitting individual model: differentiate polynomial at a data point
Rene Vidal

8. Multibody epipolar constraint
Rotation:
Translation:
Epipolar constraint
Multiple motions
Multibody epipolar constraint
Satisfied by ALL points regardless of segmentation
Segmentation is algebraically eliminated!!!
Rene Vidal

9. Multibody fundamental matrix
Lifting
Embedding
Bilinear on embedded data!
Veronese map (polynomial embedding)
Multibody fundamental matrix
Rene Vidal

10. Multibody Fundamental Matrix
n-body motion
1-body motion
Rene Vidal

11. Epipoles in the nullspace of F
Embedded epipoles are in the null space of the multibody fundamental matrix
Are all vectors in Null(F) embedded epipoles, or their linear combinations?
Are embedded epipoles linearly independent?
F The dimension of Null(F)
Rene Vidal

12. Rank constraint of F 1-body motion 2-body motion In general, we prove:
# of distinct epipoles
Key observation: repeated epipoles will enlarge the null(F) !
time each eipipole repeates
Rene Vidal

13. Dimension of Null(F) Let be
Different epipoles (up to scales), each of which repeats times
Define as the span of the l-th order derivatives of
Rene Vidal

14. Dimension of Null(F) Lemma 1: Lemma 2: when Lemma 3:
Rene Vidal

15. Projection onto the essential manifold
Characterizing essential manifold
F Extend to multiple motions!
Projection – minimizing the Frobenious norm:
Rene Vidal

16. Multibody essential manifold with common rotations
n-body motion
1-body motion
Rene Vidal

17. SVD of When the number of motions is odd When there are two motions
Theorem: The singular values of a skew-symmetric matrix are
When there are two motions
Theorem: The singular values satisfies
Rene Vidal

18. Projection onto the multibody essential manifold
Based on the proposed projection theorem, the projection process is:
Linearly estimate F
Compute SVD
Replace by the “desired” singular values
Satisfy the multi-body essential manifold constraint
Rene Vidal

19. Compute the “desired” singular values
Using Lagrangian multiplier method to solve the constraint optimization problem
Two motions
with a common rotation
Odd number of motions
Rene Vidal

20. Multibody 8-point algorithm
Multibody epipolar constraint
Multibody epipolar transfer
Multibody epipole
Fundamental matrices
Given
rank condition n
linear system F
Lifting
Embedding
Sym(F1,,,F_n)
This can be solved using a collection of GPCA’s
Rene Vidal

21. Experiments Compare three methods:
GPCA (Vidal and Ma, 2004): specifically designed for the purely translational motions.
Multibody epipolar constraint without projection (MEC-noprojection): using the proposed approach without projecting the estimated multibody fundamental matrix F onto the multibody essential manifold.
Multibody epipolar constraint with projection (MEC-projection): project the estimated multibody fundamental matrix F onto the multibody essential manifold.
Rene Vidal

22. Experimental results on synthesized data
Error in the estimation of translational component of motion (in degree) as a function of noise in the image points (std in pixels) for two pure translational motions (left column); and as a function of rotation angle (in degree) for two independent motions with a common rotation (right column).
(a) in translation
Rene Vidal

23. Experimental results on synthesized data
Error in the estimation of rotational component of motion (in degree) as a function of noise in the image points (std in pixels) for two pure translational motions (left column); and as a function of rotation angle (in degree) for two independent motions with a common rotation (right column).
(b) in rotation
Rene Vidal

24. Experimental results on synthesized data
Error in the segmentation (in percentage) as a function of noise in the image points (std in pixels) for two pure translational motions (left column); and as a function of rotation angle (in degree) for two independent motions with a common rotation (right column).
(c) in segmentation
Rene Vidal

25. Experimental results on real images
Experiment setup:
The original sequence contains three independent motions, the first two of which are translational.
Point correspondences,
Compute the rotational component R of the third motion.
Rotate the feature points in the first frame that belong to the first motion,
Then undergo a two-body motion with a common rotation.
Rene Vidal

26. Experimental results on real images
Percentage of mis-segmentation on a real sequence with two independent motions (with a common rotation) for different pairs of frames. The x-axis indicates the index of frame pairs.
Rene Vidal

27. Conclusions Unified algebraic approach to motion segmentation
Fit a polynomial to all image data
Differentiate the polynomial to obtain motion parameters
Applies to most motion models in vision
Two views
2-D: translational, similarity, affine
3-D: translational, fundamental matrices, homographies
Three views
Multibody trifocal tensor
Multiple views
Affine cameras
Two perspective views
Rank of multibody fundamental matrices
Geometry of multibody fundamental matrices
Rene Vidal