1. Affine structure from motion
Marc Pollefeys
COMP 256
Some slides and illustrations from J. Ponce, A. Zisserman, R. Hartley, Luc Van Gool, …

2. Last time: Optical Flow
Ixu Ix u
Ixu=- It It
Aperture problem:
two solutions:
- regularize (smoothness prior)
constant over window
(i.e. Lucas-Kanade)
Coarse-to-fine, parametric models, etc…

3. Tentative class schedule
Jan 16/18 -
Introduction
Jan 23/25
Cameras
Radiometry
Jan 30/Feb1
Sources & Shadows
Color
Feb 6/8
Linear filters & edges
Texture
Feb 13/15
Multi-View Geometry
Stereo
Feb 20/22
Optical flow
Project proposals
Feb27/Mar1
Affine SfM
Projective SfM
Mar 6/8
Camera Calibration
Silhouettes and Photoconsistency
Mar 13/15
Springbreak
Mar 20/22
Segmentation
Fitting
Mar 27/29
Prob. Segmentation
Project Update
Apr 3/5
Tracking
Apr 10/12
Object Recognition
Apr 17/19
Range data
Apr 24/26
Final project

4. AFFINE STRUCTURE FROM MOTION
The Affine Structure from Motion Problem
Elements of Affine Geometry
Affine Structure from Motion from two Views
A Geometric Approach
Affine Epipolar Geometry
An Algebraic Approach
Affine Structure from Motion from Multiple Views
From Affine to Euclidean Images
Structure from motion of multiple and deforming object
Reading: Chapter 12.

5. Affine Structure from Motion
Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A, 8: (1990).  Optical Society of America.
Given m pictures of n points, can we recover
the three-dimensional configuration of these points?
the camera configurations?
(structure)
(motion)

6. Orthographic Projection
Parallel Projection

7. Weak-Perspective Projection
Paraperspective Projection

8. The Affine Structure-from-Motion Problem
Given m images of n fixed points P we can write j
Problem: estimate the m 2x4 matrices M and
the n positions P from the mn correspondences p . i j ij
2mn equations in 8m+3n unknowns
Overconstrained problem, that can be solved
using (non-linear) least squares!

9. The Affine Ambiguity of Affine SFM
When the intrinsic and extrinsic parameters are unknown
If M and P are solutions, i j
So are M’ and P’ where i j
and
Q is an affine
transformation.

10. Affine Spaces: (Semi-Formal) Definition

11. 2
Example: R as an Affine Space

12. In General
The notation
is justified by the fact that choosing some origin O in X
allows us to identify the point P with the vector OP.
Warning: P+u and Q-P are defined independently of O!!

13. Barycentric Combinations
Can we add points? R=P+Q
NO!
But, when
we can define
Note:

14. Affine Subspaces

15. Affine Coordinates
Coordinate system for U:
Coordinate system for Y=O+U:
Affine coordinates:
Coordinate system for Y:
Barycentric
coordinates:

16. When do m+1 points define a p-dimensional subspace Y of an n-dimensional affine space X equipped with some coordinate frame basis?
Rank ( D ) = p+1, where
Writing that all minors of size (p+2)x(p+2) of D are equal to zero gives the equations of Y.

17. Affine Transformations
Bijections from X to Y that:
map m-dimensional subspaces of X onto m-dimensional
subspaces of Y;
map parallel subspaces onto parallel subspaces; and
preserve affine (or barycentric) coordinates.
Bijections from X to Y that:
map lines of X onto lines of Y; and
preserve the ratios of signed lengths of
line segments. 3
In E they are combinations of rigid transformations, non-uniform scalings and shears.

18. Affine Transformations II
Given two affine spaces X and Y of dimension m, and two
coordinate frames (A) and (B) for these spaces, there exists
a unique affine transformation mapping (A) onto (B).
Given an affine transformation from X to Y, one can always write:
When coordinate frames have been chosen for X and Y,
this translates into:

19. Affine projections induce affine transformations from planes
onto their images.

20. Affine Shape
Two point sets S and S’ in some affine space X are
affinely equivalent when there exists an affine
transformation y: X X such that X’ = y ( X ).
Affine structure from motion = affine shape recovery.
= recovery of the corresponding motion equivalence classes.

21. Geometric affine scene reconstruction from two images
(Koenderink and Van Doorn, 1991).

22. Affine Structure from Motion
Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A, 8: (1990).  Optical Society of America.
(Koenderink and Van Doorn, 1991)

23. The Affine Epipolar Constraint
Note: the epipolar lines are parallel.

24. Affine Epipolar Geometry

25. The Affine Fundamental Matrix
where

26. An Affine Trick..
Algebraic Scene Reconstruction

27. The Affine Structure of Affine Images
Suppose we observe a scene with m fixed cameras..
The set of all images of
a fixed scene is a 3D
affine space!

28. has rank 4!

29. From Affine to Vectorial Structure
Idea: pick one of the points (or their center of mass)
as the origin.

30. What if we could factorize D? (Tomasi and Kanade, 1992)
Affine SFM is solved!
Singular Value Decomposition
We can take

31. From uncalibrated to calibrated cameras
Weak-perspective camera:
Calibrated camera:
Problem: what is Q ?
Note: Absolute scale cannot be recovered. The Euclidean shape
(defined up to an arbitrary similitude) is recovered.

32. Reconstruction Results (Tomasi and Kanade, 1992)
Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi and
T. Kanade, Proc. IEEE Workshop on Visual Motion (1991).  1991 IEEE.

33. More examples
Tomasi Kanade’92,
Poelman & Kanade’94

34. More examples
Tomasi Kanade’92,
Poelman & Kanade’94

35. More examples
Tomasi Kanade’92,
Poelman & Kanade’94

36. Further Factorization work
Factorization with uncertainty
Factorization for indep. moving objects (now)
Factorization for articulated objects (now)
Factorization for dynamic objects (now)
Perspective factorization (next week)
Factorization with outliers and missing pts.
(Irani & Anandan, IJCV’02)
(Costeira and Kanade ‘94)
(Yan and Pollefeys ‘05)
(Bregler et al. 2000, Brand 2001)
(Sturm & Triggs 1996, …)
(Jacobs ‘97 (affine), Martinek & Pajdla‘01 Aanaes’02 (perspective))

37. Structure from motion of multiple moving objects

38. Structure from motion of multiple moving objects

39. Shape interaction matrix
Shape interaction matrix for articulated objects looses block diagonal structure
Costeira and Kanade’s approach is not usable for articulated bodies
(assumes independent motions)

40. Articulated motion subspaces
Motion subspaces for articulated bodies intersect
(Yan and Pollefeys, CVPR’05)
(Tresadern and Reid, CVPR’05)
Joint (1D intersection)
(joint=origin)
(rank=8-1)
Hinge (2D intersection)
(hinge=z-axis)
(rank=8-2)
Exploit rank constraint to obtain better estimate
Also for non-rigid parts if
(Yan & Pollefeys, 06?)

41. Results Toy truck Student Segmentation Segmentation Intersection

42. Articulated shape and motion factorization
(Yan and Pollefeys, 2006?)
Automated kinematic chain building for articulated & non-rigid obj.
Estimate principal angles between subspaces
Compute affinities based on principal angles
Compute minimum spanning tree

43. Structure from motion of deforming objects
(Bregler et al ’00;
Brand ‘01)
Extend factorization approaches to deal with dynamic shapes

44. Representing dynamic shapes
(fig. M.Brand)
represent dynamic shape as
varying linear combination of basis shapes

45. Projecting dynamic shapes
(figs. M.Brand)
Rewrite:

46. Dynamic image sequences
One image:
(figs. M.Brand)
Multiple images

47. Dynamic SfM factorization?
Problem: find J so that M has proper structure

48. Dynamic SfM factorization
(Bregler et al ’00)
Assumption: SVD preserves order and orientation of
basis shape components

49. Results
(Bregler et al ’00)

50. Dynamic SfM factorization
(Brand ’01)
constraints to be satisfied for M
constraints to be satisfied for M, use to compute J
hard!
(different methods are possible,
not so simple and also not optimal)

51. Non-rigid 3D subspace flow
(Brand ’01)
Same is also possible using optical flow in stead of features, also takes uncertainty into account

52. Results
(Brand ’01)

53. Results
(Brand ’01)

54. Results
(Bregler et al ’01)

55. Next class: Projective structure from motion
Reading: Chapter 13