1. Multiple View Geometry
Marc Pollefeys
COMP 256

2. Last class Gaussian pyramid Laplacian pyramid Gabor Fourier filters
transform

3. Histograms : co-occurrence matrix
Not last class …
Histograms : co-occurrence matrix

4. Texture synthesis [Zalesny & Van Gool 2000]
2 analysis iterations
6 analysis iterations
9 analysis iterations

5. View-dependent texture synthesis [Zalesny & Van Gool 2000]

6. Efros & Leung ’99 p Assuming Markov property, compute P(p|N(p))
non-parametric
sampling
Synthesizing a pixel
Input image
Assuming Markov property, compute P(p|N(p))
Building explicit probability tables infeasible
Instead, let’s search the input image for all similar neighborhoods — that’s our histogram for p
To synthesize p, just pick one match at random

7. Efros & Leung ’99 extendedp B
Idea: unit of synthesis = block
Exactly the same but now we want P(B|N(B))
Much faster: synthesize all pixels in a block at once
Not the same as multi-scale!
Synthesizing a block
non-parametric
sampling
Input image
Observation: neighbor pixels are highly correlated

8. constrained by overlap
block
Input texture B1 B2
Neighboring blocks
constrained by overlap B1 B2
Minimal error
boundary cut B1 B2
Random placement
of blocks

9. Minimal error boundary
overlapping blocks
vertical boundary _ = 2
overlap error
min. error boundary

13. Why do we see more flowers in the distance?
[Leung & Malik CVPR97]
Perpendicular textures

14. Shape-from-texture

15. Tentative class schedule
Jan 16/18 -
Introduction
Jan 23/25
Cameras
Radiometry
Jan 30/Feb1
Sources & Shadows
Color
Feb 6/8
Linear filters & edges
Texture
Feb 13/15
Multi-View Geometry
Stereo
Feb 20/22
Optical flow
Project proposals
Feb27/Mar1
Affine SfM
Projective SfM
Mar 6/8
Camera Calibration
Silhouettes and Photoconsistency
Mar 13/15
Springbreak
Mar 20/22
Segmentation
Fitting
Mar 27/29
Prob. Segmentation
Project Update
Apr 3/5
Tracking
Apr 10/12
Object Recognition
Apr 17/19
Range data
Apr 24/26
Final project

16. THE GEOMETRY OF MULTIPLE VIEWS
Epipolar Geometry
The Essential Matrix
The Fundamental Matrix
The Trifocal Tensor
The Quadrifocal Tensor
Reading: Chapter 10.

17. Epipolar Geometry
Epipolar Plane
Baseline
Epipoles
Epipolar Lines

18. Potential matches for p have to lie on the corresponding
Epipolar Constraint
Potential matches for p have to lie on the corresponding
epipolar line l’.
Potential matches for p’ have to lie on the corresponding
epipolar line l.

19. Epipolar Constraint: Calibrated Case
Essential Matrix
(Longuet-Higgins, 1981)

20. E p’ is the epipolar line associated with p’.
Properties of the Essential Matrix T
E p’ is the epipolar line associated with p’.
ETp is the epipolar line associated with p.
E e’=0 and ETe=0.
E is singular.
E has two equal non-zero singular values
(Huang and Faugeras, 1989). T

21. Epipolar Constraint: Small Motions
To First-Order:
Pure translation:
Focus of Expansion

22. Epipolar Constraint: Uncalibrated Case
Fundamental Matrix
(Faugeras and Luong, 1992)

23. Properties of the Fundamental Matrix
F p’ is the epipolar line associated with p’.
FT p is the epipolar line associated with p.
F e’=0 and FT e=0.
F is singular. T T

24. The Eight-Point Algorithm (Longuet-Higgins, 1981)
|F| =1.
Minimize:
under the constraint 2

25. Non-Linear Least-Squares Approach
(Luong et al., 1993)
Minimize
with respect to the coefficients of F , using an
appropriate rank-2 parameterization.

26. Problem with eight-point algorithm
linear least-squares:
unit norm vector F yielding smallest residual
What happens when there is noise?

27. The Normalized Eight-Point Algorithm (Hartley, 1995)
Center the image data at the origin, and scale it so the
mean squared distance between the origin and the data
points is 2 pixels: q = T p , q’ = T’ p’.
Use the eight-point algorithm to compute F from the
points q and q’ .
Enforce the rank-2 constraint.
Output T F T’. i i i i i i T

28. Epipolar geometry example

29. courtesy of Andrew Zisserman
Example: converging cameras
courtesy of Andrew Zisserman

30. Example: motion parallel with image plane
(simple for stereo  rectification)
courtesy of Andrew Zisserman

31. courtesy of Andrew Zisserman
Example: forward motion e’ e
courtesy of Andrew Zisserman

32. courtesy of Andrew Zisserman
Fundamental matrix for pure translation
auto-epipolar
courtesy of Andrew Zisserman

33. courtesy of Andrew Zisserman
Fundamental matrix for pure translation
courtesy of Andrew Zisserman

34. Trinocular Epipolar Constraints
These constraints are not independent!

35. Trinocular Epipolar Constraints: Transfer
Given p and p , p can be computed
as the solution of linear equations. 1 2 3

36. Trinocular Epipolar Constraints: Transfer
problem for epipolar transfer in trifocal plane!
There must be more to trifocal geometry…
image from Hartley and Zisserman

37. Trifocal Constraints

38. Trifocal Constraints Calibrated Case All 3x3 minors must be zero!
Trifocal Tensor

39. Trifocal Constraints
Uncalibrated Case
Trifocal Tensor

40. Trifocal Constraints: 3 Points
Pick any two lines l and l through p and p . 2 3 2 3
Do it again.
T( p , p , p )=0 1 2 3

41. For any matching epipolar lines, l G l = 0.
Properties of the Trifocal Tensor
For any matching epipolar lines, l G l = 0.
The matrices G are singular.
They satisfy 8 independent constraints in the
uncalibrated case (Faugeras and Mourrain, 1995). T i 2 1 3 i 1
Estimating the Trifocal Tensor
Ignore the non-linear constraints and use linear least-squares
Impose the constraints a posteriori.

42. For any matching epipolar lines, l G l = 0.2 1 3
The backprojections of the two lines do not define a line!

43. courtesy of Andrew Zisserman
Trifocal
Tensor
Example
108 putative matches
18 outliers
(26 samples)
88 inliers
95 final inliers
(0.43)
(0.23)
(0.19)
courtesy of Andrew Zisserman

44. Trifocal Tensor Example
additional line matches
images courtesy of Andrew Zisserman

45. Transfer: trifocal transfer
(using tensor notation)
doesn’t work if l’=epipolar line
image courtesy of Hartley and Zisserman

46. Image warping using T(1,2,N)
(Avidan and Shashua `97)

47. Multiple Views (Faugeras and Mourrain, 1995)

48. Two Views
Epipolar Constraint

49. Three Views
Trifocal Constraint

50. Four Views
Quadrifocal Constraint
(Triggs, 1995)

51. Geometrically, the four rays must intersect in P..

52. Quadrifocal Tensor
and Lines

53. Quadrifocal tensor determinant is multilinear
thus linear in coefficients of lines !
There must exist a tensor with 81 coefficients containing all possible combination of x,y,w coefficients for all 4 images: the quadrifocal tensor

54. Scale-Restraint Condition from Photogrammetry

55. from perspective to omnidirectional cameras
3 constraints allow to reconstruct 3D point
perspective camera
(2 constraints / feature)
more constraints also tell something about cameras
l=(y,-x)
(x,y)
(0,0)
multilinear constraints known as epipolar, trifocal and quadrifocal constraints
radial camera (uncalibrated)
(1 constraints / feature)

56. Radial quadrifocal tensor
(x,y)
Radial quadrifocal tensor
Linearly compute radial quadrifocal tensor Qijkl from 15 pts in 4 views
Reconstruct 3D scene and use it for calibration
(2x2x2x2 tensor)
Not easy for real data, hard to avoid degenerate cases (e.g. 3 optical axes intersect in single point). However, degenerate case leads to simpler 3 view algorithm for pure rotation
Radial trifocal tensor Tijk from 7 points in 3 views
Reconstruct 2D panorama and use it for calibration
(2x2x2 tensor)

57. Non-parametric distortion calibration
(Thirthala and Pollefeys, ICCV’05)
Models fish-eye lenses, cata-dioptric systems, etc.
angle
normalized radius

58. Non-parametric distortion calibration
(Thirthala and Pollefeys, ICCV’05)
Models fish-eye lenses, cata-dioptric systems, etc.
90o
angle
normalized radius

59. Next class: Stereo (x´,y´)=(x+D(x,y),y) F&P Chapter 11 image I´(x´,y´)
Disparity map D(x,y)
image I´(x´,y´)
(x´,y´)=(x+D(x,y),y)
F&P
Chapter 11