1. Multiple View Geometry in Computer Vision
Marc Pollefeys
Comp

2. Multiple View GeometryA a a c c b
f(a,b,c)=0 b
(a,b) A
(reconstruction)
(a,b,c) (a,b,c)
(calibration)
(a,b) c
(transfer)

3. Course objectives
To understand the geometric relations between multiple views of scenes.
To understand the general principles of parameter estimation.
To be able to compute scene and camera properties from real world images using state-of-the-art algorithms.

4. Relation to other vision/image courses
Focuses on geometric aspects
No image processing
Comp 254: Image Processing an Analysis
Mostly orthogonal to this course, complementary
Comp 256: Computer Vision (fall 2003)
Will be much broader, based on new book:
“Computer Vision: a modern approach”
David Forsyth and Jean Ponce

5. Material Textbook: Multiple View Geometry in Computer Vision
by Richard Hartley and Andrew Zisserman
Cambridge University Press
Alternative book:
The Geometry from Multiple Images
by Olivier Faugeras and Quan-Tuan Luong
MIT Press
On-line tutorial:

6. Learning approach
read the relevant chapters of the books and/or reading assignements before the course.  
In the course the material will then be covered in detail and motivated with real world examples and applications. 
Small hands-on assignements will be provided to give students a "feel" of the practical aspects.
Students will also read and present some seminal papers to provide a complementary view on some of the covered topics.
Finally, there will also be a project where students will implement an algorithm or approach using concepts covered by the course.   
Grade distribution
Class participation: 20%
Hands-on assignments: 10%
Paper presentation: 10%
Implementation assignment/project: 40%
Final: 20%

7. Applications MatchMoving Compute camera motion from video
(to register real an virtual object motion)

8. Applications
3D modeling

9. Content
Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation.
Single View: Camera model, Calibration, Single View Geometry.
Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies.
Three Views: Trifocal Tensor, Computing T.
More Views: N-Linearities, Multiple view reconstruction, Bundle adjustment, auto-calibration, Dynamic SfM, Cheirality, Duality

10. Multiple View Geometry course schedule (tentative)
Jan. 7, 9
Intro & motivation
Projective 2D Geometry
Jan. 14, 16
(no course)
Jan. 21, 23
Projective 3D Geometry
Parameter Estimation
Jan. 28, 30
Algorithm Evaluation
Feb. 4, 6
Camera Models
Camera Calibration
Feb. 11, 13
Single View Geometry
Epipolar Geometry
Feb. 18, 20
3D reconstruction
Fund. Matrix Comp.
Feb. 25, 27
Structure Comp.
Planes & Homographies
Mar. 4, 6
Trifocal Tensor
Three View Reconstruction
Mar. 18, 20
Multiple View Geometry
MultipleView Reconstruction
Mar. 25, 27
Bundle adjustment
Papers
Apr. 1, 3
Auto-Calibration
Apr. 8, 10
Dynamic SfM
Apr. 15, 17
Cheirality
Apr. 22, 24
Duality
Project Demos

11. Fast Forward!
Quick overview of what is coming…

12. Background
La reproduction interdite (Reproduction Prohibited), 1937, René Magritte.

13. Projective 2D Geometry Points, lines & conics Transformations
Cross-ratio and invariants

14. Projective 3D Geometry Points, lines, planes and quadrics
Transformations
П∞, ω∞ and Ω ∞

15. Estimation
How to compute a geometric relation from correspondences, e.g. 2D trafo
Linear (normalized), non-linear and Maximum Likelihood Estimation
Robust (RANSAC)

16. Evaluation and error analysis
How good are the results we get
Bounds on performance
Covariance propagation
& Monte-Carlo estimation
error
residual

17. Single-View Geometry
The Cyclops, c. 1914, Odilon Redon

18. Camera Models Mostly pinhole camera model
but also affine cameras, pushbroom camera, …

19. Camera Calibration Compute P given (m,M) Radial distortion
(normalized) linear, MLE,…
Radial distortion

20. More Single-View Geometry
Projective cameras and
planes, lines, conics and quadrics.
Camera center and camera rotation
Camera calibration and vanishing points, calibrating conic and the IAC

21. Single View Metrology
Antonio Criminisi

22. Two-View Geometry
The Birth of Venus (detail), c. 1485, Sandro Botticelli

23. Epipolar Geometry
Fundamental matrix Essential matrix

24. Two-View Reconstruction

25. Epipolar Geometry Computation
(normalized) linear:
minimal:
MLE:
RANSAC
… and automated two view matching

26. Rectification
Warp images to simplify epipolar geometry

27. Structure Computation
Points: Linear, optimal, direct optimal
Also lines and vanishing points

28. Planes and Homographies
Relation between plane and H given P and P’
Relation between H and F, H from F, F from H
The infinity homography H∞

29. Three-View Geometry
The Birth of Venus (detail), c. 1485, Sandro Botticelli

30. Trifocal Tensor

31. Three View Reconstruction
(normalized) linear
minimal (6 points)
MLE (Gold Standard)

32. Multiple-View Geometry
The Birth of Venus (detail), c. 1485, Sandro Botticelli

33. Multiple View Geometry
Quadrifocal tensor
81 parameters, but only 29 DOF!

34. Multiple View Reconstruction
Affine factorization
Projective factorization

35. Multiple View Reconstruction
Sequential reconstruction

36. Bundle Adjustment Maximum Likelyhood Estimation
for complete structure and motion U1 U2 U3 WT W V P1 P2 P3 M
12xm
3xn
(in general
much larger)

37. Bundle Adjustment Maximum Likelyhood Estimation
for complete structure and motion WT V
U-WV-1WT
11xm
3xn

38. (including radial distortion)
Bundle adjustment
No bundle adjustment
Bundle adjustment needed
to avoid drift of virtual object
throughout sequence
Bundle adjustment
(including radial distortion)

39. Auto-calibration * *
projection
constraints

40. Dynamic Structure from Motion

41. Cheirality Oriented projective geometry
Allows to use fact that points are in front of camera
to recover quasi-affine reconstruction
to determine order for image warping
to determine orientation for rectification
with epipoles in images
etc.

42. Duality
Gives possibility to interchange role of P and X in algorithms

43. Contact information
Marc Pollefeys, Room 205
Tel