1. The Image Formation Pipeline
Computer Vision : CISC4/689

2. Computer Vision : CISC4/689
Computer Graphics
Output
Image
Model
Synthetic
Camera
(slides courtesy of Michael Cohen)
Computer Vision : CISC4/689

3. Computer Vision : CISC4/689
Output
Model
Real Scene
Real Cameras
(slides courtesy of Michael Cohen)
Computer Vision : CISC4/689

4. Computer Vision : CISC4/689
Combined
Output
Image
Model
Real Scene
Synthetic
Camera
Real Cameras
(slides courtesy of Michael Cohen)
Computer Vision : CISC4/689

5. Computer Vision : CISC4/689
Pinhole cameras
Abstract camera model - box with a small hole in it
Pinhole cameras work in practice
The point to make here is that each point on the image plane sees light from only
one direction, the one that passes through the pinhole.
Computer Vision : CISC4/689

6. Distant objects are smaller
Computer Vision : CISC4/689

7. Consequences: Parallel lines meet
There exist vanishing points
Marc Pollefeys
Computer Vision : CISC4/689

8. The Effect of Perspective
Computer Vision : CISC4/689

9. Computer Vision : CISC4/689
Vanishing Points
Parallel scene lines meet at a vanishing point in the image.
Vertical Line
Vanishing Point
Horizontal Line
Vanishing Point
Andrew C. Gallagher
CRV 2005
Computer Vision : CISC4/689

10. Computer Vision : CISC4/689
Vanishing points
VP1
VP2
VP3
Different directions correspond
to different vanishing points
Marc Pollefeys
Computer Vision : CISC4/689

11. Computer Vision : CISC4/689
Vanishing points
each set of parallel lines (=direction) meets at a different point
The vanishing point for this direction
Sets of parallel lines on the same plane lead to collinear vanishing points.
The line is called the horizon for that plane
If lines are parallel to an axis, corresponding VPs are called axis vanishing points.
Good ways to spot faked images
scale and perspective don’t work
vanishing points behave badly
supermarket tabloids are a great source.
Computer Vision : CISC4/689

12. Computer Vision : CISC4/689
Slide credit: David Jacobs

13. Properties of Projection (Perspective)
Points project to points
Lines project to lines
Vanishing points for parallel lines
Parallel lines parallel to image plane donot converge
Closer objects appear bigger
Angles are not preserved
Degenerate cases
Line through focal point projects to a point.
Plane through focal point projects to line
Computer Vision : CISC4/689

14. Pinhole Camera Terminology
Image plane
Optical axis
Principal point/
image center
Focal length
Camera center/ pinhole
Camera point
Image point
Computer Vision : CISC4/689

15. The equation of projection
Computer Vision : CISC4/689

16. The equation of projection
Cartesian coordinates:
We have, by similar triangles, that (x, y, z) -> (f x/z, f y/z, -f)
Ignore the third coordinate, and get
Computer Vision : CISC4/689

17. Computer Vision : CISC4/689
The camera matrix
Turn previous expression into HC’s
HC’s for 3D point are (X,Y,Z,T)
HC’s for point in image are (U,V,W)
Computer Vision : CISC4/689

18. Computer Vision : CISC4/689
Weak perspective
Issue
perspective effects, but not over the scale of individual objects
collect points into a group at about the same depth, then divide each point by the depth of its group
Adv: easy
Disadv: wrong
Computer Vision : CISC4/689

19. Weak Perspective ProjectionZ
Reduction of
height by same
amount even though
they are at different
distances. O -x Z Z f
Computer Vision : CISC4/689

20. The Equation of Weak Perspective (scaled Orthographic)
s is constant for all points.
Parallel lines no longer converge, they remain parallel.
Computer Vision : CISC4/689
Slide credit: David Jacobs

21. Computer Vision : CISC4/689
Generalization of Orthographic Projection
When the camera is at a
(roughly constant) distance
from the scene, take m=1.
Computer Vision : CISC4/689
Marc Pollefeys

22. The projection matrix for orthographic projection
Computer Vision : CISC4/689

23. Computer Vision : CISC4/689
Pictorial Comparison
Weak perspective
Perspective
Computer Vision : CISC4/689 
Marc Pollefeys

24. Summary: Perspective Laws
Weak perspective
Orthographic
Computer Vision : CISC4/689

25. Pros and Cons of These Models
Weak perspective has simpler math.
Accurate when object is small and distant.
Most useful for recognition.
Pinhole perspective much more accurate for scenes.
Used in structure from motion.
When accuracy really matters, we must model the real camera
Use perspective projection with other calibration parameters (e.g., radial lens distortion)
Computer Vision : CISC4/689
Slide credit: David Jacobs

26. Computer Vision : CISC4/689
Affine cameras
Computer Vision : CISC4/689

27. Computer Vision : CISC4/689
Camera parameters
Issue
camera may not be at the origin, looking down the z-axis
extrinsic parameters
one unit in camera coordinates may not be the same as one unit in world coordinates
intrinsic parameters - focal length, principal point, aspect ratio, angle between axes, etc.
Note the matrix dimensions
Computer Vision : CISC4/689

28. Computer Vision : CISC4/689
Camera calibration
Issues:
what are intrinsic parameters of the camera?
what is the camera matrix? (intrinsic+extrinsic)
General strategy:
view calibration object
identify image points
obtain camera matrix by minimizing error
obtain intrinsic parameters from camera matrix
Error minimization:
Linear least squares
easy problem numerically
solution can be rather bad
Minimize image distance
more difficult numerical problem
solution usually rather good,
start with linear least squares
Numerical scaling is an issue
Computer Vision : CISC4/689

29. Computer Vision : CISC4/689
Outline
Vector, matrix basics
2-D point transformations
Translation, scaling, rotation, shear
Homogeneous coordinates and transformations
Homography, affine transformation
Computer Vision : CISC4/689

30. Computer Vision : CISC4/689
Notes on Notation
Vectors, points: x, v (assume column vectors)
Matrices: R, T
Scalars: x, a
Axes, objects: X, Y, O
Coordinate systems: W, C
Number systems: R, Z
Specials
Transpose operator: xT (as opposed to x0)
Identity matrix: Id
Matrices/vectors of zeroes, ones: 0, 1
Computer Vision : CISC4/689

31. Block Notation for Matrices
Often convenient to write matrices in terms of parts
Smaller matrices for blocks
Row, column vectors for ranges of entries on rows, columns, respectively
E.g.: If A is 3 x 3 and :
Computer Vision : CISC4/689

32. Computer Vision : CISC4/689
2-D Transformations
Types
Scaling
Rotation
Shear
Translation
Mathematical representation
Computer Vision : CISC4/689

33. Computer Vision : CISC4/689
2-D Scaling
Computer Vision : CISC4/689

34. Computer Vision : CISC4/689
2-D Scaling
Computer Vision : CISC4/689

35. Computer Vision : CISC4/689
2-D Scaling sx 1
Horizontal shift proportional to horizontal position
Computer Vision : CISC4/689

36. Computer Vision : CISC4/689
2-D Scaling sy 1
Vertical shift proportional to vertical position
Computer Vision : CISC4/689

37. Computer Vision : CISC4/689
2-D Scaling
Computer Vision : CISC4/689

38. Matrix form of 2-D Scaling
Computer Vision : CISC4/689

39. Computer Vision : CISC4/689
2-D Scaling
Computer Vision : CISC4/689

40. Computer Vision : CISC4/689
2-D Rotation
Computer Vision : CISC4/689

41. Computer Vision : CISC4/689
2-D Rotation
Computer Vision : CISC4/689

42. Computer Vision : CISC4/689
2-D Rotation
Computer Vision : CISC4/689

43. Matrix form of 2-D Rotation
(this is a counterclockwise rotation; reverse signs of sines to get a clockwise one)
Computer Vision : CISC4/689

44. Matrix form of 2-D Rotation
Computer Vision : CISC4/689

45. Computer Vision : CISC4/689
2-D Shear (Horizontal)
Computer Vision : CISC4/689

46. Computer Vision : CISC4/689
2-D Shear (Horizontal)
Horizontal displacement proportional to vertical position
Computer Vision : CISC4/689

47. 2-D Shear (Horizontal)
(Shear factor h is positive for the figure above)
Computer Vision : CISC4/689

48. Computer Vision : CISC4/689
2-D Shear (Horizontal)
Computer Vision : CISC4/689

49. Computer Vision : CISC4/689
2-D Shear (Vertical)
(v is negative for the figure above)
Computer Vision : CISC4/689

50. Computer Vision : CISC4/689
2-D Translation
Computer Vision : CISC4/689

51. Computer Vision : CISC4/689
2-D Translation
Computer Vision : CISC4/689

52. Computer Vision : CISC4/689
2-D Translation
Computer Vision : CISC4/689

53. Computer Vision : CISC4/689
2-D Translation
Computer Vision : CISC4/689

54. Computer Vision : CISC4/689
2-D Translation
Computer Vision : CISC4/689

55. Computer Vision : CISC4/689
2-D Translation
Computer Vision : CISC4/689

56. Representing Transformations
Note that we’ve defined translation as a vector addition but rotation, scaling, etc. as matrix multiplications
It’s inconvenient to have two different operations (addition and multiplication) for different forms of transformation
It would be desirable for all transformations to be expressed in a common, linear form (since matrix multiplications are linear transformations)
Computer Vision : CISC4/689

57. Example: “Trick” of additional coordinate makes this possible
Old way:
New way:
Computer Vision : CISC4/689

58. Computer Vision : CISC4/689
Translation Matrix
We can write the formula using this “expanded” transformation matrix as:
From now on, assume points are in this “expanded” form unless otherwise noted:
Computer Vision : CISC4/689

59. Homogeneous Coordinates
“Expanded” form is called homogeneous coordinates or projective space
Change to projective space by adding a scale factor (usually but not always 1):
Computer Vision : CISC4/689

60. Homogeneous Coordinates: Projective Space
Equivalence is defined up to scale ¸ (non-zero for finite points)
Think of projective points in P2 as rays in R3, where z coordinate is scale factor
All Euclidean points along ray are “same” in this sense
Computer Vision : CISC4/689

61. Leaving Projective Space
Can go back to non-homogeneous representation by dividing by scale factor and dropping extra coordinate:
This is the same as saying “Where does the ray intersect the plane defined by z = 1”?
Analogy to perspective projection, where f=1 (image plane) and lambda is z of any point in the ray. For different lambda’s along the line, projected point is the same, thus Equivalence class
Computer Vision : CISC4/689

62. Homogeneous Coordinates: Rotations, etc.
A 2-D rotation, scaling, shear or other transformation normally expressed by a 2 x 2 matrix R is written in homogeneous coordinates with the following 3 x 3 matrix:
The non-commutativity of matrix multiplication explains why different transformation orders give different results—i.e., RT  TR
In homogeneous
form
In homogeneous form
Computer Vision : CISC4/689

63. Example: Transformations Don’t Commute
Computer Vision : CISC4/689

64. Example: Transformations Don’t Commute
Computer Vision : CISC4/689

65. Example: Transformations Don’t Commute
Computer Vision : CISC4/689

66. Example: Transformations Don’t Commute
Computer Vision : CISC4/689

67. 2D projective transformations
Courtesy: Marc Pollefeys
Definition:
A projectivity is an invertible mapping h from P2 to itself such that three points x1,x2,x3 lie on the same line if and only if h(x1),h(x2),h(x3) do.
A mapping h:P2P2 is a projectivity if and only if there exist a non-singular 3x3 matrix H such that for any point in P2 represented by a vector x it is true that h(x)=Hx
Theorem:
Definition: Projective transformation or
8DOF
projectivity=collineation=projective transformation=homography
Computer Vision : CISC4/689

68. Computer Vision : CISC4/689
2-D Transformations
Full-generality 3 x 3 homogeneous transformation is called a homography
Computer Vision : CISC4/689

69. 2-D Transformations
Full-generality 3 x 3 homogeneous transformation is called a homography
Translation components
Computer Vision : CISC4/689

70. 2-D Transformations Full-generality 3 x 3 homogeneous
transformation is called a homography
Scale/rotation components
Computer Vision : CISC4/689

71. 2-D Transformations
Full-generality 3 x 3 homogeneous transformation is called a homography
Shear/rotation components
Shear/rotation off-diagonal (more about relationship between shearing and rotation at
Computer Vision : CISC4/689

72. 2-D Transformations
Full-generality 3 x 3 homogeneous transformation is called a homography
Homogeneous scaling factor
Computer Vision : CISC4/689

73. Computer Vision : CISC4/689
2-D Transformations
Full-generality 3 x 3 homogeneous transformation is called a homography
When these are zero (as they have been so far), H is an affine transformation
Computer Vision : CISC4/689

74. Hierarchy of 2D transformations
Courtesy: Marc Pollefeys
transformed squares
invariants
i.e, all share an intersection
Concurrency, collinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio
Projective
8dof
Parallellism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints), linear combinations of vectors (centroids).
The line at infinity l∞
Affine
6dof
Ratios of lengths, angles.
The circular points I,J
Similarity
4dof
Euclidean
3dof
lengths, areas.
Computer Vision : CISC4/689

75. Ideal Points and Vanishing Points
Ideal points on the projective plane are located at infinity, and have coordinates of the form (x1,x2,0) .
There is just one free parameter in the coordinates of an ideal point because the scale of a
homogeneous vector is arbitrary [x1(1,x2/x1,0)]- thus the set of all ideal points on the projective plane constitutes a line, called the ideal line.
In the same way, the ideal points of projective 3-space have the form (x1,x2,x3,0).
The image of an ideal point under a projectivity is called a vanishing point, the image of
an ideal line is called a vanishing line, and so on.
An algebraic analysis of this configuration would describe the following:
parallel lines meet at infinity at an ideal point; the image these lines meeting
at the ideal point is at the vanishing point V (the image of the ideal point is V)
- note that concurrency is preserved under a projective transformation, so
concurrency at the ideal point in the world is reflected in concurrency at V
in the image;
Computer Vision : CISC4/689