1. Cameras and Projections
Dan Witzner Hansen
Cameras and Projections

2. Outline Previously??? Projections Pinhole cameras
Perspective projection
Camera matrix
Camera calibration matrix
Ortographic projection

3. Projection and perspective effects

4. Camera obscura
"Reinerus Gemma-Frisius, observed an eclipse of the sun at Louvain on January 24, 1544, and later he used this illustration of the event in his book De Radio Astronomica et Geometrica, It is thought to be the first published illustration of a camera obscura..."
Hammond, John H., The Camera Obscura, A Chronicle
In Latin, means ‘dark room’

5. Pinhole camera
Pinhole camera is a simple model to approximate imaging process, perspective projection.
Virtual image
pinhole
Known to Aristotle
If we treat pinhole as a point, only one ray from any given point can enter the camera.
Fig from Forsyth and Ponce

6. Camera obscura An attraction in the late 19th century
Jetty at Margate England, 1898.
An attraction in the late 19th century
Around 1870s
Adapted from R. Duraiswami

7. Camera obscura at home http://room-camera-obscura.blogspot.com/
Sketch from

8. Digital cameras Film  sensor array
Often an array of charged coupled devices
Each CCD is light sensitive diode that converts photons (light energy) to electrons
camera
CCD array
frame grabber
optics
computer

9. Color sensing in digital cameras
Bayer grid
Estimate missing components from neighboring values (demosaicing)
Source: Steve Seitz

10. Pinhole size / aperture
How does the size of the aperture affect the image we’d get?
Larger
Smaller

11. Adding a lens A lens focuses light onto the film
“circle of
confusion”
A lens focuses light onto the film
There is a specific distance at which objects are “in focus”
other points project to a “circle of confusion” in the image
Changing the shape of the lens changes this distance

12. Pinhole vs. lens

13. (Center Of Projection)
Lenses F
focal point
optical center
(Center Of Projection)
A lens focuses parallel rays onto a single focal point
focal point at a distance f beyond the plane of the lens
f is a function of the shape and index of refraction of the lens
Aperture of diameter D restricts the range of rays
aperture may be on either side of the lens
Lenses are typically spherical (easier to produce)

14. Thin lenses Thin lens equation:
Any object point satisfying this equation is in focus
What is the shape of the focus region?
How can we change the focus region?
Thin lens applet: (by Fu-Kwun Hwang )

15. Focus and depth of field
Image credit: cambridgeincolour.com

16. Focus and depth of field
Depth of field: distance between image planes where blur is tolerable
Thin lens: scene points at distinct depths come in focus at different image planes.
(Real camera lens systems have greater depth of field.)
“circles of confusion”
Shapiro and Stockman

17. Depth of field Changing the aperture size affects depth of field
A smaller aperture increases the range in which the object is approximately in focus
Flower images from Wikipedia

18. Depth from focus
Images from same point of view, different camera parameters
3d shape / depth estimates
[figs from H. Jin and P. Favaro, 2002]

19. Issues with digital cameras
Noise
big difference between consumer vs. SLR-style cameras
low light is where you most notice noise
Compression
creates artifacts except in uncompressed formats (tiff, raw)
Color
color fringing artifacts from Bayer patterns
Blooming
charge overflowing into neighboring pixels
In-camera processing
oversharpening can produce halos
Interlaced vs. progressive scan video
even/odd rows from different exposures
Are more megapixels better?
requires higher quality lens
noise issues
Stabilization
compensate for camera shake (mechanical vs. electronic)
More info online, e.g.,

20. Modeling Projections

21. Perspective effects

22. Perspective and art
Use of correct perspective projection indicated in 1st century B.C. frescoes
Skill resurfaces in Renaissance: artists develop systematic methods to determine perspective projection (around )
Raphael
Durer, 1525

23. Modeling projection We will use the pin-hole model as an approximation
This way the image is right-side-up
The coordinate system
We will use the pin-hole model as an approximation
Put the optical center (Center Of Projection / camera center ) at the origin
Put the image plane (Projection Plane) in front of the COP
Why?
The camera looks down the negative z axis
we need this if we want right-handed-coordinates

24. Perspective effects Far away objects appear smaller Forsyth and Ponce
But different distances can also lead to same size
Forsyth and Ponce

25. Perspective effects Parallel lines in the scene intersect in the image
Converge in image on horizon line
Image plane
(virtual)
pinhole
Scene

26. Projection equations Pinhole camera model
Principal plane: through C and parallel to iamge plane.
Projection equations
Compute intersection with PP of ray from (x,y,z) to COP
Derived using similar triangles.

27. Field of view
Angular measure of portion of 3D space seen by the camera
Depends on focal length
Images from

28. Pinhole camera model

29. Principal point offset
calibration matrix:

30. Camera rotation and translation
Seldom aligned with the world coordinate system

31. CCD Camera

32. When is skew non-zero? arctan(1/s) g 1 resulting camera:
for CCD/CMOS, always s=0
Image from image, s≠0 possible
(non coinciding principal axis)
resulting camera:

33. Camera parameters A camera is described by several parameters
Translation T of the optical center from the origin of world coords
Rotation R of the image plane
focal length f, principle point (x’c, y’c), pixel size (sx, sy)
blue parameters are called “extrinsics,” red are “intrinsics”
Projection equation
The projection matrix models the cumulative effect of all parameters
Useful to decompose into a series of operations
projection
intrinsics
rotation
translation
identity matrix
The definitions of these parameters are not completely standardized
especially intrinsics—varies from one book to another

34. Projection properties
Many-to-one: any points along same ray map to same point in image
Points  points
Lines  lines (collinearity preserved)
Distances and angles are not preserved
Degenerate cases:
– Line through focal point projects to a point.
– Plane through focal point projects to line
– Plane perpendicular to image plane projects to part of the image.

35. Cameras? More about camera calibration later in the course.
We will see more about what information can be gathered from the images using knowledge of planes and calibrated cameras
For example orientations of planes

36. The projective plane Why do we need homogeneous coordinates?
represent points at infinity, homographies, perspective projection, multi-view relationships
What is the geometric intuition?
a point in the image is a ray in projective space
image plane
(x,y,1) -y
(sx,sy,s)
(0,0,0) x -z
Each point (x,y) on the plane is represented by a ray (sx,sy,s)
all points on the ray are equivalent: (x, y, 1)  (sx, sy, s)

37. Projective lines
What does a line in the image correspond to in projective space?
A line is a plane of rays through origin
all rays (x,y,z) satisfying: ax + by + cz = 0
A line is also represented as a homogeneous 3-vector l l p

38. Point and line duality A line l is a homogeneous 3-vector
It is  to every point (ray) p on the line: l p=0 l l1 l2 p p2 p1
What is the line l spanned by rays p1 and p2 ?
l is  to p1 and p2  l = p1  p2
l is the plane normal
What is the intersection of two lines l1 and l2 ?
p is  to l1 and l2  p = l1  l2
Points and lines are dual in projective space
given any formula, can switch the meanings of points and lines to get another formula

39. Ideal points and lines Ideal point (“point at infinity”)
Ideal line
l  (a, b, 0) – parallel to image plane
(a,b,0) -y x -z
image plane -y
(sx,sy,0) -z x
image plane
Ideal point (“point at infinity”)
p  (x, y, 0) – parallel to image plane
It has infinite image coordinates
Corresponds to a line in the image (finite coordinates)
goes through image origin (principle point)

40. Interpreting the Camera matrix Column Vectors
p1, p2 p3 are the vanishing points along the X,Y,Z axis
P4 is the camera center in world coordinates

41. Interpreting the Camera matrix Row Vectors
Rows are planes
On the plane provided PX =0
Means that camera center is on all planes since PC = 0
PX = 0.. For the x=o= P1X = (0,y,w).. Meaning y axis.. This means that P1 is defined by the camera center and the line x=0 in the image
Similarly for the plane p2
For P3 hence PX = (x,y,0)
P3 is the principal plane containing the camera center is parallel to image
plane. P1, P2 are axes planes formed by Y and X axes and camera
center.

42. Moving the camera center
motion parallax
epipolar line

43. Camera anatomy
Camera center
Column points
Principal plane
Axis plane
Principal point
Principal ray

44. Camera center null-space camera projection matrix
For all A all points on AC project on image of A, therefore C is camera center
Suppose the null space of P is generated by the vector C
We will now show that C is the camera center.
That is all points are mapoped to the same image point PA which means that the line must ne a ray through the camera center.
Image of camera center is (0,0,0)T, i.e. undefined

45. Column vectors
Image points corresponding to X,Y,Z directions and origin

46. Row vectors note: p1,p2 dependent on image reparametrization
If a point X is on the plane then p^3X = 0 implying that the camera center is there but also that P^3 =0
The camera center lies on all 3 planes
Line of intersection of the two planes P1 and P2 is a line joining the camera center and the image origin.i.e. the back projection of the origin
note: p1,p2 dependent on image reparametrization

47. Weak perspective Approximation: treat magnification as constant
Assumes scene depth << average distance to camera
Image plane
World points:

48. Orthographic projection
Given camera at constant distance from scene
World points projected along rays parallel to optical access

49. Orthographic (“telecentric”) lenses
Navitar telecentric zoom lens

50. Correcting radial distortion
from Helmut Dersch

51. Distortion Radial distortion of the image Caused by imperfect lenses
No distortion
Pin cushion
Barrel
Radial distortion of the image
Caused by imperfect lenses
Deviations are most noticeable for rays that pass through the edge of the lens

52. Modeling distortion To model lens distortion
Project to “normalized” image coordinates
Apply radial distortion
Apply focal length translate image center
To model lens distortion
Use above projection operation instead of standard projection matrix multiplication

53. Other camera types Plenoptic camera Time-of-flight Omnidirectional
image

54. Summary Image formation affected by geometry, photometry, and optics.
Projection equations express how world points mapped to 2D image.
Homogenous coordinates allow linear system for projection equations.
Lenses make pinhole model practical.
Parameters (focal length, aperture, lens diameter,…) affect image obtained.

55. What does camera calibration Give
Calculate distances to objects with known size.
Calculate angles
Rotations between two views of a plane (see later)
Perpendicular vectors
Calibration can be done automatically with multiple cameras
2 solutions….

56. Information from planes and cameras
An image line l defines a plane through the camera center with normal n=KTl measured in the camera’s Euclidean frame (e.g. orientation)
Given K and a homograpy, two possible normals and orientations are possible
Vanishing line l of the plane pi is obtained by intersecting the imageplane with a plane through the camera centre and parallel to pi
The solution is to use vanishing points

57. Angles and Orthogonality relation
Image of Absolute Conic (IAC)
Bruges til kali
1) So far we have just seen on the properties of forward and backward properties of various entities such as lines and points.
These only depend on the 3x4 form of P. So what is Gained by having a projection matrix
In particular we will see that Euclidian measurements can be estimated. This means the now we can calculate angles and lengths etc
Rays with direction d in Euclidian coordinates
brering

58. Homographies Revisited

59. 2D 2D A special case, planes
Homography matrix (3x3)
Observer 2D 2D
Image plane
(retina, film, canvas)
P [X,Y,0,1]
World plane
A plane to plane projective transformation

60. Applications of homographies

61. Rectifying slanted views
Corrected image (front-to-parallel)

62. Camera rotation

63. Action of projective camera on planes
The most general transformation that can occur between
a scene plane and an image plane under perspective
imaging is a plane projective transformation
Freedom of choosing world coordinates as we like
i.e. Z = 0 at the plane (and Camera center not on the plane)
Thus a homography
Example: Homography between world plane Z=0 and image
implies

64. Homography Estimation
Number of measurements needed for estimating H
How do do we estimate H automatically
We need to consider robustness to outliers

65. Cross product as MatRix Multiplication
Recall: If
a = (xa, ya, za)T and b = (xb, yb, zb)T,
then: c = a x b = (ya zb - za yb, za xb - xa zb, xa yb - ya xb)T

66. Estimating H: DLT Algorithm
x0i = Hxi is an equation involving homogeneous vectors, so Hxi and x0i need only be in the same direction, not strictly equal
We can specify “same directionality” by using a cross product formulation:
See Hartley & Zisserman, Chapter

67. Homography Estimation
How would solve this equation (remember that it the elements in H that needs to be estimated)?
H^j row matrix of H
X_i = (x_i,y_i,w_i)
Only two of them are liniealy independent
May also use the inhomogenious solution, however this method requres an assumption i.e. h_33=1 and may run into problems when h_33 is 0
4 point or 4 lines, but care should be taken when computing H from correspondences of mixed types (i.e. not 2 points and to lines)
Each pair (x,x’) gives us 2 pieces of information and therefore at least 4 point correspondenses are needed.
Duality and other :
Lines and conics may also be used

68. Homographies p p’ A h
2n × 9 9 2n

69. DLT algorithm Objective
Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi
Algorithm
For each correspondence xi ↔xi’ compute Ai. Usually only two first rows needed.
Assemble n 2x9 matrices Ai into a single 2nx9 matrix A
Obtain SVD of A. Solution for h is last column of V
Determine H from h (i.e. reshape from vector to matrix form)

70. Normalizing transformations
DLT is not invariant,
what is a good choice of coordinates?
e.g.
Translate centroid to origin
Scale to a average distance to the origin
Independently on both images Or

71. Normalized DLT algorithm
Objective
Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi
Algorithm
Normalize points
Apply DLT algorithm to
Denormalize solution

72. Camera Calibration

73. Camera calibration

74. How can camera calibration be done?
Any suggestions?

75. Basic Equations 5.5 corresponding points needed
There exist many opens source calibration methods that does this more accurately
5.5 corresponding points needed

76. A simple calibration device
Compute H for each square (corners (0,0),(1,0),(0,1),(1,1)
Compute the imaged circular points H(1,±i,0)T
Fit a conic to 6 circular points
Compute K from w through cholesky factorization
The circular points are on omega (IAC) which is on the circular points determine their image (location). Thus we have two points of omega thus each square gives two coordinates (2*2) of info
Thus 3 sqares yields a total of 6 points (5 enough to determine a conic)
(= Zhang’s calibration method)

77. Vanishing lines, points and calibrated Cameras
Perspective geometry gives rise to vanishing points and lines

78. Vanishing points Properties
image plane
vanishing point v
line on ground plane
camera
center C
line on ground plane
Properties
Any two parallel lines have the same vanishing point v
The ray from C through v is parallel to the lines
An image may have more than one vanishing point
in fact every pixel is a potential vanishing point
How to determine which lines in the scene vanish at a given pixel?

79. Vanishing points lines
Draw vanishing points and lines on the vejle fjord bro picture
Determine plane from vanishing lines ( at least 3) if they are equally spaced
Vanishing line from 3 lines:

80. Vanishing points projection of a point at infinity Vanishing point
image plane
vanishing point v
camera
center C
line on ground plane
Vanishing point
projection of a point at infinity

81. Computing vanishing points (from lines)
Least squares version
Better to use more than two lines and compute the “closest” point of intersection
See notes by Bob Collins for one good way of doing this: q2 q1 p2 p1
Intersect p1q1 with p2q2

82. Computing vanishing pointsD
Properties
P is a point at infinity, v is its projection
They depend only on line direction
Parallel lines P0 + tD, P1 + tD intersect at P

83. Fun with vanishing points

84. Perspective cues

85. Perspective cues

86. Perspective cues

87. Vanishing points and angles
Projection of a direction d: Back project:
2) The Calibration matrix K is the (affine) transformation between x and the ray’s direction d = K^-1 x measured in the cameras Euclidian coordinate frame,
Measure angles: so if we know K then angles can be measured.
Vanishing points are like stars (see pg 215)
So if we have two views with vanishing points v and v’ they both have directions… so if we have calibrated cameras we can then start to measure how
Vanishing points only depend on orientation not on position

88. Next week – no LEctures
WORK ON MANDATORY ASSIGNMENT