1. Image formation ECE 847: Digital Image Processing Stan Birchfield
Clemson University

2. Cameras First photograph due to Niepce
Basic abstraction is the pinhole camera
lenses required to ensure image is not too dark
various other abstractions can be applied
F. Dellaert,

3. Image formation overview
Image formation involves
geometry – path traveled by light
radiometry – optical energy flow
photometry – effectiveness of light to produce “brightness” sensation in human visual system
colorimetry – physical specifications of light stimuli that produce given color sensation
sensors – converting photons to digital form

4. Pinhole camera
D. Forsyth,

5. Parallel lines meet: vanishing point
each set of parallel lines (=direction) meets at a different point
The vanishing point for this direction
Sets of parallel lines on the same plane lead to collinear vanishing points.
The line is called the horizon for that plane

6. Perspective projection
Properties of projection:
Points go to points
Lines go to lines
Planes go to whole image
Polygons go to polygons
Degenerate cases
line through focal point to point
plane through focal point to line k O P Q j i p q C f
F. Dellaert,

7. Perspective projection (cont.)
F. Dellaert,

8. Weak perspective projection
perspective effects, but not over the scale of individual objects
collect points into a group at about the same depth, then divide each point by the depth of its group
D. Forsyth,

9. Weak perspective (cont.)
F. Dellaert,

10. Orthographic projection
Let Z0=1:
F. Dellaert,

11. Pushbroom cameras

12. Pinhole size Pinhole too big - many directions are
averaged, blurring the
image
Pinhole too small-
diffraction effects blur
the image
Generally, pinhole
cameras are dark, because
a very small set of rays
from a particular point
hits the screen.
D. Forsyth,

13. The reason for lenses
D. Forsyth,

14. The thin lens focal points
D. Forsyth,

15. Focusing

16. Thick lens thick lens has 6 cardinal points:
two focal points (F1 and F2)
two principal points (H1 and H2)
two nodal points (N1 and N2)
complex lens is formed by combining individual concave and convex lenses
D. Forsyth,

17. Complex lens
All but the simplest cameras contain lenses which are actually composed of several lens elements

18. Choosing a lens How to select focal length:
x=fX/Z
f=xZ/X
Lens format should be >= CCD format to avoid optical flaws at the rim of the lens

19. Lenses – Practical issues
standardized lens mount has two varieties:
C mount
CS mount
CS mount lenses cannot be used with C mount cameras

20. Spherical aberration perfect lens actual lens
On a real lens, even parallel rays are not focused perfectly

21. Chromatic aberration
On a real lens, different wavelengths are not focused the same

22. Radial distortion
straight lines are curved:
uncorrected
corrected

23. Radial distortion (cont.)
Two types:
barrel distortion (more common)
pincushion distortion
barrel
pincushion

24. Vignetting vignetting – reduction of brightness at periphery of image
D. Forsyth,

25. Normalized Image coordinates1 O
u=X/Z = dimensionless ! P
F. Dellaert,

26. Pixel units Pixels are on a grid of a certain dimension f O
u=k f X/Z = in pixels !
[f] = m (in meters)
[k] = pixels/m P
F. Dellaert,

27. Pixel coordinates We put the pixel coordinate origin on topleft f O
u=u0 + k f X/Z P
F. Dellaert,

28. Pixel coordinates in 2D 640 (0.5,0.5) (u0,v0) i 480 (640.5,480.5) j
F. Dellaert,

29. Summary: Intrinsic Calibration
skew
5 Degrees of Freedom !
F. Dellaert,

30. Camera Pose In order to apply the camera model, objects in the scene
must be expressed in camera coordinates.
Camera
Coordinates
World
Coordinates
Calibration target looks tilted from camera
viewpoint. This can be explained as a
difference in coordinate systems.
F. Dellaert,

31. Rigid Body Transformations
Need a way to specify the six degrees-of-freedom of a rigid body.
Why are there 6 DOF?
A rigid body is a
collection of points
whose positions
relative to each
other can’t change
Fix one point,
three DOF
Fix second point,
two more DOF
(must maintain
distance constraint)
Third point adds
one more DOF,
for rotation
around line
F. Dellaert,

32. Notations Superscript references coordinate frame
AP is coordinates of P in frame A
BP is coordinates of P in frame B
Example:
F. Dellaert,

33. Translation
F. Dellaert,

34. Translation
Using homogeneous coordinates, translation can be expressed as a matrix multiplication.
Translation is commutative
F. Dellaert,

35. Rotation means describing frame A in The coordinate system of frame B
F. Dellaert,

36. Rotation Orthogonal matrix!
F. Dellaert,

37. Example: Rotation about z axis
What is the rotation matrix?
F. Dellaert,

38. Rotation in homogeneous coordinates
Using homogeneous coordinates, rotation can be expressed as a matrix multiplication.
Rotation is not commutative
F. Dellaert,

39. Rigid transformations
F. Dellaert,

40. Rigid transformations (con’t)
Unified treatment using homogeneous coordinates.
F. Dellaert,

41. Projective Camera Matrix
5+6 DOF = 11 !
F. Dellaert,

42. Projective Camera Matrix
5+6 DOF = 11 !
F. Dellaert,

43. Columns & Rows of M
m2P=0 O
F. Dellaert,

44. Effect of Illumination
(Subject 8 from the Yale face database due to P. Belhumeur et. al.)
Light source strength and direction has a dramatic impact on distribution of brightness in the image (e.g. shadows, highlights, etc.)
F. Dellaert,

45. Image formation Light source emits photons
Absorbed, transmitted, scattered
fluorescence
source
Camera
F. Dellaert,

46. Surfaces receives and emits
Incident light from lightfield
Act as a light source
How much light ?
F. Dellaert,

47. Irradiance Irradiance – amount of light falling on a surface patch
symbol=E, units = W/m2 dA
F. Dellaert,

48. Radiosity power leaving a point per area symbol=B, units = W/m2 dA
F. Dellaert,

49. Light = Directional Light emitted varies w. direction
F. Dellaert,

50. Steradians (Solid Angle)
3D analogue of 2D angle A R
F. Dellaert,

51. Steradians (cont’d)
F. Dellaert,

52. Polar Coordinates
F. Dellaert,

53. Intensity
Intensity – amount of light emitted from a point per steradian
symbol=I, units = W/sr
F. Dellaert,

54. Irradiance and IntensitydA
F. Dellaert,

55. Radiance Radiance – amount of light passing through an area dA and
symbol=L, units = W x m-2 x sr-1
Photons passing through dA with direction in dw
F. Dellaert,

56. Radiance is important
Response of camera/eye is proportional to radiance
Pixel values
Constant along a ray
F. Dellaert,

57. Lightfield = Gibson optic array !
5DOF: Position = 3DOF, 2 DOF for direction
F. Dellaert,

58. Lightfield Sampler
F. Dellaert,

59. Lightfield Sample
F. Dellaert,

60. Lambertian Emitters Lambertian = constant radiance
More photons emitted straight up
Oblique: see fewer photons, but area looks smaller
Same brightness !
Total power is proportional to wedge area
“Cosine law”
Sun approximates Lambertian: Different angle, same brightness
Moon should be less bright at edges, as gets less light from sun.
Reflects more light at grazing angles than a Lambertian reflector
F. Dellaert,

61. Radiance Emitted/Reflected
Radiance – amount of light emitted from a surface patch per steradian per area
foreshortened ! dA
F. Dellaert,

62. Calculating Radiosity
If reflected light is not dependent on angle, then can integrate over angle: radiosity is an approximate radiometric unit
F. Dellaert,

63. Example: Sun Power= 3.91 1026 W Surface Area:6.07 1018 m2
Power = Radiance . Area . 
L = W/m2.sr
Example from P. Dutre SIGGRAPH tutorial
F. Dellaert,

64. Irradiance (again) Integrate incoming radiance over hemisphere
F. Dellaert,

65. Example: Sun
F. Dellaert,

66. BRDF E L
Symmetric in incoming and outgoing directions – this is the Helmholtz reciprocity principle
F. Dellaert,

67. BRDF Example
                                                                   
F. Dellaert,

68. Lambertian surfaces and albedo
For some surfaces, the DHR is independent of illumination direction too
cotton cloth, carpets, matte paper, matte paints, etc.
For such surfaces, radiance leaving the surface is independent of angle
Called Lambertian surfaces (same Lambert) or ideal diffuse surfaces
Use radiosity as a unit to describe light leaving the surface
DHR is often called diffuse reflectance, or albedo
for a Lambertian surface, BRDF is independent of angle, too.
Useful fact:
The ubiquitous pi again. Tell students that this derivation follows that a few
slides back.
F. Dellaert,

69. Specular surfaces
Another important class of surfaces is specular, or mirror-like.
radiation arriving along a direction leaves along the specular direction
reflect about normal
some fraction is absorbed, some reflected
on real surfaces, energy usually goes into a lobe of directions
can write a BRDF, but requires the use of funny functions
F. Dellaert,

70. Lambertian + specular Widespread model Advantages Disadvantages
all surfaces are Lambertian plus specular component
Advantages
easy to manipulate
very often quite close true
Disadvantages
some surfaces are not
e.g. underside of CD’s, feathers of many birds, blue spots on many marine crustaceans and fish, most rough surfaces, oil films (skin!), wet surfaces
Generally, very little advantage in modelling behaviour of light at a surface in more detail -- it is quite difficult to understand behaviour of L+S surfaces
F. Dellaert,

71. Radiometry vs. Photometry
F. Dellaert,

72. Sensors
CCD vs. CMOS
Types of CCDs: linear, interline, full-frame, frame-transfer
Bayer filters
progressive scan vs. interlacing
NTSC vs. PAL vs. SECAM
framegrabbers
blooming
F. Dellaert,

73. Bayer color filter

74. Adobe’s plenoptic lens
captures multiple views of the scene from slightly different viewpoints
David Salesin and Todor Georgiev
F. Dellaert,

75. Raw image from plenoptic system

76. Reconstructed image

77. Change the focus

78. Lenticular display