Image formation ECE 847: Digital Image Processing Stan Birchfield Clemson University

Cameras First photograph due to Niepce Basic abstraction is the pinhole camera –lenses required to ensure image is not too dark –various other abstractions can be applied F. Dellaert,

Image formation overview Image formation involves geometry – path traveled by light radiometry – optical energy flow photometry – effectiveness of light to produce “brightness” sensation in human visual system colorimetry – physical specifications of light stimuli that produce given color sensation sensors – converting photons to digital form

Pinhole camera D. Forsyth,

Parallel lines meet: vanishing point each set of parallel lines (=direction) meets at a different point –The vanishing point for this direction Sets of parallel lines on the same plane lead to collinear vanishing points. –The line is called the horizon for that plane

Perspective projection k O P Q j i p q C f Properties of projection: Points go to points Lines go to lines Planes go to whole image Polygons go to polygons Degenerate cases –line through focal point to point –plane through focal point to line F. Dellaert,

Perspective projection (cont.) F. Dellaert,

Weak perspective projection perspective effects, but not over the scale of individual objects collect points into a group at about the same depth, then divide each point by the depth of its group D. Forsyth,

Weak perspective (cont.) F. Dellaert,

Orthographic projection Let Z 0 =1: F. Dellaert,

Pushbroom cameras

Pinhole too big - many directions are averaged, blurring the image Pinhole too small- diffraction effects blur the image Generally, pinhole cameras are dark, because a very small set of rays from a particular point hits the screen. Pinhole size D. Forsyth,

The reason for lenses D. Forsyth,

The thin lens focal points D. Forsyth,

Focusing

Thick lens thick lens has 6 cardinal points: –two focal points (F 1 and F 2 ) –two principal points (H 1 and H 2 ) –two nodal points (N 1 and N 2 ) complex lens is formed by combining individual concave and convex lenses D. Forsyth,

Complex lens All but the simplest cameras contain lenses which are actually composed of several lens elements

Choosing a lens How to select focal length: –x=fX/Z –f=xZ/X Lens format should be >= CCD format to avoid optical flaws at the rim of the lens

Lenses – Practical issues standardized lens mount has two varieties: –C mount –CS mount CS mount lenses cannot be used with C mount cameras

Spherical aberration perfect lensactual lens On a real lens, even parallel rays are not focused perfectly en.wikipedia.org/wiki/Spherical_aberration

Chromatic aberration On a real lens, different wavelengths are not focused the same

Radial distortion straight lines are curved: uncorrectedcorrected

Radial distortion (cont.) barrel distortion (more common) pincushion distortion pincushion barrel Two types:

Vignetting vignetting – reduction of brightness at periphery of image D. Forsyth,

Normalized Image coordinates P O u=X/Z = dimensionless ! 1 F. Dellaert,

Pixel units P O u=k f X/Z = in pixels ! [f] = m (in meters) [k] = pixels/m f Pixels are on a grid of a certain dimension F. Dellaert,

Pixel coordinates P O u=u 0 + k f X/Z f We put the pixel coordinate origin on topleft F. Dellaert,

Pixel coordinates in 2D j i (u 0,v 0 ) (0.5,0.5) (640.5,480.5) F. Dellaert,

Summary: Intrinsic Calibration skew 5 Degrees of Freedom ! F. Dellaert,

Camera Pose In order to apply the camera model, objects in the scene must be expressed in camera coordinates. World Coordinates Camera Coordinates Calibration target looks tilted from camera viewpoint. This can be explained as a difference in coordinate systems. F. Dellaert,

Rigid Body Transformations Need a way to specify the six degrees-of-freedom of a rigid body. Why are there 6 DOF? A rigid body is a collection of points whose positions relative to each other can’t change Fix one point, three DOF Fix second point, two more DOF (must maintain distance constraint) Third point adds one more DOF, for rotation around line F. Dellaert,

Notations Superscript references coordinate frame A P is coordinates of P in frame A B P is coordinates of P in frame B Example: F. Dellaert,

Translation F. Dellaert,

Translation Using homogeneous coordinates, translation can be expressed as a matrix multiplication. Translation is commutative F. Dellaert,

Rotation means describing frame A in The coordinate system of frame B F. Dellaert,

Rotation Orthogonal matrix! F. Dellaert,

Example: Rotation about z axis What is the rotation matrix? F. Dellaert,

Rotation in homogeneous coordinates Using homogeneous coordinates, rotation can be expressed as a matrix multiplication. Rotation is not communicative F. Dellaert,

Rigid transformations F. Dellaert,

Rigid transformations (con ’ t) Unified treatment using homogeneous coordinates. F. Dellaert,

Projective Camera Matrix 5+6 DOF = 11 ! F. Dellaert,

Projective Camera Matrix 5+6 DOF = 11 ! F. Dellaert,

Columns & Rows of M m 2 P=0 O F. Dellaert,

Effect of Illumination Light source strength and direction has a dramatic impact on distribution of brightness in the image (e.g. shadows, highlights, etc.) (Subject 8 from the Yale face database due to P. Belhumeur et. al.) F. Dellaert,

Image formation Light source emits photons Absorbed, transmitted, scattered fluorescence Camera source F. Dellaert,

Surfaces receives and emits Incident light from lightfield Act as a light source How much light ? F. Dellaert,

Irradiance Irradiance – amount of light falling on a surface patch symbol=E, units = W/m 2 dA F. Dellaert,

Radiosity power leaving a point per area symbol=B, units = W/m 2 dA F. Dellaert,

Light = Directional Light emitted varies w. direction F. Dellaert,

Steradians (Solid Angle) 3D analogue of 2D angle A R F. Dellaert,

Steradians (cont’d) F. Dellaert,

Polar Coordinates F. Dellaert,

Intensity Intensity – amount of light emitted from a point per steradian symbol=I, units = W/sr F. Dellaert,

Irradiance and Intensity dA F. Dellaert,

Radiance Radiance – amount of light passing through an area dA and symbol=L, units = W x m -2 x sr -1 F. Dellaert,

Radiance is important Response of camera/eye is proportional to radiance Pixel values Constant along a ray F. Dellaert,

Lightfield = Gibson optic array ! 5DOF: Position = 3DOF, 2 DOF for direction F. Dellaert,

Lightfield Sampler F. Dellaert,

Lightfield Sample F. Dellaert,

Lambertian Emitters Lambertian = constant radiance More photons emitted straight up Oblique: see fewer photons, but area looks smaller Same brightness ! Total power is proportional to wedge area “Cosine law” Sun approximates Lambertian: Different angle, same brightness Moon should be less bright at edges, as gets less light from sun. Reflects more light at grazing angles than a Lambertian reflector F. Dellaert,

Radiance Emitted/Reflected Radiance – amount of light emitted from a surface patch per steradian per area foreshortened ! dA F. Dellaert,

Calculating Radiosity If reflected light is not dependent on angle, then can integrate over angle: radiosity is an approximate radiometric unit F. Dellaert,

Example: Sun Power= W Surface Area: m 2 Power = Radiance. Area.  L = W/m 2.sr Example from P. Dutre SIGGRAPH tutorial F. Dellaert,

Irradiance (again) Integrate incoming radiance over hemisphere F. Dellaert,

Example: Sun F. Dellaert,

BRDF L E Symmetric in incoming and outgoing directions – this is the Helmholtz reciprocity principle F. Dellaert,

BRDF Example F. Dellaert,

Lambertian surfaces and albedo For some surfaces, the DHR is independent of illumination direction too –cotton cloth, carpets, matte paper, matte paints, etc. For such surfaces, radiance leaving the surface is independent of angle Called Lambertian surfaces (same Lambert) or ideal diffuse surfaces Use radiosity as a unit to describe light leaving the surface DHR is often called diffuse reflectance, or albedo for a Lambertian surface, BRDF is independent of angle, too. Useful fact: F. Dellaert,

Specular surfaces Another important class of surfaces is specular, or mirror-like. –radiation arriving along a direction leaves along the specular direction –reflect about normal –some fraction is absorbed, some reflected –on real surfaces, energy usually goes into a lobe of directions –can write a BRDF, but requires the use of funny functions F. Dellaert,

Lambertian + specular Widespread model –all surfaces are Lambertian plus specular component Advantages –easy to manipulate –very often quite close true Disadvantages –some surfaces are not e.g. underside of CD’s, feathers of many birds, blue spots on many marine crustaceans and fish, most rough surfaces, oil films (skin!), wet surfaces –Generally, very little advantage in modelling behaviour of light at a surface in more detail -- it is quite difficult to understand behaviour of L+S surfaces F. Dellaert,

Radiometry vs. Photometry F. Dellaert,

Sensors CCD vs. CMOS Types of CCDs: linear, interline, full- frame, frame-transfer Bayer filters progressive scan vs. interlacing NTSC vs. PAL vs. SECAM framegrabbers blooming F. Dellaert,

Bayer color filter