Computer Vision Cameras, lenses and sensors Marc Pollefeys COMP 256

Computer Vision Camera Models – Pinhole Perspective Projection – Affine Projection Camera with Lenses Sensing The Human Eye Reading: Chapter 1. Cameras, lenses and sensors

Computer Vision Images are two-dimensional patterns of brightness values. They are formed by the projection of 3D objects. Figure from US Navy Manual of Basic Optics and Optical Instruments, prepared by Bureau of Naval Personnel. Reprinted by Dover Publications, Inc., 1969.

Computer Vision Animal eye: a looonnng time ago. Pinhole perspective projection: Brunelleschi, XV th Century. Camera obscura: XVI th Century. Photographic camera: Niepce, 1816.

Computer Vision Distant objects appear smaller

Computer Vision Parallel lines meet vanishing point

Computer Vision Vanishing points VPL VPR H VP 1 VP 2 VP 3 To different directions correspond different vanishing points

Computer Vision Geometric properties of projection Points go to points Lines go to lines Planes go to whole image or half-plane Polygons go to polygons Degenerate cases: –line through focal point yields point –plane through focal point yields line

Computer Vision Pinhole Perspective Equation

Computer Vision Affine projection models: Weak perspective projection is the magnification. When the scene relief is small compared its distance from the Camera, m can be taken constant: weak perspective projection.

Computer Vision Affine projection models: Orthographic projection When the camera is at a (roughly constant) distance from the scene, take m=1.

Computer Vision Planar pinhole perspective Orthographic projection Spherical pinhole perspective

Computer Vision Limits for pinhole cameras

Computer Vision Camera obscura + lens 

Computer Vision Lenses Snell’s law n 1 sin  1 = n 2 sin  2 Descartes’ law

Computer Vision Paraxial (or first-order) optics Snell’s law: n 1 sin  1 = n 2 sin  2 Small angles: n 1  1  n 2  2

Computer Vision Thin Lenses spherical lens surfaces; incoming light  parallel to axis; thickness << radii; same refractive index on both sides

Computer Vision Thin Lenses

Computer Vision Thick Lens

Computer Vision The depth-of-field 

Computer Vision The depth-of-field  yields Similar formula for

Computer Vision The depth-of-field decreases with d, increases with Z 0  strike a balance between incoming light and sharp depth range

Computer Vision Deviations from the lens model 3 assumptions : 1. all rays from a point are focused onto 1 image point 2. all image points in a single plane 3. magnification is constant deviations from this ideal are aberrations 

Computer Vision Aberrations chromatic : refractive index function of wavelength 2 types : 1. geometrical 2. chromatic geometrical : small for paraxial rays  study through 3 rd order optics

Computer Vision Geometrical aberrations q spherical aberration q astigmatism q distortion q coma aberrations are reduced by combining lenses 

Computer Vision Spherical aberration rays parallel to the axis do not converge outer portions of the lens yield smaller focal lenghts 

Computer Vision Astigmatism Different focal length for inclined rays

Computer Vision Distortion magnification/focal length different for different angles of inclination Can be corrected! (if parameters are know) pincushion (tele-photo) barrel (wide-angle)

Computer Vision Coma point off the axis depicted as comet shaped blob

Computer Vision Chromatic aberration rays of different wavelengths focused in different planes cannot be removed completely sometimes achromatization is achieved for more than 2 wavelengths 

Computer Vision Lens materials reference wavelengths : F = nm d = nm C = nm lens characteristics : 1. refractive index n d 2. Abbe number V d = ( n d - 1) / ( n F - n C ) typically, both should be high allows small components with sufficient refraction notation : e.g. glass BK7(517642) n d = and V d = 64.2 

Computer Vision Lens materials additional considerations : humidity and temperature resistance, weight, price,...  Crown Glass Fused Quartz & Fused Silica Plastic (PMMA) Calcium Fluoride Saphire Zinc Selenide Germanium WAVELENGTH (nm)

Computer Vision Vignetting Figure from

Computer Vision Photographs (Niepce, “La Table Servie,” 1822) Milestones: Daguerreotypes (1839) Photographic Film (Eastman,1889) Cinema (Lumière Brothers,1895) Color Photography (Lumière Brothers, 1908) Television (Baird, Farnsworth, Zworykin, 1920s) CCD Devices (1970) more recently CMOS Collection Harlingue-Viollet.

Computer Vision Cameras we consider 2 types :  1. CCD 2. CMOS

Computer Vision CCD separate photo sensor at regular positions no scanning charge-coupled devices (CCDs) area CCDs and linear CCDs 2 area architectures : interline transfer and frame transfer photosensitive storage 

Computer Vision The CCD camera

Computer Vision CMOS Same sensor elements as CCD Each photo sensor has its own amplifier More noise (reduced by subtracting ‘black’ image) Lower sensitivity (lower fill rate) Uses standard CMOS technology Allows to put other components on chip ‘Smart’ pixels Foveon 4k x 4k sensor 0.18  process 70M transistors

Computer Vision CCD vs. CMOS Mature technology Specific technology High production cost High power consumption Higher fill rate Blooming Sequential readout Recent technology Standard IC technology Cheap Low power Less sensitive Per pixel amplification Random pixel access Smart pixels On chip integration with other components

Computer Vision Color cameras We consider 3 concepts: 1.Prism (with 3 sensors) 2.Filter mosaic 3.Filter wheel … and X3

Computer Vision Prism color camera Separate light in 3 beams using dichroic prism Requires 3 sensors & precise alignment Good color separation

Computer Vision Prism color camera

Computer Vision Filter mosaic Coat filter directly on sensor Demosaicing (obtain full colour & full resolution image)

Computer Vision Filter wheel Rotate multiple filters in front of lens Allows more than 3 colour bands Only suitable for static scenes

Computer Vision Prism vs. mosaic vs. wheel Wheel 1 Good Average Low Motion 3 or more approach # sensors Separation Cost Framerate Artefacts Bands Prism 3 High Low 3 High-end cameras Mosaic 1 Average Low High Aliasing 3 Low-end cameras Scientific applications

Computer Vision new color CMOS sensor Foveon’s X3 better image quality smarter pixels

Computer Vision The Human Eye Helmoltz’s Schematic Eye Reproduced by permission, the American Society of Photogrammetry and Remote Sensing. A.L. Nowicki, “Stereoscopy.” Manual of Photogrammetry, Thompson, Radlinski, and Speert (eds.), third edition, 1966.

Computer Vision The distribution of rods and cones across the retina Reprinted from Foundations of Vision, by B. Wandell, Sinauer Associates, Inc., (1995).  1995 Sinauer Associates, Inc. Cones in the fovea Rods and cones in the periphery Reprinted from Foundations of Vision, by B. Wandell, Sinauer Associates, Inc., (1995).  1995 Sinauer Associates, Inc.

Computer Vision Geometric camera model (Man Drawing a Lute, woodcut, 1525, Albrecht Dürer) perspective projection

Computer Vision Models for camera projection the pinhole model revisited : center of the lens = center of projection notice the virtual image plane this is called perspective projection 

Computer Vision Perspective projection q origin lies at the center of projection  the Z c axis coincides with the optical axis  X c -axis  to image rows, Y c -axis  to columns  YcYc ZcZc XcXc v u

Computer Vision Pseudo-orthographic projection If Z is constant  x= kX and y = kY, where k=f/Z i.e. orthographic projection + a scaling Good approximation if ƒ/ Z ± constant, i.e. if objects are small compared to their distance from the camera 

Computer Vision Pictoral comparison  Pseudo - orthographic Perspective

Computer Vision Projection matrices the perspective projection model is incomplete : what if : 1. 3D coordinates are specified in a world coordinate frame 2. Image coordinates are expressed as row and column numbers We will not consider additional refinements, such as radial distortions,... 

Computer Vision Projection matrices X Z Y 0 v u (u,v) C (X,Y,Z) r1r1 r2r2 r3r3 

Computer Vision  (x 0, y 0 ) the pixel coordinates of the principal point  f x the number of pixels per unit length horizontally  f y the number of pixels per unit length vertically  s indicates the skew ; typically s = 0 NB7 : fully calibrated means internally and externally calibrated Projection matrices Image coordinates are to be expressed as pixel coordinates with : NB1: often only integer pixel coordinates matter NB2 : k y /k x is called the aspect ratio NB3 : k x,k y,s,x 0 and y 0 are called internal camera parameters NB4 : when they are known, the camera is internally calibrated NB5 : vector C and matrix R  SO (3) are the external camera parameters NB6 : when these are known, the camera is externally calibrated x y m n 

Computer Vision Projection matrices Exploiting homogeneous coordinates : We also have  We define the calibration matrix :

Computer Vision Projection matrices We define yielding for some non-zero   or, with rank M = 3  or,

Computer Vision Next class Radiometry: lights and surfaces