1. Computer Vision Cameras, lenses and sensors Marc Pollefeys COMP 256

2. Computer Vision Camera Models – Pinhole Perspective Projection – Affine Projection Camera with Lenses Sensing The Human Eye Reading: Chapter 1. Cameras, lenses and sensors

3. Computer Vision Images are two-dimensional patterns of brightness values. They are formed by the projection of 3D objects. Figure from US Navy Manual of Basic Optics and Optical Instruments, prepared by Bureau of Naval Personnel. Reprinted by Dover Publications, Inc., 1969.

4. Computer Vision Animal eye: a looonnng time ago. Pinhole perspective projection: Brunelleschi, XV th Century. Camera obscura: XVI th Century. Photographic camera: Niepce, 1816.

5. Computer Vision Distant objects appear smaller

6. Computer Vision Parallel lines meet vanishing point

7. Computer Vision Vanishing points VPL VPR H VP 1 VP 2 VP 3 To different directions correspond different vanishing points

8. Computer Vision Geometric properties of projection Points go to points Lines go to lines Planes go to whole image or half-plane Polygons go to polygons Degenerate cases: –line through focal point yields point –plane through focal point yields line

9. Computer Vision Pinhole Perspective Equation

10. Computer Vision Affine projection models: Weak perspective projection is the magnification. When the scene relief is small compared its distance from the Camera, m can be taken constant: weak perspective projection.

11. Computer Vision Affine projection models: Orthographic projection When the camera is at a (roughly constant) distance from the scene, take m=1.

12. Computer Vision Planar pinhole perspective Orthographic projection Spherical pinhole perspective

13. Computer Vision Limits for pinhole cameras

14. Computer Vision Camera obscura + lens 

15. Computer Vision Lenses Snell’s law n 1 sin  1 = n 2 sin  2 Descartes’ law

16. Computer Vision Paraxial (or first-order) optics Snell’s law: n 1 sin  1 = n 2 sin  2 Small angles: n 1  1  n 2  2

17. Computer Vision Thin Lenses spherical lens surfaces; incoming light  parallel to axis; thickness << radii; same refractive index on both sides

18. Computer Vision Thin Lenses http://www.phy.ntnu.edu.tw/java/Lens/lens_e.html

19. Computer Vision Thick Lens

20. Computer Vision The depth-of-field 

21. Computer Vision The depth-of-field  yields Similar formula for

22. Computer Vision The depth-of-field decreases with d, increases with Z 0  strike a balance between incoming light and sharp depth range

23. Computer Vision Deviations from the lens model 3 assumptions : 1. all rays from a point are focused onto 1 image point 2. all image points in a single plane 3. magnification is constant deviations from this ideal are aberrations 

24. Computer Vision Aberrations chromatic : refractive index function of wavelength 2 types : 1. geometrical 2. chromatic geometrical : small for paraxial rays  study through 3 rd order optics

25. Computer Vision Geometrical aberrations q spherical aberration q astigmatism q distortion q coma aberrations are reduced by combining lenses 

26. Computer Vision Spherical aberration rays parallel to the axis do not converge outer portions of the lens yield smaller focal lenghts 

27. Computer Vision Astigmatism Different focal length for inclined rays

28. Computer Vision Distortion magnification/focal length different for different angles of inclination Can be corrected! (if parameters are know) pincushion (tele-photo) barrel (wide-angle)

29. Computer Vision Coma point off the axis depicted as comet shaped blob

30. Computer Vision Chromatic aberration rays of different wavelengths focused in different planes cannot be removed completely sometimes achromatization is achieved for more than 2 wavelengths 

31. Computer Vision Lens materials reference wavelengths : F = 486.13 nm d = 587.56 nm C = 656.28 nm lens characteristics : 1. refractive index n d 2. Abbe number V d = ( n d - 1) / ( n F - n C ) typically, both should be high allows small components with sufficient refraction notation : e.g. glass BK7(517642) n d = 1.517 and V d = 64.2 

32. Computer Vision Lens materials additional considerations : humidity and temperature resistance, weight, price,...  Crown Glass Fused Quartz & Fused Silica Plastic (PMMA) Calcium Fluoride Saphire Zinc Selenide 6000 18000 Germanium 14000 9000 100 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 WAVELENGTH (nm)

33. Computer Vision Vignetting Figure from http://www.vanwalree.com/optics/vignetting.html

34. Computer Vision Photographs (Niepce, “La Table Servie,” 1822) Milestones: Daguerreotypes (1839) Photographic Film (Eastman,1889) Cinema (Lumière Brothers,1895) Color Photography (Lumière Brothers, 1908) Television (Baird, Farnsworth, Zworykin, 1920s) CCD Devices (1970) more recently CMOS Collection Harlingue-Viollet.

35. Computer Vision Cameras we consider 2 types :  1. CCD 2. CMOS

36. Computer Vision CCD separate photo sensor at regular positions no scanning charge-coupled devices (CCDs) area CCDs and linear CCDs 2 area architectures : interline transfer and frame transfer photosensitive storage 

37. Computer Vision The CCD camera

38. Computer Vision CMOS Same sensor elements as CCD Each photo sensor has its own amplifier More noise (reduced by subtracting ‘black’ image) Lower sensitivity (lower fill rate) Uses standard CMOS technology Allows to put other components on chip ‘Smart’ pixels Foveon 4k x 4k sensor 0.18  process 70M transistors

39. Computer Vision CCD vs. CMOS Mature technology Specific technology High production cost High power consumption Higher fill rate Blooming Sequential readout Recent technology Standard IC technology Cheap Low power Less sensitive Per pixel amplification Random pixel access Smart pixels On chip integration with other components

40. Computer Vision Color cameras We consider 3 concepts: 1.Prism (with 3 sensors) 2.Filter mosaic 3.Filter wheel … and X3

41. Computer Vision Prism color camera Separate light in 3 beams using dichroic prism Requires 3 sensors & precise alignment Good color separation

42. Computer Vision Prism color camera

43. Computer Vision Filter mosaic Coat filter directly on sensor Demosaicing (obtain full colour & full resolution image)

44. Computer Vision Filter wheel Rotate multiple filters in front of lens Allows more than 3 colour bands Only suitable for static scenes

45. Computer Vision Prism vs. mosaic vs. wheel Wheel 1 Good Average Low Motion 3 or more approach # sensors Separation Cost Framerate Artefacts Bands Prism 3 High Low 3 High-end cameras Mosaic 1 Average Low High Aliasing 3 Low-end cameras Scientific applications

46. Computer Vision new color CMOS sensor Foveon’s X3 better image quality smarter pixels

47. Computer Vision The Human Eye Helmoltz’s Schematic Eye Reproduced by permission, the American Society of Photogrammetry and Remote Sensing. A.L. Nowicki, “Stereoscopy.” Manual of Photogrammetry, Thompson, Radlinski, and Speert (eds.), third edition, 1966.

48. Computer Vision The distribution of rods and cones across the retina Reprinted from Foundations of Vision, by B. Wandell, Sinauer Associates, Inc., (1995).  1995 Sinauer Associates, Inc. Cones in the fovea Rods and cones in the periphery Reprinted from Foundations of Vision, by B. Wandell, Sinauer Associates, Inc., (1995).  1995 Sinauer Associates, Inc.

49. Computer Vision Geometric camera model (Man Drawing a Lute, woodcut, 1525, Albrecht Dürer) perspective projection

50. Computer Vision Models for camera projection the pinhole model revisited : center of the lens = center of projection notice the virtual image plane this is called perspective projection 

51. Computer Vision Perspective projection q origin lies at the center of projection  the Z c axis coincides with the optical axis  X c -axis  to image rows, Y c -axis  to columns  YcYc ZcZc XcXc v u

52. Computer Vision Pseudo-orthographic projection If Z is constant  x= kX and y = kY, where k=f/Z i.e. orthographic projection + a scaling Good approximation if ƒ/ Z ± constant, i.e. if objects are small compared to their distance from the camera 

53. Computer Vision Pictoral comparison  Pseudo - orthographic Perspective

54. Computer Vision Projection matrices the perspective projection model is incomplete : what if : 1. 3D coordinates are specified in a world coordinate frame 2. Image coordinates are expressed as row and column numbers We will not consider additional refinements, such as radial distortions,... 

55. Computer Vision Projection matrices X Z Y 0 v u (u,v) C (X,Y,Z) r1r1 r2r2 r3r3 

56. Computer Vision  (x 0, y 0 ) the pixel coordinates of the principal point  f x the number of pixels per unit length horizontally  f y the number of pixels per unit length vertically  s indicates the skew ; typically s = 0 NB7 : fully calibrated means internally and externally calibrated Projection matrices Image coordinates are to be expressed as pixel coordinates with : NB1: often only integer pixel coordinates matter NB2 : k y /k x is called the aspect ratio NB3 : k x,k y,s,x 0 and y 0 are called internal camera parameters NB4 : when they are known, the camera is internally calibrated NB5 : vector C and matrix R  SO (3) are the external camera parameters NB6 : when these are known, the camera is externally calibrated x y m n 0 1 2 01230123 

57. Computer Vision Projection matrices Exploiting homogeneous coordinates : We also have  We define the calibration matrix :

58. Computer Vision Projection matrices We define yielding for some non-zero   or, with rank M = 3  or,

59. Computer Vision Next class Radiometry: lights and surfaces