1. Algorithms and Applications in Computer Vision Lihi Zelnik-Manor lihi@ee.technion.ac.il

2. Let’s get started: Image formation How are objects in the world captured in an image?

3. Physical parameters of image formation Geometric – Type of projection – Camera pose Optical – Sensor’s lens type – focal length, field of view, aperture Photometric – Type, direction, intensity of light reaching sensor – Surfaces’ reflectance properties

4. Image formation Let’s design a camera –Idea 1: put a piece of film in front of an object –Do we get a reasonable image? Slide by Steve Seitz

5. Pinhole camera Slide by Steve Seitz Add a barrier to block off most of the rays –This reduces blurring –The opening is known as the aperture –How does this transform the image?

6. Pinhole camera Pinhole camera is a simple model to approximate imaging process, perspective projection. Fig from Forsyth and Ponce If we treat pinhole as a point, only one ray from any given point can enter the camera. Virtual image pinhole Image plane

7. Camera obscura "Reinerus Gemma-Frisius, observed an eclipse of the sun at Louvain on January 24, 1544, and later he used this illustration of the event in his book De Radio Astronomica et Geometrica, 1545. It is thought to be the first published illustration of a camera obscura..." Hammond, John H., The Camera Obscura, A Chronicle http://www.acmi.net.au/AIC/CAMERA_OBSCURA.html In Latin, means ‘dark room’

8. Camera obscura Jetty at Margate England, 1898. Adapted from R. Duraiswami http://brightbytes.com/cosite/collection2.html Around 1870s An attraction in the late 19 th century

9. Camera obscura at home Sketch from http://www.funsci.com/fun3_en/sky/sky.htm http://blog.makezine.com/archive/2006/02/how_to_room_siz ed_camera_obscu.html

10. Perspective effects

11. Far away objects appear smaller Forsyth and Ponce

12. Perspective effects

13. Parallel lines in the scene intersect in the image Converge in image on horizon line Image plane (virtual) Scene pinhole

14. Physical parameters of image formation Geometric – Type of projection – Camera pose Optical – Sensor’s lens type – focal length, field of view, aperture Photometric – Type, direction, intensity of light reaching sensor – Surfaces’ reflectance properties Sensor – sampling, etc.

15. Physical parameters of image formation Geometric – Type of projection – Camera pose Optical – Sensor’s lens type – focal length, field of view, aperture Photometric – Type, direction, intensity of light reaching sensor – Surfaces’ reflectance properties Sensor – sampling, etc.

16. Perspective and art Use of correct perspective projection indicated in 1 st century B.C. frescoes Skill resurfaces in Renaissance: artists develop systematic methods to determine perspective projection (around 1480-1515) Durer, 1525Raphael K. Grauman

17. Perspective projection equations 3d world mapped to 2d projection in image plane Forsyth and Ponce Camera frame Image plane Optical axis Focal length Scene / world points

18. Perspective projection equations 3d world mapped to 2d projection in image plane Forsyth and Ponce

19. Perspective projection equations 3d world mapped to 2d projection in image plane Forsyth and Ponce

20. Homogeneous coordinates Is this a linear transformation? Trick: add one more coordinate: homogeneous image coordinates homogeneous scene coordinates Converting from homogeneous coordinates no—division by z is nonlinear Slide by Steve Seitz

21. Perspective Projection Matrix divide by the third coordinate to convert back to non- homogeneous coordinates Projection is a matrix multiplication using homogeneous coordinates: Slide by Steve Seitz Complete mapping from world points to image pixel positions?

22. Perspective projection & calibration Camera frame Intrinsic: Image coordinates relative to camera  Pixel coordinates Extrinsic: Camera frame  World frame World frame World to camera coord. trans. matrix (4x4) Perspective projection matrix (3x4) Camera to pixel coord. trans. matrix (3x3) = 2D point (3x1) 3D point (4x1) K. Grauman

23. Perspective projection & calibration Camera frame Intrinsic: Image coordinates relative to camera  Pixel coordinates Extrinsic: Camera frame  World frame World frame World to camera coord. trans. matrix (4x4) Perspective projection matrix (3x4) Camera to pixel coord. trans. matrix (3x3) = 2D point (3x1) 3D point (4x1) K. Grauman So far we defined only the perspective projection matrix

24. World – to -camera K. Grauman World to camera coord. trans. matrix (4x4) = 3D point (4x1) 3D point (4x1) World camera

25. Extrinsic parameters: translation and rotation of camera frame Non-homogeneous coordinates Homogeneous coordinates W. Freeman

26. Perspective projection & calibration Camera frame Intrinsic: Image coordinates relative to camera  Pixel coordinates Extrinsic: Camera frame  World frame World frame World to camera coord. trans. matrix (4x4) Perspective projection matrix (3x4) Camera to pixel coord. trans. matrix (3x3) = 2D point (3x1) 3D point (4x1) K. Grauman

27. Intrinsic parameters: from idealized world coordinates to pixel values Forsyth&Ponce Perspective projection W. Freeman

28. Intrinsic parameters But “pixels” are in some arbitrary spatial units W. Freeman

29. Intrinsic parameters Maybe pixels are not square W. Freeman

30. Intrinsic parameters We don’t know the origin of our camera pixel coordinates W. Freeman

31. Intrinsic parameters May be skew between camera pixel axes W. Freeman

32. Intrinsic parameters, homogeneous coordinates Using homogenous coordinates, we can write this as: or: In camera-based coords In pixels W. Freeman

33. Intrinsic parameters, homogeneous coordinates Using homogenous coordinates, we can write this as: or: In camera-based coords In pixels W. Freeman

34. Extrinsic parameters: translation and rotation of camera frame Non-homogeneous coordinates Homogeneous coordinates W. Freeman

35. Combining extrinsic and intrinsic calibration parameters, in homogeneous coordinates Forsyth&Ponce Intrinsic Extrinsic World coordinates Camera coordinates pixels W. Freeman

36. Other ways to write the same equation pixel coordinates world coordinates Conversion back from homogeneous coordinates leads to: W. Freeman

37. Calibration target http://www.kinetic.bc.ca/CompVision/opti-CAL.html Find the position, u i and v i, in pixels, of each calibration object feature point.

38. Camera calibration So for each feature point, i, we have: From before, we had these equations relating image positions, u,v, to points at 3-d positions P (in homogeneous coordinates): W. Freeman

39. Stack all these measurements of i=1…n points into a big matrix: Camera calibration W. Freeman

40. Showing all the elements: In vector form: Camera calibration W. Freeman

41. We want to solve for the unit vector m (the stacked one) that minimizes Q m = 0 The minimum eigenvector of the matrix Q T Q gives us that (see Forsyth&Ponce, 3.1), because it is the unit vector x that minimizes x T Q T Q x. Camera calibration W. Freeman

42. Once you have the M matrix, can recover the intrinsic and extrinsic parameters as in Forsyth&Ponce, sect. 3.2.2. Camera calibration W. Freeman

43. Perspective projection & calibration Camera frame Intrinsic: Image coordinates relative to camera  Pixel coordinates Extrinsic: Camera frame  World frame World frame World to camera coord. trans. matrix (4x4) Perspective projection matrix (3x4) Camera to pixel coord. trans. matrix (3x3) = 2D point (3x1) 3D point (4x1) K. Grauman

44. Recall, perspective effects… Far away objects appear smaller Forsyth and Ponce

45. Perspective effects

47. Projection properties Many-to-one: all points along same ray map to same point in image Points  ? – points Lines  ? – lines (collinearity preserved) Distances and angles are / are not ? preserved – are not Degenerate cases: – Line through focal point projects to a point. – Plane through focal point projects to line – Plane perpendicular to image plane projects to part of the image.

48. Weak perspective Approximation: treat magnification as constant Assumes scene depth << average distance to camera World points Image plane

49. Orthographic projection Given camera at constant distance from scene World points projected along rays parallel to optical access

50. From 3D to 2D

51. Other types of projection Lots of intriguing variants… (I’ll just mention a few fun ones) S. Seitz

52. 360 degree field of view… Basic approach – Take a photo of a parabolic mirror with an orthographic lens (Nayar) – Or buy one a lens from a variety of omnicam manufacturers… See http://www.cis.upenn.edu/~kostas/omni.html http://www.cis.upenn.edu/~kostas/omni.html S. Seitz

53. Tilt-shift Titlt-shift images from Olivo Barbieri and Photoshop imitationsOlivo Barbieriimitations http://www.northlight-images.co.uk/article_pages/tilt_and_shift_ts-e.html S. Seitz

54. tilt, shift http://en.wikipedia.org/wiki/Tilt-shift_photography

55. Tilt-shift perspective correction http://en.wikipedia.org/wiki/Tilt-shift_photography

56. normal lenstilt-shift lens http://www.northlight-images.co.uk/article_pages/tilt_and_shift_ts-e.html

57. Rollout Photographs © Justin Kerr http://research.famsi.org/kerrmaya.html Rotating sensor (or object) Also known as “cyclographs”, “peripheral images” S. Seitz

58. Photofinish S. Seitz 1. A single vertical slit instead of a shutter The film is advanced continuously at a similar speed to the racers' images 2.A high speed camera takes a continuous series of partial frame photos at a fast rate

59. Physical parameters of image formation Geometric – Type of projection – Camera pose Optical – Sensor’s lens type – focal length, field of view, aperture Photometric – Type, direction, intensity of light reaching sensor – Surfaces’ reflectance properties Sensor – sampling, etc.

60. Pinhole size / aperture Smaller Larger How does the size of the aperture affect the image we’d get? K. Grauman

61. Adding a lens A lens focuses light onto the film –Rays passing through the center are not deviated –All parallel rays converge to one point on a plane located at the focal length f Slide by Steve Seitz focal point f

62. Pinhole vs. lens K. Grauman

63. Cameras with lenses focal point F optical center (Center Of Projection) A lens focuses parallel rays onto a single focal point Gather more light, while keeping focus; make pinhole perspective projection practical K. Grauman

64. Human eye Fig from Shapiro and Stockman Pupil/Iris – control amount of light passing through lens Retina - contains sensor cells, where image is formed Fovea – highest concentration of cones Rough analogy with human visual system:

65. Thin lens Rays entering parallel on one side go through focus on other, and vice versa. In ideal case – all rays from P imaged at P’. Left focus Right focus Focal length fLens diameter d K. Grauman

66. Thin lens equation Any object point satisfying this equation is in focus K. Grauman

67. Focus and depth of field Image credit: cambridgeincolour.com

68. Focus and depth of field Depth of field: distance between image planes where blur is tolerable Thin lens: scene points at distinct depths come in focus at different image planes. (Real camera lens systems have greater depth of field.) Shapiro and Stockman “circles of confusion”

69. Focus and depth of field Images from Wikipedia http://en.wikipedia.org/wiki/Depth_of_fieldhttp://en.wikipedia.org/wiki/Depth_of_field 1.Blurred 2.In focus 3.Blurred

70. Focus and depth of field How does the aperture affect the depth of field? A smaller aperture increases the range in which the object is approximately in focus Flower images from Wikipedia http://en.wikipedia.org/wiki/Depth_of_fieldhttp://en.wikipedia.org/wiki/Depth_of_field Slide from S. Seitz

71. Depth from focus [figs from H. Jin and P. Favaro, 2002] Images from same point of view, different camera parameters 3d shape / depth estimates

72. Field of view Angular measure of portion of 3d space seen by the camera Images from http://en.wikipedia.org/wiki/Angle_of_view K. Grauman

73. As f gets smaller, image becomes more wide angle – more world points project onto the finite image plane As f gets larger, image becomes more telescopic – smaller part of the world projects onto the finite image plane Field of view depends on focal length from R. Duraiswami

74. Field of view depends on focal length Smaller FOV = larger Focal Length Slide by A. Efros

75. Vignetting http://www.ptgui.com/examples/vigntutorial.html http://www.tlucretius.net/Photo/eHolga.html

76. Vignetting “natural”: “mechanical”: intrusion on optical path

77. Chromatic aberration

79. Physical parameters of image formation Geometric – Type of projection – Camera pose Optical – Sensor’s lens type – focal length, field of view, aperture Photometric – Type, direction, intensity of light reaching sensor – Surfaces’ reflectance properties Sensor – sampling, etc.

80. Environment map http://www.sparse.org/3d.html

81. BRDF

82. Diffuse / Lambertian

83. Foreshortening The object will appear “compressed”

84. Specular reflection Ideal reflector: the specular reflection is visible only when line-of-sight == reflected-ray.

85. Phong Ambient+ diffuse+specular:

86. Physical parameters of image formation Geometric – Type of projection – Camera pose Optical – Sensor’s lens type – focal length, field of view, aperture Photometric – Type, direction, intensity of light reaching sensor – Surfaces’ reflectance properties Sensor – sampling, etc.

87. Digital cameras Film  sensor array Often an array of charge coupled devices Each CCD is light sensitive diode that converts photons (light energy) to electrons camera CCD array optics frame grabber computer K. Grauman

88. Historical context Pinhole model: Mozi (470-390 BCE), Aristotle (384-322 BCE) Principles of optics (including lenses): Alhacen (965-1039 CE) Camera obscura: Leonardo da Vinci (1452-1519), Johann Zahn (1631-1707) First photo: Joseph Nicephore Niepce (1822) Daguerréotypes (1839) Photographic film (Eastman, 1889) Cinema (Lumière Brothers, 1895) Color Photography (Lumière Brothers, 1908) Television (Baird, Farnsworth, Zworykin, 1920s) First consumer camera with CCD: Sony Mavica (1981) First fully digital camera: Kodak DCS100 (1990) Niepce, “La Table Servie,” 1822 CCD chip Alhacen’s notes Slide credit: L. Lazebnik K. Grauman

90. Digital Sensors

91. Resolution Sensor: size of real world scene element a that images to a single pixel Image: number of pixels Implications: – what analysis is feasible – affects best representation choice. [fig from Mori et al]

92. Digital images Think of images as matrices taken from CCD array. K. Grauman

93. im[176][201] has value 164 im[194][203] has value 37 width 520 j=1 500 height i=1 Intensity : [0,255] Digital images K. Grauman

94. Color sensing in digital cameras Source: Steve Seitz Estimate missing components from neighboring values (demosaicing) Bayer grid

95. RG B Color images, RGB color space K. Grauman Much more on color in next lecture…

96. Physical parameters of image formation Geometric – Type of projection – Camera pose Optical – Sensor’s lens type – focal length, field of view, aperture Photometric – Type, direction, intensity of light reaching sensor – Surfaces’ reflectance properties Sensor – sampling, etc.

97. Summary Image formation affected by geometry, photometry, and optics. Projection equations express how world points mapped to 2d image. Homogenous coordinates allow linear system for projection equations. Lenses make pinhole model practical Photometry models: Lambertian, BRDF Digital imagers, Bayer demosaicing Parameters (focal length, aperture, lens diameter, sensor sampling…) strongly affect image obtained. K. Grauman

98. Slide Credits Trevor Darrell Bill Freeman Steve Seitz Kristen Grauman Forsyth and Ponce Rick Szeliski and others, as marked…