2. Sebastian Thrun CS223B Computer Vision, Winter 2005 1 Stanford CS223B Computer Vision, Winter 2005 Lecture 2 Lenses and Camera Calibration Sebastian Thrun, Stanford Rick Szeliski, Microsoft Hendrik Dahlkamp, Stanford

3. Sebastian Thrun CS223B Computer Vision, Winter 2005 2 News of the Day Homework assignment 1 is up(?) Reading list on the Web 14 projects on the Web

4. Sebastian Thrun CS223B Computer Vision, Winter 2005 3 Today’s Goals Thin Lens Aberrations Calibration

5. Sebastian Thrun CS223B Computer Vision, Winter 2005 4 Pinhole Camera Marc Pollefeys comp256, Lect 2 -- Brunelleschi, XVth Century

6. Sebastian Thrun CS223B Computer Vision, Winter 2005 5 Snell’s Law Snell’s law n 1 sin  1 = n 2 sin  2

7. Sebastian Thrun CS223B Computer Vision, Winter 2005 6 Thin Lens: Definition optical axis focus f Spherical lense surface: Parallel rays are refracted to single point

8. Sebastian Thrun CS223B Computer Vision, Winter 2005 7 Thin Lens: Projection optical axis z Spherical lense surface: Parallel rays are refracted to single point Image plane f

9. Sebastian Thrun CS223B Computer Vision, Winter 2005 8 Thin Lens: Projection optical axis z Spherical lense surface: Parallel rays are refracted to single point Image plane ff

10. Sebastian Thrun CS223B Computer Vision, Winter 2005 9 Thin Lens: Properties 1.Any ray entering a thin lens parallel to the optical axis must go through the focus on other side 2.Any ray entering through the focus on one side will be parallel to the optical axis on the other side

11. Sebastian Thrun CS223B Computer Vision, Winter 2005 10 Thin Lens: Model Zff O P Q R FrFr FlFl p z

12. Sebastian Thrun CS223B Computer Vision, Winter 2005 11 The Thin Lens Law Zff O P Q R FrFr FlFl p z

13. Sebastian Thrun CS223B Computer Vision, Winter 2005 12 The Thin Lens Law

14. Sebastian Thrun CS223B Computer Vision, Winter 2005 13 Limits of the Thin Lens Model 3 assumptions : 1.all rays from a point are focused onto 1 image point Remember thin lens small angle assumption 2. all image points in a single plane 3. magnification is constant Deviations from this ideal are aberrations

15. Sebastian Thrun CS223B Computer Vision, Winter 2005 14 Today’s Goals Thin Lens Aberrations Calibration

16. Sebastian Thrun CS223B Computer Vision, Winter 2005 15 Aberrations chromatic : refractive index function of wavelength 2 types : geometrical : geometry of the lense, small for paraxial rays Marc Pollefeys

17. Sebastian Thrun CS223B Computer Vision, Winter 2005 16 Geometrical Aberrations q spherical aberration q astigmatism q distortion q coma aberrations are reduced by combining lenses

18. Sebastian Thrun CS223B Computer Vision, Winter 2005 17 Spherical Aberration rays parallel to the axis do not converge outer portions of the lens yield smaller focal lenghts

19. Sebastian Thrun CS223B Computer Vision, Winter 2005 18 Astigmatism Marc Pollefeys Different focal length for inclined rays

20. Sebastian Thrun CS223B Computer Vision, Winter 2005 19 Distortion Can be corrected! (if parameters are know) pincushion (tele-photo) barrel (wide-angle) Marc Pollefeys magnification/focal length different for different angles of inclination

21. Sebastian Thrun CS223B Computer Vision, Winter 2005 20 Coma point off the axis depicted as comet shaped blob Marc Pollefeys

22. Sebastian Thrun CS223B Computer Vision, Winter 2005 21 Chromatic Aberration rays of different wavelengths focused in different planes cannot be removed completely Marc Pollefeys

23. Sebastian Thrun CS223B Computer Vision, Winter 2005 22 Vignetting Effect: Darkens pixels near the image boundary

24. Sebastian Thrun CS223B Computer Vision, Winter 2005 23 CCD vs. CMOS Mature technology Specific technology High production cost High power consumption Higher fill rate Blooming Sequential readout Recent technology Standard IC technology Cheap Low power Less sensitive Per pixel amplification Random pixel access Smart pixels On chip integration with other components Marc Pollefeys

25. Sebastian Thrun CS223B Computer Vision, Winter 2005 24 Today’s Goals Thin Lens Aberrations Calibration –Problem definition –Solution with Homogeneous Parameters –Solution by nonlinear Least Squares method –Distortion

26. Sebastian Thrun CS223B Computer Vision, Winter 2005 25 Intrinsic Camera Parameters Determine the intrinsic parameters of a camera (with lens) What are Intrinsic Parameters?

27. Sebastian Thrun CS223B Computer Vision, Winter 2005 26 Intrinsic Camera Parameters Determine the intrinsic parameters of a camera (with lens) Intrinsic Parameters: –Focal Length f –Pixel size s x, s y –Distortion coefficients k 1, k 2 … –Image center o x, o y

28. Sebastian Thrun CS223B Computer Vision, Winter 2005 27 A Quiz Can we determine all intrinsic parameters by … exposing the camera to many known objects?

29. Sebastian Thrun CS223B Computer Vision, Winter 2005 28 Example Calibration Pattern

30. Sebastian Thrun CS223B Computer Vision, Winter 2005 29 Another Quiz (the last today) How Many Flat Calibration Targets are Needed for Calibration? 1: 2: 3: 4: 5: 10 How Many Corner Points do we need in Total? 1: 2: 3: 4: 10: 20

31. Sebastian Thrun CS223B Computer Vision, Winter 2005 30 Experiment 1: Parallel Board

32. Sebastian Thrun CS223B Computer Vision, Winter 2005 31 30cm10cm20cm Projective Perspective of Parallel Board

33. Sebastian Thrun CS223B Computer Vision, Winter 2005 32 Experiment 2: Tilted Board

34. Sebastian Thrun CS223B Computer Vision, Winter 2005 33 30cm10cm20cm 500cm50cm100cm Projective Perspective of Tilted Board

35. Sebastian Thrun CS223B Computer Vision, Winter 2005 34 Perspective Camera Model Object Space

36. Sebastian Thrun CS223B Computer Vision, Winter 2005 35 Calibration: 2 steps Step 1: Transform into camera coordinates Step 2: Transform into image coordinates

37. Sebastian Thrun CS223B Computer Vision, Winter 2005 36 Calibration Model (extrinsic) Homogeneous Coordinates

38. Sebastian Thrun CS223B Computer Vision, Winter 2005 37 Homogeneous Coordinates Idea: Most Operations Become Linear! Extract Image Coordinates by Z- normalization

39. Sebastian Thrun CS223B Computer Vision, Winter 2005 38 Advantage of Homogeneous C’s i-th data point

40. Sebastian Thrun CS223B Computer Vision, Winter 2005 39 Calibration Model (intrinsic) Pixel size Focal length Image center

41. Sebastian Thrun CS223B Computer Vision, Winter 2005 40 Intrinsic Transformation

42. Sebastian Thrun CS223B Computer Vision, Winter 2005 41 Plugging the Model Together!

43. Sebastian Thrun CS223B Computer Vision, Winter 2005 42 Summary Parameters Extrinsic –Rotation –Translation Intrinsic –Focal length –Pixel size –Image center coordinates –(Distortion coefficients)

44. Sebastian Thrun CS223B Computer Vision, Winter 2005 43 Q: Can We recover all Intrinsic Params? No

45. Sebastian Thrun CS223B Computer Vision, Winter 2005 44 Summary Parameters, Revisited Extrinsic –Rotation –Translation Intrinsic –Focal length –Pixel size –Image center coordinates –(Distortion coefficients) Focal length, in pixel units Aspect ratio

46. Sebastian Thrun CS223B Computer Vision, Winter 2005 45 Today’s Goals Thin Lens Aberrations Calibration –Problem definition –Solution with Homogeneous Parameters –Solution by nonlinear Least Squares method –Distortion

47. Sebastian Thrun CS223B Computer Vision, Winter 2005 46 Calibration a la Trucco Substitute Advantage: Equations are linear in params If over-constrained, minimize Least Mean Square fct One possible solution: Enforce constraint that R is rotation matrix Lots of considerations to recover individual params…

48. Sebastian Thrun CS223B Computer Vision, Winter 2005 47 Today’s Goals Thin Lens Aberrations Calibration –Problem definition –Solution with Homogeneous Parameters –Solution by nonlinear Least Squares method –Distortion

49. Sebastian Thrun CS223B Computer Vision, Winter 2005 48 Calibration by nonlinear Least Squares Calibration Examples: …

50. Sebastian Thrun CS223B Computer Vision, Winter 2005 49 Calibration by nonlinear Least Squares Least Squares

51. Sebastian Thrun CS223B Computer Vision, Winter 2005 50 Calibration by nonlinear Least Squares Least Mean Square Gradient descent:

52. Sebastian Thrun CS223B Computer Vision, Winter 2005 51 Trucco Versus LQ Trucco: Minimization of Squared distance in parameter space Nonlin Least Squares Minimization of Squared distance in Image space

53. Sebastian Thrun CS223B Computer Vision, Winter 2005 52 Q: How Many Images Do We Need? Assumption: K images with M corners each 4+6K parameters 2KM constraints 2KM  4+6K  M>3 and K  2/(M-3) 2 images with 4 points, but will 1 images with 5 points work? No, since points cannot be co-planar!

54. Sebastian Thrun CS223B Computer Vision, Winter 2005 53 Today’s Goals Thin Lens Aberrations Calibration –Problem definition –Solution with Homogeneous Parameters –Solution by nonlinear Least Squares method –Distortion

55. Sebastian Thrun CS223B Computer Vision, Winter 2005 54 Advanced Calibration: Nonlinear Distortions Barrel and Pincushion Tangential

56. Sebastian Thrun CS223B Computer Vision, Winter 2005 55 Barrel and Pincushion Distortion telewideangle

57. Sebastian Thrun CS223B Computer Vision, Winter 2005 56 Models of Radial Distortion distance from center

58. Sebastian Thrun CS223B Computer Vision, Winter 2005 57 Tangential Distortion cheap glue cheap CMOS chip cheap lense image cheap camera

59. Sebastian Thrun CS223B Computer Vision, Winter 2005 58 Image Rectification (to be continued)

60. Sebastian Thrun CS223B Computer Vision, Winter 2005 59 Summary Thin Lens Aberrations Calibration –Problem definition –Solution with Homogeneous Parameters –Solution by nonlinear Least Squares method –Distortion