Camera models and calibration Read tutorial chapter 2 and Szeliski’s book pp.29-73

Schedule (tentative) 2 #datetopic 1Sep.17Introduction and geometry 2Sep.24Camera models and calibration 3Oct.1Invariant features 4Oct.8Multiple-view geometry 5Oct.15Model fitting (RANSAC, EM, …) 6Oct.22Stereo Matching 7Oct.29Structure from motion 8Nov.5Segmentation 9Nov.12Shape from X (silhouettes, …) 10Nov.19Optical flow 11Nov.26Tracking (Kalman, particle filter) 12Dec.3Object category recognition 13Dec.10Specific object recognition 14Dec.17Research overview

Brief geometry reminder 3 2D line-point coincidence relation: Point from lines: 2D Ideal points 2D line at infinity 3D plane-point coincidence relation: Point from planes: Plane from points: Line from points: 3D line representation: (as two planes or two points) 3D Ideal points3D plane at infinity

Conics and quadrics 4 l=Cx l x C Conics Quadrics

2D projective transformations A projectivity is an invertible mapping h from P 2 to itself such that three points x 1,x 2,x 3 lie on the same line if and only if h(x 1 ),h(x 2 ),h(x 3 ) do. Definition: A mapping h : P 2  P 2 is a projectivity if and only if there exist a non-singular 3x3 matrix H such that for any point in P 2 reprented by a vector x it is true that h(x)=Hx Theorem: Definition: Projective transformation or 8DOF projectivity=collineation=projective transformation=homography

Transformation of 2D points, lines and conics Transformation for lines Transformation for conics Transformation for dual conics For a point transformation

Fixed points and lines (eigenvectors H =fixed points) (eigenvectors H - T =fixed lines) ( 1 = 2  pointwise fixed line)

Hierarchy of 2D transformations Projective 8dof Affine 6dof Similarity 4dof Euclidean 3dof Concurrency, collinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio Parallellism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints), linear combinations of vectors (centroids). The line at infinity l ∞ Ratios of lengths, angles. The circular points I,J lengths, areas. invariants transformed squares

The line at infinity The line at infinity l  is a fixed line under a projective transformation H if and only if H is an affinity Note: not fixed pointwise

Affine properties from images projection rectification

Affine rectification v1v1 v2v2 l1l1 l2l2 l4l4 l3l3 l∞l∞

The circular points The circular points I, J are fixed points under the projective transformation H iff H is a similarity

The circular points “circular points” l∞l∞ Algebraically, encodes orthogonal directions

Conic dual to the circular points The dual conic is fixed conic under the projective transformation H iff H is a similarity Note: has 4DOF l ∞ is the nullvector l∞l∞

Angles Euclidean: Projective: (orthogonal)

Transformation of 3D points, planes and quadrics Transformation for lines Transformation for quadrics Transformation for dual quadrics For a point transformation(cfr. 2D equivalent)

Hierarchy of 3D transformations Projective 15dof Affine 12dof Similarity 7dof Euclidean 6dof Intersection and tangency Parallellism of planes, Volume ratios, centroids, The plane at infinity π ∞ Angles, ratios of length The absolute conic Ω ∞ Volume

The plane at infinity The plane at infinity π  is a fixed plane under a projective transformation H iff H is an affinity 1.canonical position 2.contains directions 3.two planes are parallel  line of intersection in π ∞ 4.line // line (or plane)  point of intersection in π ∞

The absolute conic The absolute conic Ω ∞ is a fixed conic under the projective transformation H iff H is a similarity The absolute conic Ω ∞ is a (point) conic on π . In a metric frame: or conic for directions: (with no real points) 1.Ω ∞ is only fixed as a set 2.Circle intersect Ω ∞ in two circular points 3.Spheres intersect π ∞ in Ω ∞

The absolute dual quadric The absolute dual quadric Ω * ∞ is a fixed conic under the projective transformation H iff H is a similarity 1.8 dof 2.plane at infinity π ∞ is the nullvector of Ω ∞ 3.Angles:

Camera model Relation between pixels and rays in space ?

Pinhole camera Gemma Frisius, 1544

23 Distant objects appear smaller

24 Parallel lines meet vanishing point

25 Vanishing points VPL VPR H VP 1 VP 2 VP 3 To different directions correspond different vanishing points

26 Geometric properties of projection Points go to points Lines go to lines Planes go to whole image or half-plane Polygons go to polygons Degenerate cases: –line through focal point yields point –plane through focal point yields line

Pinhole camera model linear projection in homogeneous coordinates!

Pinhole camera model

Principal point offset principal point

Principal point offset calibration matrix

Camera rotation and translation ~

CCD camera

General projective camera non-singular 11 dof (5+3+3) intrinsic camera parameters extrinsic camera parameters

Radial distortion Due to spherical lenses (cheap) Model: R R straight lines are not straight anymore

Camera model Relation between pixels and rays in space ?

Projector model Relation between pixels and rays in space (dual of camera) (main geometric difference is vertical principal point offset to reduce keystone effect) ?

Meydenbauer camera vertical lens shift to allow direct ortho-photographs

Affine cameras

Action of projective camera on points and lines forward projection of line back-projection of line projection of point

Action of projective camera on conics and quadrics back-projection to cone projection of quadric

Resectioning

Direct Linear Transform (DLT) rank-2 matrix

Direct Linear Transform (DLT) Minimal solution Over-determined solution  5½ correspondences needed (say 6) P has 11 dof, 2 independent eq./points n  6 points minimize subject to constraint use SVD

Singular Value Decomposition Homogeneous least-squares

Degenerate configurations (i)Points lie on plane or single line passing through projection center (ii)Camera and points on a twisted cubic

Scale data to values of order 1 1.move center of mass to origin 2.scale to yield order 1 values Data normalization

Line correspondences Extend DLT to lines (back-project line) (2 independent eq.)

Geometric error

Gold Standard algorithm Objective Given n≥6 2D to 3D point correspondences {X i ↔x i ’}, determine the Maximum Likelyhood Estimation of P Algorithm (i)Linear solution: (a)Normalization: (b)DLT (ii)Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error: (iii)Denormalization: ~~ ~

Calibration example (i)Canny edge detection (ii)Straight line fitting to the detected edges (iii)Intersecting the lines to obtain the images corners typically precision <1/10 (H&Z rule of thumb: 5 n constraints for n unknowns)

Errors in the world Errors in the image and in the world Errors in the image (standard case)

Restricted camera estimation Minimize geometric error  impose constraint through parametrization Find best fit that satisfies skew s is zero pixels are square principal point is known complete camera matrix K is known Minimize algebraic error  assume map from param q  P=K[R|-RC], i.e. p=g(q)  minimize ||Ag(q)||

Restricted camera estimation Initialization Use general DLT Clamp values to desired values, e.g. s=0,  x =  y Note: can sometimes cause big jump in error Alternative initialization Use general DLT Impose soft constraints gradually increase weights

Image of absolute conic

A simple calibration device (i)compute H for each square (corners  (0,0),(1,0),(0,1),(1,1)) (ii)compute the imaged circular points H(1,±i,0) T (iii)fit a conic to 6 circular points (iv)compute K from  through cholesky factorization (≈ Zhang’s calibration method)

Some typical calibration algorithms Tsai calibration Zhangs calibration Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11): , Z. Zhang. Flexible Camera Calibration By Viewing a Plane From Unknown Orientations. International Conference on Computer Vision (ICCV'99), Corfu, Greece, pages , September

58 Animal eye: a looonnng time ago. Pinhole perspective projection: Brunelleschi, XV th Century. Camera obscura: XVI th Century. Photographic camera: Niepce, 1816.

59 Limits for pinhole cameras

60 Camera obscura + lens 

61 Lenses Snell’s law n 1 sin  1 = n 2 sin  2 Descartes’ law

62 Paraxial (or first-order) optics Snell’s law: n 1 sin  1 = n 2 sin  2 Small angles: n 1  1  n 2  2

63 Thin Lenses spherical lens surfaces; incoming light  parallel to axis; thickness << radii; same refractive index on both sides

64 Thin Lenses

65 Thick Lens

66 The depth-of-field 

67 The depth-of-field  yields Similar formula for

68 The depth-of-field decreases with d, increases with Z 0  strike a balance between incoming light and sharp depth range

69 Deviations from the lens model 3 assumptions : 1. all rays from a point are focused onto 1 image point 2. all image points in a single plane 3. magnification is constant deviations from this ideal are aberrations 

70 Aberrations chromatic : refractive index function of wavelength 2 types : 1. geometrical 2. chromatic geometrical : small for paraxial rays  study through 3 rd order optics

71 Geometrical aberrations q spherical aberration q astigmatism q distortion q coma aberrations are reduced by combining lenses 

72 Spherical aberration rays parallel to the axis do not converge outer portions of the lens yield smaller focal lenghts 

73 Astigmatism Different focal length for inclined rays

74 Distortion magnification/focal length different for different angles of inclination Can be corrected! (if parameters are know) pincushion (tele-photo) barrel (wide-angle)

75 Ultra wide-angle optics Sometimes distortion is what you want Fisheye lens Cata-dioptric system (lens + mirror)

76 Coma point off the axis depicted as comet shaped blob

77 Chromatic aberration rays of different wavelengths focused in different planes cannot be removed completely sometimes achromatization is achieved for more than 2 wavelengths 

78 Lens materials reference wavelengths : F = nm d = nm C = nm lens characteristics : 1. refractive index n d 2. Abbe number V d = ( n d - 1) / ( n F - n C ) typically, both should be high allows small components with sufficient refraction notation : e.g. glass BK7(517642) n d = and V d = 64.2 

79 Lens materials additional considerations : humidity and temperature resistance, weight, price,...  Crown Glass Fused Quartz & Fused Silica Plastic (PMMA) Calcium Fluoride Saphire Zinc Selenide Germanium WAVELENGTH (nm)

80 Vignetting Figure from

81 from Szeliski’s book

82 Next week: Image features