The Pinhole Camera Model F X Z M C Y f x y c m image plane focal plane The geometric model of pinhole camera consists of an image plane I and an eyepoint point C on the focal plane F. The fundamental property of perspective is that every image point m is collinear with the C and its corresponding world point M. The point C is also called optical center, or the focus. The line Cc, perpendicular to I and to F, is called optical axis, c is called the principal point.

Equations for Perspective Projection Let (C,X,Y,Z) be the camera coordinate system (c.s.) and (c,x,y) be the image c.s. By similar triangles, I F X Z M C Y f x y c m

The Pinhole Camera Model X I x f C c Z y Y m M From the geometric viewpoint, there is no difference to replace the image plane by a virtual image plane located on the other side of the focal plane. In this new c.s, an image point (x,y) has 3D coordinates (x,y,f). lM

Perspective Projection Matrix In projective geometry any point along the ray going through the optical center projects to the same image point. So rescaling world coordinates of a point does not change the image coordinates of the point: (X,Y,Z) ~ s(X,Y,Z) = (s X, s Y, s Z) It can be seen from (1): Equations (1) can be rewritten linearly (s arbitrary):

Perspective Projection Matrix and Extrinsic Parameters Given a vector x=[x,y,…]T, we use to denote augmented vector by adding 1 as the last element (homogeneous coordinates). The 3x4 matrix P is called the camera perspective projection matrix.

Perspective Projection Matrix and Extrinsic Parameters Given a 3D point M=[X,Y,Z]T and its image m=[x,y]T (2) can be written in matrix form as (with arbitrary scalar s):

Perspective Projection Matrix and Extrinsic Parameters In practice 3D points can be expressed in arbitrary world c.s. (not only the camera c.s.). We go from the old c.s. centered at the optical center C to the new c.s. centered at point O (world c.s.) by a rotation R followed by a translation t=CO. A relation between coordinates of a single point in a camera c.s. Mc and in the world c.s. Mw is: Mc = R Mw + t

Perspective Projection Matrix and Extrinsic Parameters More compactly, where D is Euclidean transformation of the 3D space.

Perspective Projection Matrix and Extrinsic Parameters The matrix R and the vector t describe the orientation and position of the camera with respect to the world c.s. They are called the extrinsic parameters of the camera (3 rotations +3 translations). X (R,t) Xw I x Zw C c Z y O camera c.s. Yw world c.s. Y m M

Perspective Projection Matrix From (3) and (4) we have: Therefore the new perspective projection matrix is: In real images, the origin of the image c.s. is not the principal point and the scaling along each image axis is different, so the image coordinates undergo a further transformation described by some matrix K, and finally we have:

Intrinsic Parameters of the Camera K is independent of the camera position. It contains the interior (or intrinsic) parameters of the camera. It is represented as an upper triangular matrix: where x and y stand for the scaling along the x and y axes of the image plane (e.g. due to different number of pixels per unit distance in x and y direction in a CCD camera), (x0, y0) are the coordinates of the principal point. s gives the skew (non-orthogonality) between the axes (e.g. as a result of taking an image of an image when the axes of the magnifying lens are not perpendicular to the film plane)

Intrinsic Parameters of the Camera We have and thus Normalized coordinate system of the camera is a system where the image plane is located at a unit distance from the optical center (i.e. f=1). The perspective projection matrix P in such c.s. is given by

Intrinsic Parameters of the Camera For a world point its coordinates in normalized coordinate system are A matrix Pnew defined by (10) can be decomposed: Matrix A contains only intrinsic parameters, and is called camera intrinsic matrix.

Intrinsic Parameters of the Camera It is thus clear that the normalized image coordinates are given by So we have a transformation from coordinates of any camera to coordinates of ideal camera and back. Therefore we may consider only ideal camera case (and keep in mind A matrix for a given real camera)

The General Form of the Perspective Projection Matrix Camera can be considered as a system with intrinsic and extrinsic parameters. Here are 5 intrinsic parameters: fx, fy , the coordinates x0,y0 of principal point, and skew s. There are 6 extrinsic parameters, three for the rotation and three for the translation, which define the transformation from the world coordinate system, to the standard coordinate system of the camera. So the general form of the perspective projection matrix of the camera: P = A PN D = A [R|t] (16) The projection of 3D world coordinates M = [X, Y, Z]T to 2D pixel coordinates m = [x, y]T is then given by

The General Form of the Perspective Projection Matrix cont. Matrix P has 3x4=12 elements, but has only 11 degrees of freedom. Why? Let be the (i,j) entry of the matrix P. Eliminating the scalar s in (17) yields two nonlinear equations:

The General Form of the Perspective Projection Matrix cont. Problem 1. Given the perspective projection matrix P find coordinates of the optical center C of the camera in the world coordinate system. Solution. Decompose the 3x4 matrix P as the concatenation of 3x3 matrix B and a 3-vector b, i.e. P =[B b]. Assume that the rank of B is 3. Under the pinhole model, the optical center projects to [0 0 0]T (i.e. s=0). Therefore, the optical center can be obtained by solving The solution is

The General Form of the Perspective Projection Matrix cont. Problem 2. Given matrix P and an image point m find an optical ray going through this point. Solution. The optical center C is on the optical ray. Any point on this ray is also projected on m. Without loss of generality, we can choose the point D such that the scale factor s =1, i.e. This gives A point on the optical ray is thus given by Where l varies from 0 to

Perspective Approximations The perspective projection (2) is a nonlinear mapping which makes it difficult to solve many vision problems. It also ill-conditioned when perspective effects are small. There are several linear mappings, approximating the perspective projection: Orthographic Projection. It ignores the depth dimension. It can be used if distance and position effects can be ignored.

Orthographic and Weak Perspective Projection Orthographic Projection X I x C c Z y Y

Orthographic and Weak Perspective Projection Orthographic Projection X I C Z Y

Weak Perspective Projection Much more reasonable approximation is Weak Perspective Projection. When the object size is small enough with respect to the distance from the camera to the object, Z can be replaced by a common depth Zc . Then the equations (1) become linear: Here we assumed that the focal length f is normalized to 1

Weak Perspective Projection Two step projection: image plane average depth plane X I C Zc Z Y

Weak Perspective Projection Let Equation (12) can be written as

Weak Perspective Projection Taking into account the intrinsic and extrinsic parameters of the camera yields: where A is the intrinsic matrix (14), and D is the rigid transformation (5).