One view geometry---camera modeling u v X O X’ u’ P3 P2

Modeling ‘abstract’ camera: projection from P3 to P2 Math: central proj. Physics: pin-hole As lines are preserved so that it is a linear transformation and can be represented by a 3*4 matrix This is the most general camera model without considering optical distortion

X’ P3 u X u’ u v O P2

Properties of the 3*4 matrix P d.o.f.? Rank(P) = ? ker(P)? row vectors, planes column vectors, directions principal plane: w=0 calibration, how many pts? decomposition by QR, K intrinsic (5). R, t, extrinsic (6) geometric interpretation of K, R, t (backward from u/x=v/y=f/z to P) internal parameters and absolute conic Ker(P)=-R^T t

Geometric description From concrete phy. paramters to algebraic param. Central projection in Cartesian coordinates Camera coordinate frame image coordinate frame world coordinate frame

Camera coordinate frame X Y Z x y u v O f

In more familiar matrix form:

Image coordinate frame X Y Z x y u v O f x o y u v

Image coordiante frame: intrinsic parameters Camera calibration matrix 30mm film, Pixel size 13.5mm/500pixels = 0.026mm per pixel, k=1/0.26=38.5, f=16mm, alpha=16*38.5=615 Focal length in horizontal/vertical pixels (2) (aspect ratio) the principal point (2) the skew (1) ? one rough example: 135 film

World (object) coordinate frame Xw Yw Zw X Y Z x y u v O f Xw

World coordinate frame: extrinsic parameters Finally, we should count properly ... Relation between the abstract algebraic and geometric models is in the intrinsic/extrinsic parameters!

Summary of camera modelling 3 coordinate frame projection matrix decomposition intrinsic/extrinsic param

What is the calibration matrix K? It is the image of the absolute conic, prove it first! Point conic: The dual conic:

It turns the camera into an spherical one, or angular/direction sensor! Direction vector: Angle between two rays ...

Don’t forget: when the world is planar … A general plane homography!

Camera calibration Given Estimate C from image processing or by hand  Estimate C decompose C into intrinsic/extrinsic

Calibration set-up: 3D calibration object

The remaining pb is how to solve this ‘trivial’ system of equations!

Review of some basic numerical algorithms linear algebra non-linear optimisation statistics Go to see the slides ‘calibration.ppt’ for math review

Linear algebra in 5 mins Gaussian elimination LU decomposition Choleski (sym positive) LL^T orthogonal decomposition QR (Gram-Schmidt) SVD (the high(est)light of linear algebra!) LU for solving simple linear system, Instead of inversing the matrix, It reduces to two triangular system which can be easily computed by forward/backward substitution QR can solve full-rank least squares Then if you don’t know the rank, do it with SVD and pseudo-inverse row space: first Vs null space: last Vs col space: first Us null space of the trans : last Us

Linear methods of computing P Ax=b, the least squares solution is given be Pseudo-inverse, x = (A^TA)^-1A^Tb, in practice using SVD to compute the pseudo-inverse For homogeneous min(Ax)^2 subject to x^2=1, The sol. Is the eigen vector of A^TA corresonding to the smallest eigen value of A^TA P3 is the principal plane, P34 is the z-coordinate of the origin in the camera frame P3 is the normal vector of the principal plane or the direction of the optical axis (Toscani-Faugeras method) Geometric interpretation of these constraints

Non-linear … ||p3||=1 and no skew (p1 X p3) . (p2 X p3) = 0 Ax=b, the least squares solution is given be Pseudo-inverse, x = (A^TA)^-1A^Tb, in practice using SVD to compute the pseudo-inverse For homogeneous min(Ax)^2 subject to x^2=1, The sol. Is the eigen vector of A^TA corresonding to the smallest eigen value of A^TA P3 is the principal plane, P34 is the z-coordinate of the origin in the camera frame P3 is the normal vector of the principal plane or the direction of the optical axis (Toscani-Faugeras method) It’s constrained optimizaion, but non linear …

Decomposition analytical by equating K(R,t)=P QR (more exactly it is RQ) P3*3 * P3*3 ^T = KK^T dual conic, Trouble with cholesky, do it directly, analytically, very easy! Expand K(R t), then let K(R,t)=P, then solve one term after another …

Linear, but non-optimal, but we want optima, but non-linear, methods of computing P Each observation has a (0,sigma) independent Gaussian distribution: The probability of obtaining all measurements x given that the camera matrix is P: The measurement probability is in fact the conditional probability (w.r.t. the model) Which is called likelihood This is MLE, We can also have Maximum a posteriori (MAP), then we can integrate any prior knowledge On parameters (for instance, here the 3d coordinates and knowledge of camera parameters …) Look Nister’s thesis MLE (maximum likelihood estimate) is:

Even linear models, but end up with non-linear optimization … Non-linear models for optimization, always end-up iterative linear computing! Yesterday, closed-form, algebraic methods (gradients), for small scale, Today, everything numerical in big scale. The measurement probability is in fact the conditional probability (w.r.t. the model) Which is called likelihood This is MLE, We can also have Maximum a posteriori (MAP), then we can integrate any prior knowledge On parameters (for instance, here the 3d coordinates and knowledge of camera parameters …) Look Nister’s thesis

How to solve this non-linear system of equations?

Non-linear iterative optimisation J d = r from vector F(x+d)=F(x)+J d minimize the square of y-F(x+d)=y-F(x)-J d = r – J d normal equation is J^T J d = J^T r (Gauss-Newton) (H+lambda I) d = J^T r (LM) this means: g is gradient vector, H is the hessian matrix jocobien is for squared model, r^T r … Note: F is a vector of functions, i.e. min f=(y-F)^T(y-F)

General non-linear optimisation 1-order , d gradient descent d= g and H =I 2-order, Newton step: H d = -g Gauss-Newton for LS: f=(y-F)^T(y-F), H=J^TJ, g=-J^T r ‘restricted step’, trust-region, LM: (H+lambda W) d = -g R. Fletcher: practical method of optimisation this means: g is gradient vector, H is the hessian matrix jocobien is for squared model, r^T r … f(x+d) = f(x)+g^T d + ½ d^T H d Note: f is a scalar valued function here.

statistics ‘small errors’ --- classical estimation theory analytical based on first order appoxi. Monte Carlo ‘big errors’ --- robust stat. RANSAC LMS M-estimators Talk about it later

(Non-linear) Optical distorsion Radial distorsion could be modeled by alpha_i and eventually (xc,yc) are distorsion parameters you may have any degree for r, but probably the correction is not sufficient for first degree x is normalised coordinates, x’ is actual distorted measures (people are confusing x and x’  normalised as calibrated for ‘photogrammetrists’ (more difficult to write down the cost function), while normalised just to [-1,1] to compute r (easy to integrate into the cost function, still pixels)

Two approaches: add these terms into the non-linear projection equations using a calibration object estimate them independently using the ‘colinearity’ constraint (see Devernay, photogrammetry) to fit lines

Solution uniqueness? Rank of the matrix A?

Using a planar pattern Why? it is more convenient to have a planar calibration pattern than a 3D calibration object, so it’s very popular now for amateurs. Cf. the paper by Zhenyou Zhang (ICCV99), Sturm and Maybank (CVPR99) Homework: read these papers.

first estimate the plane homogrphies Hi from u and x, 1. How to estimate H? 2. Why one may not be sufficient? extract parameters from the plane homographies First look for internal parameters, as once we have int. we can uniquely Determine the external ones by pose (linear algo) H has 8 d.o.f. and external has 6, so we can only have 2 for internal ones.

How to extract intrinsic parameters? The absolute conic in image The (transformed) absolute conic in the plane: The circular points of the Euclidean plane (i,1,0) and (-i,1,0) go thru this conic: two equations on K!

Summary of calibration Get image-space points Solve the linear system Optimal sol. by non-linear method Decomposition by RQ

Where is the camera? Given Camera pose (given K) where is the camera? rom image processing or by hand  Camera pose (given K) where is the camera?

Camera pose: 3-point algorithm fundamental Euclidean constraint 3 point algorithm quarternion for rotation?

Where is the camera? A projective setting on a plane, 1D camera, Circle by 3 pts and a constant angle Chasles conics 2D camera calibration singularities