1. One view geometry---camera modelingu v X O X’ u’ P3 P2

2. Modeling ‘abstract’ camera: projection from P3 to P2
Math: central proj. Physics: pin-hole
As lines are preserved so that it is a linear transformation and can be represented by a 3*4 matrix
This is the most general camera model without considering optical distortion

3. X’ P3 u X u’ u v O P2

4. Properties of the 3*4 matrix P
d.o.f.?
Rank(P) = ?
ker(P)?
row vectors, planes
column vectors, directions
principal plane: w=0
calibration, how many pts?
decomposition by QR,
K intrinsic (5). R, t, extrinsic (6)
geometric interpretation of K, R, t (backward from u/x=v/y=f/z to P)
internal parameters and absolute conic
Ker(P)=-R^T t

5. Geometric description
From concrete phy. paramters to algebraic param.
Central projection in Cartesian coordinates
Camera coordinate frame
image coordinate frame
world coordinate frame

6. Camera coordinate frameX Y Z x y u v O f

7. In more familiar matrix form:

8. Image coordinate frameX Y Z x y u v O f x o y u v

9. Image coordiante frame: intrinsic parameters
Camera calibration matrix
30mm film,
Pixel size 13.5mm/500pixels = 0.026mm per pixel, k=1/0.26=38.5,
f=16mm, alpha=16*38.5=615
Focal length in horizontal/vertical pixels (2) (aspect ratio)
the principal point (2)
the skew (1) ?
one rough example: 135 film

10. World (object) coordinate frameXw Yw Zw X Y Z x y u v O f Xw

11. World coordinate frame: extrinsic parameters
Finally, we should count properly ...
Relation between the abstract algebraic and geometric models is in the intrinsic/extrinsic parameters!

12. Summary of camera modelling
3 coordinate frame
projection matrix
decomposition
intrinsic/extrinsic param

13. What is the calibration matrix K?
It is the image of the absolute conic, prove it first!
Point conic:
The dual conic:

15. It turns the camera into an spherical one, or angular/direction sensor!
Direction vector:
Angle between two rays ...

16. Don’t forget: when the world is planar …
A general plane homography!

17. Camera calibration Given Estimate C
from image processing or by hand 
Estimate C
decompose C into intrinsic/extrinsic

18. Calibration set-up:
3D calibration object

19. The remaining pb is how to solve this ‘trivial’ system of equations!

20. Review of some basic numerical algorithms
linear algebra
non-linear optimisation
statistics
Go to see the slides ‘calibration.ppt’ for math review

21. Linear algebra in 5 mins Gaussian elimination LU decomposition
Choleski (sym positive) LL^T
orthogonal decomposition
QR (Gram-Schmidt)
SVD (the high(est)light of linear algebra!)
LU for solving simple linear system,
Instead of inversing the matrix,
It reduces to two triangular system which can be easily computed by forward/backward substitution
QR can solve full-rank least squares
Then if you don’t know the rank, do it with SVD and pseudo-inverse
row space: first Vs
null space: last Vs
col space: first Us
null space of the trans : last Us

22. Linear methods of computing P
Ax=b, the least squares solution is given be
Pseudo-inverse, x = (A^TA)^-1A^Tb, in practice using SVD to compute the pseudo-inverse
For homogeneous min(Ax)^2 subject to x^2=1,
The sol. Is the eigen vector of A^TA corresonding to the smallest eigen value of A^TA
P3 is the principal plane,
P34 is the z-coordinate of the origin in the camera frame
P3 is the normal vector of the principal plane or the direction of the optical axis (Toscani-Faugeras method)
Geometric interpretation of these constraints

23. Non-linear … ||p3||=1 and no skew (p1 X p3) . (p2 X p3) = 0
Ax=b, the least squares solution is given be
Pseudo-inverse, x = (A^TA)^-1A^Tb, in practice using SVD to compute the pseudo-inverse
For homogeneous min(Ax)^2 subject to x^2=1,
The sol. Is the eigen vector of A^TA corresonding to the smallest eigen value of A^TA
P3 is the principal plane,
P34 is the z-coordinate of the origin in the camera frame
P3 is the normal vector of the principal plane or the direction of the optical axis (Toscani-Faugeras method)
It’s constrained optimizaion, but non linear …

24. Decomposition analytical by equating K(R,t)=P
QR (more exactly it is RQ)
P3*3 * P3*3 ^T = KK^T dual conic,
Trouble with cholesky, do it directly, analytically, very easy!
Expand K(R t), then let K(R,t)=P, then solve one term after another …

25. Linear, but non-optimal, but we want optima, but non-linear, methods of computing P
Each observation has a (0,sigma) independent Gaussian distribution:
The probability of obtaining all measurements x given that the camera matrix is P:
The measurement probability is in fact the conditional probability (w.r.t. the model)
Which is called likelihood
This is MLE,
We can also have Maximum a posteriori (MAP), then we can integrate any prior knowledge
On parameters (for instance, here the 3d coordinates and knowledge of camera parameters …)
Look Nister’s thesis
MLE (maximum likelihood estimate) is:

26. Even linear models, but end up with non-linear optimization … Non-linear models for optimization, always end-up iterative linear computing! Yesterday, closed-form, algebraic methods (gradients), for small scale, Today, everything numerical in big scale.
The measurement probability is in fact the conditional probability (w.r.t. the model)
Which is called likelihood
This is MLE,
We can also have Maximum a posteriori (MAP), then we can integrate any prior knowledge
On parameters (for instance, here the 3d coordinates and knowledge of camera parameters …)
Look Nister’s thesis

27. How to solve this non-linear system of equations?

28. Non-linear iterative optimisation
J d = r from vector F(x+d)=F(x)+J d
minimize the square of y-F(x+d)=y-F(x)-J d = r – J d
normal equation is J^T J d = J^T r (Gauss-Newton)
(H+lambda I) d = J^T r (LM)
this means: g is gradient vector, H is the hessian matrix
jocobien is for squared model, r^T r …
Note: F is a vector of functions, i.e. min f=(y-F)^T(y-F)

29. General non-linear optimisation
1-order , d gradient descent d= g and H =I
2-order,
Newton step: H d = -g
Gauss-Newton for LS: f=(y-F)^T(y-F), H=J^TJ, g=-J^T r
‘restricted step’, trust-region, LM: (H+lambda W) d = -g
R. Fletcher: practical method of optimisation
this means: g is gradient vector, H is the hessian matrix
jocobien is for squared model, r^T r …
f(x+d) = f(x)+g^T d + ½ d^T H d
Note: f is a scalar valued function here.

30. statistics ‘small errors’ --- classical estimation theory
analytical based on first order appoxi.
Monte Carlo
‘big errors’ --- robust stat.
RANSAC
LMS
M-estimators
Talk about it later

31. (Non-linear) Optical distorsion
Radial distorsion could be modeled by
alpha_i and eventually (xc,yc) are distorsion parameters
you may have any degree for r, but probably the correction is not sufficient for first degree
x is normalised coordinates, x’ is actual distorted measures (people are confusing x and x’ 
normalised as calibrated for ‘photogrammetrists’ (more difficult to write down the cost function), while normalised just to [-1,1] to compute r (easy to integrate into the cost function, still pixels)

32. Two approaches:
add these terms into the non-linear projection equations using a calibration object
estimate them independently using the ‘colinearity’ constraint (see Devernay, photogrammetry) to fit lines

33. Solution uniqueness?
Rank of the matrix A?

34. Using a planar pattern
Why? it is more convenient to have a planar calibration pattern than a 3D calibration object, so it’s very popular now for amateurs.
Cf. the paper by Zhenyou Zhang (ICCV99),
Sturm and Maybank (CVPR99)
Homework: read these papers.

35. first estimate the plane homogrphies Hi from u and x,
1. How to estimate H?
2. Why one may not be sufficient?
extract parameters from the plane homographies
First look for internal parameters, as once we have int. we can uniquely
Determine the external ones by pose (linear algo)
H has 8 d.o.f. and external has 6, so we can only have 2 for internal ones.

36. How to extract intrinsic parameters?
The absolute conic in image
The (transformed) absolute conic in the plane:
The circular points of the Euclidean plane (i,1,0) and (-i,1,0) go thru this conic: two equations on K!

37. Summary of calibration
Get image-space points
Solve the linear system
Optimal sol. by non-linear method
Decomposition by RQ

38. Where is the camera? Given Camera pose (given K) where is the camera?
rom image processing or by hand 
Camera pose (given K)
where is the camera?

39. Camera pose: 3-point algorithm
fundamental Euclidean constraint
3 point algorithm
quarternion for rotation?

40. Where is the camera? A projective setting
on a plane, 1D camera,
Circle by 3 pts and a constant angle
Chasles conics
2D camera
calibration singularities