Cameras and Projections Dan Witzner Hansen Cameras and Projections

Outline Previously??? Projections Pinhole cameras Perspective projection Camera matrix Camera calibration matrix Ortographic projection

Projection and perspective effects

Camera obscura "Reinerus Gemma-Frisius, observed an eclipse of the sun at Louvain on January 24, 1544, and later he used this illustration of the event in his book De Radio Astronomica et Geometrica, 1545. It is thought to be the first published illustration of a camera obscura..." Hammond, John H., The Camera Obscura, A Chronicle In Latin, means ‘dark room’ http://www.acmi.net.au/AIC/CAMERA_OBSCURA.html

Pinhole camera Pinhole camera is a simple model to approximate imaging process, perspective projection. Virtual image pinhole Known to Aristotle If we treat pinhole as a point, only one ray from any given point can enter the camera. Fig from Forsyth and Ponce

Camera obscura An attraction in the late 19th century Jetty at Margate England, 1898. An attraction in the late 19th century Around 1870s http://brightbytes.com/cosite/collection2.html Adapted from R. Duraiswami

Camera obscura at home http://room-camera-obscura.blogspot.com/ Sketch from http://www.funsci.com/fun3_en/sky/sky.htm

Digital cameras Film  sensor array Often an array of charged coupled devices Each CCD is light sensitive diode that converts photons (light energy) to electrons camera CCD array frame grabber optics computer

Color sensing in digital cameras Bayer grid Estimate missing components from neighboring values (demosaicing) Source: Steve Seitz

Pinhole size / aperture How does the size of the aperture affect the image we’d get? Larger Smaller

Adding a lens A lens focuses light onto the film “circle of confusion” A lens focuses light onto the film There is a specific distance at which objects are “in focus” other points project to a “circle of confusion” in the image Changing the shape of the lens changes this distance

Pinhole vs. lens

(Center Of Projection) Lenses F focal point optical center (Center Of Projection) A lens focuses parallel rays onto a single focal point focal point at a distance f beyond the plane of the lens f is a function of the shape and index of refraction of the lens Aperture of diameter D restricts the range of rays aperture may be on either side of the lens Lenses are typically spherical (easier to produce)

Thin lenses Thin lens equation: Any object point satisfying this equation is in focus What is the shape of the focus region? How can we change the focus region? Thin lens applet: http://www.phy.ntnu.edu.tw/java/Lens/lens_e.html (by Fu-Kwun Hwang )

Focus and depth of field Image credit: cambridgeincolour.com

Focus and depth of field Depth of field: distance between image planes where blur is tolerable Thin lens: scene points at distinct depths come in focus at different image planes. (Real camera lens systems have greater depth of field.) “circles of confusion” Shapiro and Stockman

Depth of field Changing the aperture size affects depth of field A smaller aperture increases the range in which the object is approximately in focus Flower images from Wikipedia http://en.wikipedia.org/wiki/Depth_of_field

Depth from focus Images from same point of view, different camera parameters 3d shape / depth estimates [figs from H. Jin and P. Favaro, 2002]

Issues with digital cameras Noise big difference between consumer vs. SLR-style cameras low light is where you most notice noise Compression creates artifacts except in uncompressed formats (tiff, raw) Color color fringing artifacts from Bayer patterns Blooming charge overflowing into neighboring pixels In-camera processing oversharpening can produce halos Interlaced vs. progressive scan video even/odd rows from different exposures Are more megapixels better? requires higher quality lens noise issues Stabilization compensate for camera shake (mechanical vs. electronic) More info online, e.g., http://electronics.howstuffworks.com/digital-camera.htm http://www.dpreview.com/

Modeling Projections

Perspective effects

Perspective and art Use of correct perspective projection indicated in 1st century B.C. frescoes Skill resurfaces in Renaissance: artists develop systematic methods to determine perspective projection (around 1480-1515) Raphael Durer, 1525

Modeling projection We will use the pin-hole model as an approximation This way the image is right-side-up The coordinate system We will use the pin-hole model as an approximation Put the optical center (Center Of Projection / camera center ) at the origin Put the image plane (Projection Plane) in front of the COP Why? The camera looks down the negative z axis we need this if we want right-handed-coordinates

Perspective effects Far away objects appear smaller Forsyth and Ponce But different distances can also lead to same size Forsyth and Ponce

Perspective effects Parallel lines in the scene intersect in the image Converge in image on horizon line Image plane (virtual) pinhole Scene

Projection equations Pinhole camera model Principal plane: through C and parallel to iamge plane. Projection equations Compute intersection with PP of ray from (x,y,z) to COP Derived using similar triangles.

Field of view Angular measure of portion of 3D space seen by the camera Depends on focal length Images from http://en.wikipedia.org/wiki/Angle_of_view

Pinhole camera model

Principal point offset calibration matrix:

Camera rotation and translation Seldom aligned with the world coordinate system

CCD Camera

When is skew non-zero? arctan(1/s) g 1 resulting camera: for CCD/CMOS, always s=0 Image from image, s≠0 possible (non coinciding principal axis) resulting camera:

Camera parameters A camera is described by several parameters Translation T of the optical center from the origin of world coords Rotation R of the image plane focal length f, principle point (x’c, y’c), pixel size (sx, sy) blue parameters are called “extrinsics,” red are “intrinsics” Projection equation The projection matrix models the cumulative effect of all parameters Useful to decompose into a series of operations projection intrinsics rotation translation identity matrix The definitions of these parameters are not completely standardized especially intrinsics—varies from one book to another

Projection properties Many-to-one: any points along same ray map to same point in image Points  points Lines  lines (collinearity preserved) Distances and angles are not preserved Degenerate cases: – Line through focal point projects to a point. – Plane through focal point projects to line – Plane perpendicular to image plane projects to part of the image.

Cameras? More about camera calibration later in the course. We will see more about what information can be gathered from the images using knowledge of planes and calibrated cameras For example orientations of planes

The projective plane Why do we need homogeneous coordinates? represent points at infinity, homographies, perspective projection, multi-view relationships What is the geometric intuition? a point in the image is a ray in projective space image plane (x,y,1) -y (sx,sy,s) (0,0,0) x -z Each point (x,y) on the plane is represented by a ray (sx,sy,s) all points on the ray are equivalent: (x, y, 1)  (sx, sy, s)

Projective lines What does a line in the image correspond to in projective space? A line is a plane of rays through origin all rays (x,y,z) satisfying: ax + by + cz = 0 A line is also represented as a homogeneous 3-vector l l p

Point and line duality A line l is a homogeneous 3-vector It is  to every point (ray) p on the line: l p=0 l l1 l2 p p2 p1 What is the line l spanned by rays p1 and p2 ? l is  to p1 and p2  l = p1  p2 l is the plane normal What is the intersection of two lines l1 and l2 ? p is  to l1 and l2  p = l1  l2 Points and lines are dual in projective space given any formula, can switch the meanings of points and lines to get another formula

Ideal points and lines Ideal point (“point at infinity”) Ideal line l  (a, b, 0) – parallel to image plane (a,b,0) -y x -z image plane -y (sx,sy,0) -z x image plane Ideal point (“point at infinity”) p  (x, y, 0) – parallel to image plane It has infinite image coordinates Corresponds to a line in the image (finite coordinates) goes through image origin (principle point)

Interpreting the Camera matrix Column Vectors p1, p2 p3 are the vanishing points along the X,Y,Z axis P4 is the camera center in world coordinates

Interpreting the Camera matrix Row Vectors Rows are planes On the plane provided PX =0 Means that camera center is on all planes since PC = 0 PX = 0.. For the x=o= P1X = (0,y,w).. Meaning y axis.. This means that P1 is defined by the camera center and the line x=0 in the image Similarly for the plane p2 For P3 hence PX = (x,y,0) P3 is the principal plane containing the camera center is parallel to image plane. P1, P2 are axes planes formed by Y and X axes and camera center.

Moving the camera center motion parallax epipolar line

Camera anatomy Camera center Column points Principal plane Axis plane Principal point Principal ray

Camera center null-space camera projection matrix For all A all points on AC project on image of A, therefore C is camera center Suppose the null space of P is generated by the vector C We will now show that C is the camera center. That is all points are mapoped to the same image point PA which means that the line must ne a ray through the camera center. Image of camera center is (0,0,0)T, i.e. undefined

Column vectors Image points corresponding to X,Y,Z directions and origin

Row vectors note: p1,p2 dependent on image reparametrization If a point X is on the plane then p^3X = 0 implying that the camera center is there but also that P^3 =0 The camera center lies on all 3 planes Line of intersection of the two planes P1 and P2 is a line joining the camera center and the image origin.i.e. the back projection of the origin note: p1,p2 dependent on image reparametrization

Weak perspective Approximation: treat magnification as constant Assumes scene depth << average distance to camera Image plane World points:

Orthographic projection Given camera at constant distance from scene World points projected along rays parallel to optical access

Orthographic (“telecentric”) lenses Navitar telecentric zoom lens http://www.lhup.edu/~dsimanek/3d/telecent.htm

Correcting radial distortion from Helmut Dersch

Distortion Radial distortion of the image Caused by imperfect lenses No distortion Pin cushion Barrel Radial distortion of the image Caused by imperfect lenses Deviations are most noticeable for rays that pass through the edge of the lens

Modeling distortion To model lens distortion Project to “normalized” image coordinates Apply radial distortion Apply focal length translate image center To model lens distortion Use above projection operation instead of standard projection matrix multiplication

Other camera types Plenoptic camera Time-of-flight Omnidirectional image

Summary Image formation affected by geometry, photometry, and optics. Projection equations express how world points mapped to 2D image. Homogenous coordinates allow linear system for projection equations. Lenses make pinhole model practical. Parameters (focal length, aperture, lens diameter,…) affect image obtained.

What does camera calibration Give Calculate distances to objects with known size. Calculate angles Rotations between two views of a plane (see later) Perpendicular vectors Calibration can be done automatically with multiple cameras 2 solutions….

Information from planes and cameras An image line l defines a plane through the camera center with normal n=KTl measured in the camera’s Euclidean frame (e.g. orientation) Given K and a homograpy, two possible normals and orientations are possible Vanishing line l of the plane pi is obtained by intersecting the imageplane with a plane through the camera centre and parallel to pi The solution is to use vanishing points

Angles and Orthogonality relation Image of Absolute Conic (IAC) Bruges til kali 1) So far we have just seen on the properties of forward and backward properties of various entities such as lines and points. These only depend on the 3x4 form of P. So what is Gained by having a projection matrix In particular we will see that Euclidian measurements can be estimated. This means the now we can calculate angles and lengths etc Rays with direction d in Euclidian coordinates brering

Homographies Revisited

2D 2D A special case, planes Homography matrix (3x3) Observer 2D 2D Image plane (retina, film, canvas) P [X,Y,0,1] World plane A plane to plane projective transformation

Applications of homographies

Rectifying slanted views Corrected image (front-to-parallel)

Camera rotation

Action of projective camera on planes The most general transformation that can occur between a scene plane and an image plane under perspective imaging is a plane projective transformation Freedom of choosing world coordinates as we like i.e. Z = 0 at the plane (and Camera center not on the plane) Thus a homography Example: Homography between world plane Z=0 and image implies

Homography Estimation Number of measurements needed for estimating H How do do we estimate H automatically We need to consider robustness to outliers

Cross product as MatRix Multiplication Recall: If a = (xa, ya, za)T and b = (xb, yb, zb)T, then: c = a x b = (ya zb - za yb, za xb - xa zb, xa yb - ya xb)T

Estimating H: DLT Algorithm x0i = Hxi is an equation involving homogeneous vectors, so Hxi and x0i need only be in the same direction, not strictly equal We can specify “same directionality” by using a cross product formulation: See Hartley & Zisserman, Chapter 3.1-3.1.1

Homography Estimation How would solve this equation (remember that it the elements in H that needs to be estimated)? H^j row matrix of H X_i = (x_i,y_i,w_i) Only two of them are liniealy independent May also use the inhomogenious solution, however this method requres an assumption i.e. h_33=1 and may run into problems when h_33 is 0 4 point or 4 lines, but care should be taken when computing H from correspondences of mixed types (i.e. not 2 points and to lines) Each pair (x,x’) gives us 2 pieces of information and therefore at least 4 point correspondenses are needed. Duality and other : Lines and conics may also be used

Homographies p p’ A h 2n × 9 9 2n

DLT algorithm Objective Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi Algorithm For each correspondence xi ↔xi’ compute Ai. Usually only two first rows needed. Assemble n 2x9 matrices Ai into a single 2nx9 matrix A Obtain SVD of A. Solution for h is last column of V Determine H from h (i.e. reshape from vector to matrix form)

Normalizing transformations DLT is not invariant, what is a good choice of coordinates? e.g. Translate centroid to origin Scale to a average distance to the origin Independently on both images Or

Normalized DLT algorithm Objective Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi Algorithm Normalize points Apply DLT algorithm to Denormalize solution

Camera Calibration

Camera calibration

How can camera calibration be done? Any suggestions?

Basic Equations 5.5 corresponding points needed There exist many opens source calibration methods that does this more accurately 5.5 corresponding points needed

A simple calibration device Compute H for each square (corners (0,0),(1,0),(0,1),(1,1) Compute the imaged circular points H(1,±i,0)T Fit a conic to 6 circular points Compute K from w through cholesky factorization The circular points are on omega (IAC) which is on the circular points determine their image (location). Thus we have two points of omega thus each square gives two coordinates (2*2) of info Thus 3 sqares yields a total of 6 points (5 enough to determine a conic) (= Zhang’s calibration method)

Vanishing lines, points and calibrated Cameras Perspective geometry gives rise to vanishing points and lines

Vanishing points Properties image plane vanishing point v line on ground plane camera center C line on ground plane Properties Any two parallel lines have the same vanishing point v The ray from C through v is parallel to the lines An image may have more than one vanishing point in fact every pixel is a potential vanishing point How to determine which lines in the scene vanish at a given pixel?

Vanishing points lines Draw vanishing points and lines on the vejle fjord bro picture Determine plane from vanishing lines ( at least 3) if they are equally spaced Vanishing line from 3 lines:

Vanishing points projection of a point at infinity Vanishing point image plane vanishing point v camera center C line on ground plane Vanishing point projection of a point at infinity

Computing vanishing points (from lines) Least squares version Better to use more than two lines and compute the “closest” point of intersection See notes by Bob Collins for one good way of doing this: http://www-2.cs.cmu.edu/~ph/869/www/notes/vanishing.txt q2 q1 p2 p1 Intersect p1q1 with p2q2

Computing vanishing points D Properties P is a point at infinity, v is its projection They depend only on line direction Parallel lines P0 + tD, P1 + tD intersect at P

Fun with vanishing points

Perspective cues

Perspective cues

Perspective cues

Vanishing points and angles Projection of a direction d: Back project: 2) The Calibration matrix K is the (affine) transformation between x and the ray’s direction d = K^-1 x measured in the cameras Euclidian coordinate frame, Measure angles: so if we know K then angles can be measured. Vanishing points are like stars (see pg 215) So if we have two views with vanishing points v and v’ they both have directions… so if we have calibrated cameras we can then start to measure how Vanishing points only depend on orientation not on position

Next week – no LEctures WORK ON MANDATORY ASSIGNMENT