Lecturer: Dr. A.H. Abdul Hafez Image Processing and Analysis Computer Vision Lecture 2: Image Formation Geometry Lecturer: Dr. A.H. Abdul Hafez abdul.hafez@hku.edu.tr Hasan Kalyoncu University Faculty of Computer Engineering Fall 2015-16 30 November 2018 HKU, AH Abdul Hafez

Physical parameters of image formation Geometric Type of projection Camera pose Optical Sensor’s lens type focal length, field of view, aperture Photometric Type, direction, intensity of light reaching sensor Surfaces’ reflectance properties Sensor sampling, etc.

Physical parameters of image formation Geometric Type of projection Camera pose Optical Sensor’s lens type focal length, field of view, aperture Photometric Type, direction, intensity of light reaching sensor Surfaces’ reflectance properties Sensor sampling, etc.

Perspective and art Use of correct perspective projection indicated in 1st century B.C. frescoes Skill resurfaces in Renaissance: artists develop systematic methods to determine perspective projection (around 1480-1515) Raphael Durer, 1525 K. Grauman

Perspective projection equations 3d world mapped to 2d projection in image plane Image plane Focal length Optical axis Camera frame ‘ ’ Scene / world points Scene point Image coordinates ‘’ Forsyth and Ponce

Homogeneous coordinates Is this a linear transformation? no—division by z is nonlinear Trick: add one more coordinate: homogeneous image coordinates homogeneous scene coordinates Converting from homogeneous coordinates Slide by Steve Seitz

Perspective Projection Matrix Projection is a matrix multiplication using homogeneous coordinates: divide by the third coordinate to convert back to non-homogeneous coordinates Complete mapping from world points to image pixel positions? Slide by Steve Seitz

Perspective projection & calibration Perspective equations so far in terms of camera’s reference frame…. Camera’s intrinsic and extrinsic parameters needed to calibrate geometry. Camera frame K. Grauman 8

Perspective projection & calibration World frame Extrinsic: Camera frame World frame Intrinsic: Image coordinates relative to camera  Pixel coordinates Camera frame World to camera coord. trans. matrix (4x4) Perspective projection matrix (3x4) Camera to pixel coord. trans. matrix (3x3) = 2D point (3x1) 3D point (4x1) K. Grauman 9

Intrinsic parameters: from idealized world coordinates to pixel values Forsyth&Ponce Perspective projection W. Freeman

Intrinsic parameters But “pixels” are in some arbitrary spatial units Maybe pixels are not square W. Freeman

Intrinsic parameters We don’t know the origin of our camera pixel coordinates W. Freeman

Intrinsic parameters May be skew between camera pixel axes W. Freeman

Intrinsic parameters, homogeneous coordinates Using homogenous coordinates, we can write this as: or: In pixels In camera-based coords W. Freeman

Extrinsic parameters: translation and rotation of camera frame Non-homogeneous coordinates Homogeneous coordinates W. Freeman

Combining extrinsic and intrinsic calibration parameters, in homogeneous coordinates pixels Intrinsic Camera coordinates World coordinates Extrinsic Forsyth&Ponce W. Freeman

Other ways to write the same equation pixel coordinates world coordinates Conversion back from homogeneous coordinates leads to: W. Freeman

Calibration target Find the position, ui and vi, in pixels, of each calibration object feature point. http://www.kinetic.bc.ca/CompVision/opti-CAL.html

Camera calibration From before, we had these equations relating image positions, u,v, to points at 3-d positions P (in homogeneous coordinates): So for each feature point, i, we have: W. Freeman

Camera calibration Stack all these measurements of i=1…n points into a big matrix: W. Freeman

Camera calibration In vector form: Showing all the elements: W. Freeman

Camera calibration Q m = 0 We want to solve for the unit vector m (the stacked one) that minimizes The minimum eigenvector of the matrix QTQ gives us that (see Forsyth&Ponce, 3.1), because it is the unit vector x that minimizes xT QTQ x. W. Freeman

Camera calibration Once you have the M matrix, can recover the intrinsic and extrinsic parameters as in Forsyth&Ponce, sect. 3.2.2. W. Freeman

Recall, perspective effects… Far away objects appear smaller Forsyth and Ponce

Perspective effects

Perspective effects

Perspective effects Parallel lines in the scene intersect in the image Converge in image on horizon line Image plane (virtual) pinhole Scene

Projection Types/properties Many-to-one: any points along same ray map to same point in image Points  ? points Lines  ? lines (collinearity preserved) Distances and angles are / are not ? preserved are not Degenerate cases: – Line through focal point projects to a point. – Plane through focal point projects to line – Plane perpendicular to image plane projects to part of the image.

Projection Types/Weak perspective Approximation: treat magnification as constant Assumes scene depth << average distance to camera Image plane World points:

Projection Types/Orthographic projection Given camera at constant distance from scene World points projected along rays parallel to optical access

2D rigid Transformations

2D Rigid Transformations

3D Rigid Transformations

Slide Credits Bill Freeman Steve Seitz Kristen Grauman Forsyth and Ponce Rick Szeliski Trevol Darrell and others, as marked…

END

Other types of projection Lots of intriguing variants… (I’ll just mention a few fun ones) S. Seitz

360 degree field of view… Basic approach Take a photo of a parabolic mirror with an orthographic lens (Nayar) Or buy one a lens from a variety of omnicam manufacturers… See http://www.cis.upenn.edu/~kostas/omni.html S. Seitz

Titlt-shift images from Olivo Barbieri and Photoshop imitations Tilt-shift http://www.northlight-images.co.uk/article_pages/tilt_and_shift_ts-e.html Titlt-shift images from Olivo Barbieri and Photoshop imitations S. Seitz

tilt, shift http://en.wikipedia.org/wiki/Tilt-shift_photography

Tilt-shift perspective correction http://en.wikipedia.org/wiki/Tilt-shift_photography

normal lens tilt-shift lens http://www.northlight-images.co.uk/article_pages/tilt_and_shift_ts-e.html

Rotating sensor (or object) Rollout Photographs © Justin Kerr http://research.famsi.org/kerrmaya.html Also known as “cyclographs”, “peripheral images” S. Seitz

Photofinish S. Seitz