Computer Graphics Recitation 12

The plan today Projective geometry basics Epipolar geometry Reminder of homogeneous coordinates Algebraic properties Epipolar geometry Pinhole camera model Fundamental matrix of two views Reprojection using the fundamental matrix

Motivation Applications in computer vision Object recognition in images Reconstruction of 3D models from images

Motivation – image based rendering We are given several images of a 3D world. Want to generate images of novel views without reconstructing the 3D model and rendering it Reconstructing 3D model is a hard task Rendering complex 3D models under complex lighting environment is hard

Motivation – image based rendering Input: two images of a statue

Motivation – image based rendering Output: images of novel views, without reconstruction of 3D!!

Holes may appear… Due to disocclusion

Holes may appear… Due to discretization

Projective geometry Images of 3D world are some projective transformations 3D  2D Expressed in matrix (linear) form when homogeneous coordinates are used

Relation between different images The different images are related by Fundamental Matrices

Projective transformations A general projective transformation Preserves collinearity (three points on a line will stay on the line) Doesn’t preserve parallelism, lengths, angles

Projective transformations Projective transformation is not linear Points at infinity may be mapped to finite points in the image and vice versa

Homogeneous coordinates Add one dimension to the representation Any linear transformation in homogeneous coordinates is in fact projective transformation

Homogeneous coordinates The representation is unique up to scale

Homogeneous coordinates Points at infinity:

Representation of 2D lines ax + by +c = 0 Homogeneous vector (a, b, c)T (a, b, c)T,   0 represents the same line The vector (0, 0, 1) represents the line at infinity

Incidence relation Point p = (x, y, 1)T is on line l = (a, b, c)T  lTp = pTl = 0

Therefore, both p1 and p2 lie on l Incidence relation A line through two points p1 and p2 is l = p1  p2 p1T l = <p1, p1  p2> = 0 p2T l = <p2, p1  p2> = 0 Therefore, both p1 and p2 lie on l

Therefore, p lies on both l1 and l2 Lines intersection Two lines in the plane always intersect (maybe at infinite point) The intersection point of l1 and l2 is p = l1  l2 l1Tp = <l1, l1  l2> = 0 l2Tp = <l2, l1  l2> = 0 Therefore, p lies on both l1 and l2

Projective transformation Linear transformation of homogeneous coordinates From 3D model to 2D image: 34 matrix

Simple pinhole camera model The camera center C is in the origin of R3 The image plane is z = f Y Z f X

Simple pinhole camera model Local coordinates in the image plane principal point Y y Z x X

Simple pinhole camera model The simple camera matrix: y Iy / f = y / z Iy = fy/z Iy f z

Simple pinhole camera model The simple camera matrix:

General pinhole camera model 1) The image plane origin may be translated: Y Z (px, py) X

General pinhole camera model K is called calibration matrix (internal camera parameters):

General pinhole camera model 2) The world coordinates may be rotated and translated camera center (31) rotation 33 matrix from world to camera Y (px, py) Z X

Geometry of two views P p p’ C’ C

The epipoles e is the image of C’ (second camera origin) under the first camera P p p’ e’ e C’ C

The epipoles Clearly, C, C’, e, e’, P, p, p’ all lie in the same plane! A plane that contains the baseline CC’ is called epipolar plane P p p’ e’ e C’ C

Epipolar lines Epipolar plane intersects the image plane at epipolar line The epipolar line will always pass through the epipole P p p’ e’ e C’ C

Epipolar lines Epipolar line l’ is the image of the ray CP in the second camera P Q p’ p e’ e C’ C

Much nicer figure C C’

Another nice figure C C’

Relation between corresponding points p is the image of P in the first camera p’ is the corresponding point in the second camera p’ must lie on the intersection of the image plane with the epipolar plane through C, p, e! P l’ p p’ e’ e C’ C

The basic incidence equation p’ must lie on the epipolar line in the second image plane: p’T l’ = 0 It can be shown that the epipolar line l’ is: l’ = Fp where F is a 33 matrix that depends on the camera matrices only

The fundamental matrix F is a 33 matrix Expresses the incidence relation: p’T F p = 0 Fp is the epipolar line, so we get back p’T l’ = 0

Properties of the fundamental matrix Rank = 2 (because Fe = 0) Thus, 7 degrees of freedom out of 9 coefficients one constraint is det F = 0 scaling – multiplying F by a constant doesn’t change anything because of homogeneity

Finding the fundamental matrix Given 2 images with unknown camera matrices We can find F by 7 point correspondences (gives us 7 equations for the coefficients of F) Practical algorithms find many more correspondences and find a least-squares solution – more robust

Generating novel views C3 e23 e21 e23 e12 e31 e13 C2 C2 C1 We have three cameras C1, C2, C3 eij denotes intersection of CiCj with the image plane of Ci

Generating novel views C3 p’’ e23 e21 P e23 e12 p’ p e31 e13 C2 C2 C1 The image of P in the 3rd camera is the intersection of the epipolar lines from C1, C2

The algorithm p’ p p’’ Given: Fundamental matrices Point correspondences between the two given images This is found using “optical flow” algorithm For each correspondence p  p’ find the position of the corresponding point p’’ in the third view Copy the intensity to p’’ p’ p p’’

Generating novel views Denote the fundamental matrix from Cj to Ci by Fij (i, j = 1, 2, 3) The intersection of the epipolar lines is given by: p’’ = (F31 p)(F32 p’)

Degeneracies Can’t reproject points on the plane of C1, C2 , C3 (no proper intersection of the epipolar lines) Numerical problems for all points close to that plane Epipolar geometry is not defined when Ci = Ci (only rotational movement of the camera) Can’t reconstruct third view if C1, C2 , C3 are on one line

Good luck!