Single View Metrology A. Criminisi, I. Reid, and A. Zisserman University of Oxford IJCV Nov 2000 Presentation by Kenton Anderson CMPT820 March 24, 2005

Single View Metrology 2 Overview Introduction Introduction Geometry Geometry Algebraic Representation Algebraic Representation Uncertainty Analysis Uncertainty Analysis Applications Applications Conclusions Conclusions

March 24, 2005 Single View Metrology 3 Problem Is it possible to extract 3D geometric information from single images? Is it possible to extract 3D geometric information from single images?YES How? How? Why? Why?

March 24, 2005 Single View Metrology 4 Background 2D 3D Optical centre Painter, Linear perspective Real or imaginary object Painting Camera, Laws of Optics Real object Photograph Architect, Descriptive Geometry A mental model Drawing Projective Geometry Reconstructed 3D model Flat image

March 24, 2005 Single View Metrology 5 Introduction 3D affine measurements may be measured from a single perspective image 3D affine measurements may be measured from a single perspective image

March 24, 2005 Single View Metrology 6 Introduction 1. Measurements of the distance between any of the planes 2. Measurements on these planes 3. Determine the camera’s position Results are sufficient for a partial or complete 3D reconstruction of the observed scene Results are sufficient for a partial or complete 3D reconstruction of the observed scene

March 24, 2005 Single View Metrology 7 Sample La Flagellazione di Cristo

March 24, 2005 Single View Metrology 8 Geometry Overview Overview Measurements between parallel lines Measurements between parallel lines Measurements on parallel planes Measurements on parallel planes Determining the camera position Determining the camera position

March 24, 2005 Single View Metrology 9 Geometry Overview Possible to obtain geometric interpretations for key features in a scene Possible to obtain geometric interpretations for key features in a scene Derive how 3D affine measurements may be extracted from the image Derive how 3D affine measurements may be extracted from the image Use results to analyze and/or model the scene Use results to analyze and/or model the scene

March 24, 2005 Single View Metrology 10 Assumptions Assume that images are obtained by perspective projection Assume that images are obtained by perspective projection Assume that, from the image, a: Assume that, from the image, a: vanishing line of a reference planevanishing line of a reference plane vanishing point of another reference directionvanishing point of another reference direction may be determined from the image

March 24, 2005 Single View Metrology 11 Geometric Cues Vanishing Line ℓ Vanishing Line ℓ Projection of the line at infinity of the reference plane into the imageProjection of the line at infinity of the reference plane into the image

March 24, 2005 Single View Metrology 12 Geometric Cues Vanishing Point(s) v Vanishing Point(s) v A point at infinity in the reference directionA point at infinity in the reference direction Reference direction is NOT parallel to reference planeReference direction is NOT parallel to reference plane Also known as the vertical vanishing pointAlso known as the vertical vanishing point

March 24, 2005 Single View Metrology 13 Vanishing point Vanishing line Vanishing point Vertical vanishing point (at infinity) Geometric Cues

March 24, 2005 Single View Metrology 14 Generic Algorithm 1)Edge detection and straight line fitting to obtain the set of straight edge segments S A 2)Repeat a)Randomly select two segments s 1, s 2 € S A and intersect them to give point p b)The support set S p is the set of straight edges in S A going through point p 3)Set the dominant vanishing point as the point p with the largest support S p 4)Remove all edges in S p from S A and repeat step 2

March 24, 2005 Single View Metrology 15 Automatic estimation of vanishing points and lines RANSAC algorithm Candidate vanishing point

March 24, 2005 Single View Metrology 16 Automatic estimation of vanishing points and lines

March 24, 2005 Single View Metrology 17 Measurements between Parallel Lines Wish to measure the distance between two parallel planes, in the reference direction Wish to measure the distance between two parallel planes, in the reference direction The aim is to compute the height of an object relative to a referenceThe aim is to compute the height of an object relative to a reference

March 24, 2005 Single View Metrology 18 Cross Ratio Point b on plane ∏ correspond to point t on plane ∏’ Point b on plane ∏ correspond to point t on plane ∏’ Aligned to vanishing point v Aligned to vanishing point v Point i is the intersection with the vanishing line Point i is the intersection with the vanishing line

March 24, 2005 Single View Metrology 19 Cross Ratio The cross ratio is between the points provides an affine length ratio The cross ratio is between the points provides an affine length ratio The value of the cross ratio determines a ratio of distances between planes in the worldThe value of the cross ratio determines a ratio of distances between planes in the world Thus, if we know the length for an object in the scene, we can use it as a reference to calculate the length of other objects Thus, if we know the length for an object in the scene, we can use it as a reference to calculate the length of other objects

March 24, 2005 Single View Metrology 20 Estimating Height The distance || t r – b r || is known Used to estimate the height of the man in the scene

March 24, 2005 Single View Metrology 21 Measurements on Parallel Planes If the reference plane is affine calibrated, then from the image measurements the following can be computed: If the reference plane is affine calibrated, then from the image measurements the following can be computed: i.Ratios of lengths of parallel line segments on the plane ii.Ratios of areas on the plane

March 24, 2005 Single View Metrology 22 Parallel Line Segments Basis points are manually selected and measured in the real world Using ratios of lengths, the size of the windows are calculated

March 24, 2005 Single View Metrology 23 Planar Homology Using the same principals, affine measurements can be made on two separate planes, so long as the planes are parallel to each other Using the same principals, affine measurements can be made on two separate planes, so long as the planes are parallel to each other A map in the world between parallel planes induces a map between images of points on the two planes A map in the world between parallel planes induces a map between images of points on the two planes

March 24, 2005 Single View Metrology 24 Homology Mapping between Parallel Planes A point X on plane ∏ is mapped into the point X’ on ∏’ by a parallel projection A point X on plane ∏ is mapped into the point X’ on ∏’ by a parallel projection

March 24, 2005 Single View Metrology 25 Planar Homology Points in one plane are mapped into the corresponding points in the other plane as follows: Points in one plane are mapped into the corresponding points in the other plane as follows: X’ = HX where (in homogeneous coordinates): X is an image pointX is an image point X’ is its corresponding pointX’ is its corresponding point H is the 3 x 3 matrix representing the homography transformationH is the 3 x 3 matrix representing the homography transformation

March 24, 2005 Single View Metrology 26 Measurements on Parallel Planes This means that we can compare measurements made on two separate planes by mapping between the planes in the reference direction via the homology This means that we can compare measurements made on two separate planes by mapping between the planes in the reference direction via the homology

March 24, 2005 Single View Metrology 27 Parallel Line Segments lying on two Parallel Planes

March 24, 2005 Single View Metrology 28 Camera Position Using the techniques we developed in the previous sections, we can: Using the techniques we developed in the previous sections, we can: Determine the distance of the camera from the sceneDetermine the distance of the camera from the scene Determine the height of the camera relative to the reference planeDetermine the height of the camera relative to the reference plane

March 24, 2005 Single View Metrology 29 Camera Distance from Scene In Measurements between Parallel Lines, distances between planes are computed as a ratio relative to the camera’s distance from the reference plane In Measurements between Parallel Lines, distances between planes are computed as a ratio relative to the camera’s distance from the reference plane Thus we can compute the camera’s distance from a particular frame knowing a single reference distance Thus we can compute the camera’s distance from a particular frame knowing a single reference distance

March 24, 2005 Single View Metrology 30 Camera Position Relative to Reference Plane The location of the camera relative to the reference plane is the back-projection of the vanishing point onto the reference plane The location of the camera relative to the reference plane is the back-projection of the vanishing point onto the reference plane

March 24, 2005 Single View Metrology 31 Algebraic Representation Overview Overview Measurements between parallel lines Measurements between parallel lines Measurements on parallel planes Measurements on parallel planes Determining the camera position Determining the camera position

March 24, 2005 Single View Metrology 32 Overview Algebraic approach offers many advantages (over direct geometry): Algebraic approach offers many advantages (over direct geometry): 1.Avoid potential problems with ordering for the cross ratio 2.Minimal and over-constrained configurations can be dealt with uniformly 3.Unifies the different types of measurements 4.Are able to develop an uncertainty analysis

March 24, 2005 Single View Metrology 33 Coordinate Systems Define an affine coordinate system XYZ in space Define an affine coordinate system XYZ in space Origin lies on reference planeOrigin lies on reference plane X, Y axes span the reference planeX, Y axes span the reference plane Z axis is the reference directionZ axis is the reference direction Define image coordinate system xy Define image coordinate system xy y in the vertical directiony in the vertical direction x in the horizontal directionx in the horizontal direction

March 24, 2005 Single View Metrology 34 Coordinate Systems

March 24, 2005 Single View Metrology 35 Projection Matrix If X’ is a point in world space, it is projected to an image point x’ in image space via a 3 x 4 projection matrix P If X’ is a point in world space, it is projected to an image point x’ in image space via a 3 x 4 projection matrix P x’ = PX’ = [ p 1 p 2 p 3 p 4 ]X’ x’ = PX’ = [ p 1 p 2 p 3 p 4 ]X’ where x’ and X’ are homogeneous vectors: x’ = (x, y,w) and X’ = (X, Y, Z, W)

March 24, 2005 Single View Metrology 36 Vanishing Points Denote the vanishing points for the X, Y and Z directions as v X, v Y, and v Denote the vanishing points for the X, Y and Z directions as v X, v Y, and v By inspection, the first 3 columns of matrix P are the vanishing points: By inspection, the first 3 columns of matrix P are the vanishing points: p 1 = v Xp 1 = v X p 2 = v Yp 2 = v Y p 3 = vp 3 = v Origin of the world coordinate system is p 4 Origin of the world coordinate system is p 4

March 24, 2005 Single View Metrology 37 Vanishing Line Furthermore, v X and v Y are on the vanishing line l Furthermore, v X and v Y are on the vanishing line l Choosing these points fixes the X and Y affine coordinate axesChoosing these points fixes the X and Y affine coordinate axes Denote them as l 1, l 2 where l i · l = 0Denote them as l 1, l 2 where l i · l = 0 Note: Note: Columns 1, 2 and 4 make up the reference plane to image homography matrix HColumns 1, 2 and 4 make up the reference plane to image homography matrix H TTT

March 24, 2005 Single View Metrology 38 Projection Matrix Redux o = p4 = l /|| l || = l ^ o = p4 = l /|| l || = l ^ o is the Origin of the coordinate systemo is the Origin of the coordinate system Thus, the parametrization of P is: Thus, the parametrization of P is: P = [ l 1 l 2 α v l ^ ] TT α is the affine scale factor

March 24, 2005 Single View Metrology 39 Measurements between Parallel Lines The aim is to compute the height of an object relative to a reference The aim is to compute the height of an object relative to a reference Height is measured in the Z direction Height is measured in the Z direction

March 24, 2005 Single View Metrology 40 Measurements between Parallel Lines Base point B on the reference plane Base point B on the reference plane Top point T in the scene Top point T in the scene b = n(Xp 1 + Yp 2 + p 4 ) t = m(Xp 1 + Yp 2 + Zp 3 + p 4 ) n and m are unknown scale factors

March 24, 2005 Single View Metrology 41 Affine Scale Factor If α is known, then we can obtain Z If α is known, then we can obtain Z If Z is known, we can compute α, removing affine ambiguity If Z is known, we can compute α, removing affine ambiguity

March 24, 2005 Single View Metrology 42 Representation

March 24, 2005 Single View Metrology 43 Measurements on Parallel Planes Projection matrix P from the world to the image is defined with respect to a coordinate frame on the reference plane Projection matrix P from the world to the image is defined with respect to a coordinate frame on the reference plane The translation from the reference plane to another plane along the reference direction can be parametrized into a new projection matrix P’ The translation from the reference plane to another plane along the reference direction can be parametrized into a new projection matrix P’

March 24, 2005 Single View Metrology 44 Plane to Image Homographies P = [ l 1 l 2 α v l ^ ] P’ = [ l 1 l 2 α v αZ v + l ^ ] where Z is the distance between the planes TT TT

March 24, 2005 Single View Metrology 45 Plane to Image Homographies Homographies can be extracted: Homographies can be extracted: H = [ p 1 p 2 αZ v + l ^ ] H’ = [ p 1 p 2 l ^ ] Then H” = H’H -1 maps points from the reference plane to the second plane, and so defines the homology Then H” = H’H -1 maps points from the reference plane to the second plane, and so defines the homology

March 24, 2005 Single View Metrology 46 Generic Algorithm 1. Given an image of a planar surface estimate the image-to-world homography matrix H 2. Repeat a)Select two points x 1 and x 2 on the image plane b)Back-project each image point into the world plane using H to obtain the two world points X 1 and X 2 c)Compute the Euclidean distance dist(X 1, X 2 ) i. dist(A, B) = || A – B ||

March 24, 2005 Single View Metrology 47 Camera Position Camera position C = (X c, Y c, Z c, W c ) Camera position C = (X c, Y c, Z c, W c ) PC = 0 PC = 0 Implies: Implies: Using Cramer’s Rule: Using Cramer’s Rule:

March 24, 2005 Single View Metrology 48 Camera In Scene

March 24, 2005 Single View Metrology 49 Uncertainty Analysis Errors arise from the finite accuracy of the feature detection and extraction Errors arise from the finite accuracy of the feature detection and extraction ie- edge detection, point specificationsie- edge detection, point specifications Uncertainty analysis attempts to quantify this error Uncertainty analysis attempts to quantify this error

March 24, 2005 Single View Metrology 50 Uncertainty Analysis Uncertainty in Uncertainty in Projection matrix PProjection matrix P Top point tTop point t Base point bBase point b Location of vanishing line lLocation of vanishing line l Affine scale factor αAffine scale factor α As the number of reference distances increases, so the uncertainty decreases As the number of reference distances increases, so the uncertainty decreases

March 24, 2005 Single View Metrology 51 Illustration Ellipses are user specified Ellipses are user specified t and b are then aligned to the vertical vanishing point Alignment constraint v · (t X b) = 0

March 24, 2005 Single View Metrology 52 Applications Forensic Science Forensic Science Height of suspectHeight of suspect Virtual Modeling Virtual Modeling 3D reconstruction of a scene3D reconstruction of a scene Art History Art History Modeling paintingsModeling paintings

March 24, 2005 Single View Metrology 53 Forensic Science

March 24, 2005 Single View Metrology 54 Virtual Modeling

March 24, 2005 Single View Metrology 55 Art History

March 24, 2005 Single View Metrology 56 Conclusions Affine structure of 3D space may be partially recovered from perspective images Affine structure of 3D space may be partially recovered from perspective images Measurements between and on parallel planes can be determined Measurements between and on parallel planes can be determined Practical applications can be derived Practical applications can be derived