2. Computer Vision The 2D projective plane and it’s applications HZ Ch 2. In particular: Ch 2.1-4, 2.7, Szelisky: Ch 2.1.1, 2.1.2 Estimation:HZ: Ch 4.1-4.2.5, 4.4.4-4.8 cursorly Richard Hartley and Andrew Zisserman, Multiple View Geometry, Cambridge University Publishers, 2 nd ed. 2004

3. Homogeneous coordinates Homogeneous representation of 2D points and lines equivalence class of vectors, any vector is representative Set of all equivalence classes in R 3  (0,0,0) T forms P 2 The point x lies on the line l if and only if Homogeneous coordinates Inhomogeneous coordinates but only 2DOF Note that scale is unimportant for incidence relation

4. X Y Z Homogeneous coordinates  X Y Z s s  0 X∞X∞ X Y 0 X∞X∞ = Perspective imaging models 2d projective space Each 3D ray is a point in P 2 : homogeneous coords. Ideal points P 2 is R 2 plus a “line at infinity” l∞l∞ The 2D projective plane Projective point l∞l∞ x y X Y 1 Z = Inhomogeneous equivalent

5. Lines HZ Ideal line ~ the plane parallel to the image A B C l = X = l T X = X T l = AX + BY + CZ = 0 l∞l∞ = 0 0 1 Projective line ~ a plane through the origin For any 2d projective property, a dual property holds when the role of points and lines are interchanged. Duality: X Y 0 X∞X∞ = l1l1 l2l2 X= X1X1 X2X2 =l The line joining two pointsThe point joining two lines X Y Z X l “line at infinity”

6. Points from lines and vice-versa Intersections of lines The intersection of two lines and is Line joining two points The line through two points and is Example Note: with

7. Ideal points and the line at infinity Intersections of parallel lines Example Ideal points Line at infinity tangent vector normal direction Note that in P 2 there is no distinction between ideal points and others

8. Conics Curve described by 2 nd -degree equation in the plane or homogenized or in matrix form with 5DOF:

9. Five points define a conic For each point the conic passes through or stacking constraints yields

10. Tangent lines to conics The line l tangent to C at point x on C is given by l=Cx l x C

11. Dual conics A line tangent to the conic C satisfies Dual conics = line conics = conic envelopes In general ( C full rank):

12. Degenerate conics A conic is degenerate if matrix C is not of full rank e.g. two lines (rank 2) e.g. repeated line (rank 1) Degenerate line conics: 2 points (rank 2), double point (rank1) Note that for degenerate conics

13. Conics Conic:Conic: –Euclidean geometry: hyperbola, ellipse, parabola & degenerate –Projective geometry: equivalent under projective transform –Defined by 5 points Tangent lineTangent line Dual conic C*Dual conic C* inhomogeneous homogeneous

14. Projective transformations Homographies, collineations, projectivities Homographies, collineations, projectivities 3x3 nonsingular H 3x3 nonsingular H maps P 2 to P 2 8 degrees of freedom determined by 4 corresponding points Transforming Lines? subspaces preserved substitution dual transformation

15. Planar Projective Warping HZ A novel view rendered via four points with known structure

16. Planar Projective Warping HZ OriginalTop-downFacing right Artifacts are apparent where planarity is violated...

17. 2d Homographies 2 images of a plane 2 images from the same viewpoint (Perspectivity)

18. Panoramic imaging Appl: Quicktime VR, robot navigation etc. Homographies of the world, unite!

19. Image mosaics are stitched by homographies HZ

20. The line at infinity The line at infinity l  is a fixed line under a projective transformation H if and only if H is an affinity Note: But points on l  can be rearranged to new points on l 

21. Affine properties from images Projection (Imaging) Rectification Post-processing

22. Affine rectification v1v1 v2v2 l1l1 l2l2 l4l4 l3l3 l∞l∞ Point transformation for Aff Rect: Exercise: Verify

23. GroupTransformationInvariantsDistortion Projective 8 DOF Cross ratioCross ratio IntersectionIntersection TangencyTangency Affine 6 DOF ParallelismParallelism Relative dist in 1dRelative dist in 1d Line at infinityLine at infinity Metric 4 DOF Relative distances Relative distances Angles Angles Dual conic Dual conic Euclidean 3 DOF Lengths Lengths Areas Areas Geometric strata: 2d overview  C*C* 2 dof  l  l  C*C*

24. Parameter estimation in geometric transforms 2D homography2D homography Given a set of (x i,x i ’), compute H (x i ’=Hx i ) 3D to 2D camera projection3D to 2D camera projection Given a set of (X i,x i ), compute P (x i =PX i ) Fundamental matrixFundamental matrix Given a set of (x i,x i ’), compute F (x i ’ T Fx i =0) Trifocal tensorTrifocal tensor Given a set of (x i,x i ’,x i ”), compute T cs499 Useful in Grad research

25. Math tools 1: Solving Linear Systems If m = n ( A is a square matrix), then we can obtain the solution by simple inversion:If m = n ( A is a square matrix), then we can obtain the solution by simple inversion: If m > n, then the system is over-constrained and A is not invertibleIf m > n, then the system is over-constrained and A is not invertible –Use Matlab “\” to obtain least-squares solution x = A\b to Ax =b internally Matlab uses QR-factorization (cmput418/340) to solve this. –Can also write this using pseudoinverse A + = (A T A) -1 A T to obtain least-squares solution x = A + b

26. Fitting Lines A 2-D point x = (x, y) is on a line with slope m and intercept b if and only if y = mx + bA 2-D point x = (x, y) is on a line with slope m and intercept b if and only if y = mx + b Equivalently,Equivalently, So the line defined by two points x 1, x 2 is the solution to the following system of equations:So the line defined by two points x 1, x 2 is the solution to the following system of equations:

27. Fitting Lines With more than two points, there is no guarantee that they will all be on the same lineWith more than two points, there is no guarantee that they will all be on the same line Least-squares solution obtained from pseudoinverse is line that is “closest” to all of the pointsLeast-squares solution obtained from pseudoinverse is line that is “closest” to all of the points courtesy of Vanderbilt U.

28. Example: Fitting a Line Suppose we have points (2, 1), (5, 2), (7, 3), and (8, 3)Suppose we have points (2, 1), (5, 2), (7, 3), and (8, 3) ThenThen and x = A + b = (0.3571, 0.2857) T Matlab: x = A\b

29. Example: Fitting a Line

30. Homogeneous Systems of Equations Suppose we want to solve A x = 0Suppose we want to solve A x = 0 There is a trivial solution x = 0, but we don’t want this. For what other values of x is A x close to 0 ?There is a trivial solution x = 0, but we don’t want this. For what other values of x is A x close to 0 ? This is satisfied by computing the singular value decomposition (SVD) A = UDV T (a non-negative diagonal matrix between two orthogonal matrices) and taking x as the last column of VThis is satisfied by computing the singular value decomposition (SVD) A = UDV T (a non-negative diagonal matrix between two orthogonal matrices) and taking x as the last column of V –Note that Matlab returns [U, D, V] = svd(A)

31. Line-Fitting as a Homogeneous System A 2-D homogeneous point x = (x, y, 1) T is on the line l = (a, b, c) T only whenA 2-D homogeneous point x = (x, y, 1) T is on the line l = (a, b, c) T only when ax + by + c = 0 ax + by + c = 0 We can write this equation with a dot product:We can write this equation with a dot product: x l = 0, and hence the following system is implied for multiple points x 1, x 2,..., x n : x l = 0, and hence the following system is implied for multiple points x 1, x 2,..., x n :

32. Example: Homogeneous Line-Fitting Again we have 4 points, but now in homogeneous form:Again we have 4 points, but now in homogeneous form: (2, 1, 1), (5, 2, 1), (7, 3, 1), and (8, 3, 1) (2, 1, 1), (5, 2, 1), (7, 3, 1), and (8, 3, 1) Our system is:Our system is: Taking the SVD of A, we get:Taking the SVD of A, we get: compare to x = (0.3571, 0.2857) T

33. Parameter estimation in geometric transforms 2D homography2D homography Given a set of (x i,x i ’), compute H (x i ’=Hx i ) 3D to 2D camera projection3D to 2D camera projection Given a set of (X i,x i ), compute P (x i =PX i ) Fundamental matrixFundamental matrix Given a set of (x i,x i ’), compute F (x i ’ T Fx i =0) Trifocal tensorTrifocal tensor Given a set of (x i,x i ’,x i ”), compute T cs499 Useful in Grad research

34. Estimating Homography H given image points x HZ A novel view rendered via four points with known structure

35. Number of measurements required At least as many independent equations as degrees of freedom requiredAt least as many independent equations as degrees of freedom required Example:Example: 2 independent equations / point 8 degrees of freedom 4x2≥8

36. Approximate solutions Minimal solutionMinimal solution 4 points yield an exact solution for H More pointsMore points –No exact solution, because measurements are inexact (“noise”) –Search for “best” according to some cost function –Algebraic or geometric/statistical cost

37. Many ways to solve: Different Cost functions => differences in solution Algebraic distanceAlgebraic distance Geometric distanceGeometric distance Reprojection errorReprojection error ComparisonComparison Geometric interpretationGeometric interpretation

38. Gold Standard algorithm Cost function that is optimal for some assumptionsCost function that is optimal for some assumptions Computational algorithm that minimizes it is called “Gold Standard” algorithmComputational algorithm that minimizes it is called “Gold Standard” algorithm Other algorithms can then be compared to itOther algorithms can then be compared to it

39. Estimating H : The Direct Linear Transformation (DLT) Algorithm x i =HX i is an equation involving homogeneous vectors, so HX i and x i need only be in the same direction, not strictly equalx i =HX i is an equation involving homogeneous vectors, so HX i and x i need only be in the same direction, not strictly equal We can specify “same directionality” by using a cross product formulation:We can specify “same directionality” by using a cross product formulation:

40. Direct Linear Transformation (DLT)

41. Equations are linear in hEquations are linear in h Only 2 out of 3 are linearly independentOnly 2 out of 3 are linearly independent (indeed, 2 eq/pt) (only drop third row if w i ’≠0) Holds for any homogeneous representation, e.g. ( x i ’, y i ’,1)Holds for any homogeneous representation, e.g. ( x i ’, y i ’,1)

42. Direct Linear Transformation (DLT) Solving for HSolving for H size A is 8x9 or 12x9, but rank 8 Trivial solution is h =0 9 T is not interesting 1-D null-space yields solution of interest pick for example the one with

43. Direct Linear Transformation (DLT) Over-determined solutionOver-determined solution No exact solution because of inexact measurement i.e. “noise” Find approximate solution - Additional constraint needed to avoid 0, e.g. - not possible, so minimize

44. DLT algorithm Objective Given n≥4 2D to 2D point correspondences {x i ↔x i ’}, determine the 2D homography matrix H such that x i ’=Hx i Algorithm (i)For each correspondence x i ↔x i ’ compute A i. Usually only two first rows needed. (ii)Assemble n 2x9 matrices A i into a single 2 n x9 matrix A (iii)Obtain SVD of A. Solution for h is last column of V (iv)Determine H from h (reshape)

45. Inhomogeneous solution Since h can only be computed up to scale, pick h j =1, e.g. h 9 =1, and solve for 8-vector Solve using Gaussian elimination (4 points) or using linear least-squares (more than 4 points) However, if h 9 =0 this approach fails also poor results if h 9 close to zero Therefore, not recommended for general homographies Note h 9 =H 33 =0 if origin is mapped to infinity

46. Normalizing transformations Since DLT is not invariant to coordinate transforms, what is a good choice of coordinates?Since DLT is not invariant to coordinate transforms, what is a good choice of coordinates?e.g. –Translate centroid to origin –Scale to a average distance to the origin –Independently on both images Or

47. Normalized DLT algorithm Objective Given n≥4 2D to 2D point correspondences {x i ↔x i ’}, determine the 2D homography matrix H such that x i ’=Hx i Algorithm (i)Normalize points (ii)Apply DLT algorithm to (iii)Denormalize solution

48. Importance of normalization ~10 2 ~10 4 ~10 2 1 1 orders of magnitude difference!

49. Degenerate configurations x4x4 x1x1 x3x3 x2x2 x4x4 x1x1 x3x3 x2x2 H? H’? x1x1 x3x3 x2x2 x4x4 Constraints: i =1,2,3,4 Define: Then, H * is rank-1 matrix and thus not a homography (case A) (case B) If H * is unique solution, then no homography mapping x i →x i ’(case B) If further solution H exist, then also αH * +βH (case A) (2-D null-space in stead of 1-D null-space)

50. First 3D proj geomFirst 3D proj geom Then review and more on camera modelsThen review and more on camera models Then following P estimationThen following P estimation

51. Projection equationProjection equation x i =P i X x i =P i X Resection:Resection: –x i,X P i A 3D Vision Problem: Multi-view geometry - resection Given image points and 3D points calculate camera projection matrix.

52. Estimating camera matrix P Given a number of correspondences between 3- D points and their 2-D image projections X i  x i, we would like to determine the camera projection matrix P such that x i = PX i for all iGiven a number of correspondences between 3- D points and their 2-D image projections X i  x i, we would like to determine the camera projection matrix P such that x i = PX i for all i

53. A Calibration Target courtesy of B. Wilburn XZ Y XiXi xixi

54. Estimating P : The Direct Linear Transformation (DLT) Algorithm x i = PX i is an equation involving homogeneous vectors, so PX i and x i need only be in the same direction, not strictly equalx i = PX i is an equation involving homogeneous vectors, so PX i and x i need only be in the same direction, not strictly equal We can specify “same directionality” by using a cross product formulation:We can specify “same directionality” by using a cross product formulation:

55. DLT Camera Matrix Estimation: Preliminaries Let the image point x i = (x i, y i, w i ) T (remember that X i has 4 elements)Let the image point x i = (x i, y i, w i ) T (remember that X i has 4 elements) Denoting the j th row of P by p jT (a 4-element row vector), we have:Denoting the j th row of P by p jT (a 4-element row vector), we have:

56. DLT Camera Matrix Estimation: Step 1 Then by the definition of the cross product,Then by the definition of the cross product, x i  PX i is: x i  PX i is:

57. DLT Camera Matrix Estimation: Step 2 The dot product commutes, so p jT X i = X T i p j, and we can rewrite the preceding as:The dot product commutes, so p jT X i = X T i p j, and we can rewrite the preceding as:

58. DLT Camera Matrix Estimation: Step 3 Collecting terms, this can be rewritten as a matrix product:Collecting terms, this can be rewritten as a matrix product: where 0 T = (0, 0, 0, 0). This is a 3 x 12 matrix times a 12-element column vector p = (p 1T, p 2T, p 3T ) T

59. What We Just Did

60. DLT Camera Matrix Estimation: Step 4 There are only two linearly independent rows hereThere are only two linearly independent rows here –The third row is obtained by adding x i times the first row to y i times the second and scaling the sum by -1/w i

61. DLT Camera Matrix Estimation: Step 4 So we can eliminate one row to obtain the following linear matrix equation for the i th pair of corresponding points:So we can eliminate one row to obtain the following linear matrix equation for the i th pair of corresponding points: Write this as A i p = 0Write this as A i p = 0

62. DLT Camera Matrix Estimation: Step 5 Remember that there are 11 unknowns which generate the 3 x 4 homogeneous matrix P (represented in vector form by p )Remember that there are 11 unknowns which generate the 3 x 4 homogeneous matrix P (represented in vector form by p ) Each point correspondence yields 2 equations (the two row of A i )Each point correspondence yields 2 equations (the two row of A i )  We need at least 5 ½ point correspondences to solve for p Stack A i to get homogeneous linear system A p = 0Stack A i to get homogeneous linear system A p = 0

63. Experiment:

64. short and long focal length Radial Distortion

67. Correction of distortion Choice of the distortion function and center Computing the parameters of the distortion function (i)Minimize with additional unknowns (ii)Straighten lines (iii)… Radial Distortion