1. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 1 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Algorithms Point correspondences –Salient point detection –Local descriptors Matrix decompositions –RQ decomposition –Singular value decomposition - SVD Estimation –Systems of linear equations –Solving systems of linear equations Direct Linear Transform – DLT Normalization Iterative error / cost minimization Outliers Robustness, RANSAC –Pose estimation Perspective n-point problem – PnP

2. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 2 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Point Correspondences - Example 1 Structure and motion from natural landmarks [Schweighofer] Stereo reconstruction of Harris corners Stereo reconstruction of Harris corners

3. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 3 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Point Correspondences - Example 2 [Mikolajczyk+Schmid] normalization canonical view elliptical support correspondence

4. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 4 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) based on 1 st derivatives Autocorrelation of 2D image signal [Moravec] –Approximation by sum of squared differences (SSD) –Window W –Differences between grayvalues in W and a window shifted by (Δx,Δy) –Four different shift directions f i (x,y): –A corner is detected, when f Moravec >th

5. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 5 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) based on 1 st derivatives Autocorrelation (second moment) matrix: –Avoids various shift directions –Approximate I(x w +Δx,y w +Δy) by Taylor expansion: –Rewrite f(x,y): second moment matrix M

6. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 6 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) based on 1 st derivatives Autocorrelation (second moment) matrix: –M can be used to derive a measure of cornerness –Independent of various displacements (Δx,Δy) –Corner: significant gradients in >1 directions rank M = 2 –Edge: significant gradient in 1 direction rank M = 1 –Homogeneous region rank M = 0 Several variants of this corner detector: –KLT corners, Förstner corners

7. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 7 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) based on 1 st derivatives Harris corners –Most popular variant of a detector based on M –Local derivatives with derivation scale σ D –Convolution with a Gaussian with integration scale σ I –M Harris for each point x in the image –Cornerness c Harris does not require to compute eigenvalues –Corner detection: c Harris > t Harris

8. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 8 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) based on 1 st derivatives Harris corners

9. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 9 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) based on 2 nd derivatives Hessian determinant –Local maxima of det H [Beaudet] –Zero crossings of det H [Dreschler+Nagel] –Detectors are related to curvature –Invariant to rotation –Similar cornerness measure: local maxima of K [Kitchen+Rosenfeld]

10. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 10 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) based on 2 nd derivatives DoG / LoG [Marr+Hildreth] –Zero crossings –Mexican hat, Sombrero –Edge detector ! Lowes DoG keypoints [Lowe] –Edge zero-crossing –Blob at corresponding scale: local extremum ! –Low contrast corner suppression: threshold –Assess curvature distinguish corners from edges –Keypoint detection:

11. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 11 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) without derivatives Morphological corner detector [Laganière] –4 structuring elements: +,, x, –Assymetrical closing

12. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 12 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) without derivatives SUSAN corners [Smith+Brady] –Sliding window –Faster than Harris

13. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 13 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) without derivatives Kadir/Brady saliency [Kadir+Brady] –Histograms –Shannon entropy –Scale selection –Used in constellation model [Fergus et al.]

14. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 14 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) without derivatives MSER – maximally stable extremal regions [Matas et al.] –Successive thresholds –Stability: regions survive over many thresholds

15. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 15 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Affine covariant corner detectors Locally planar patch affine distortion Detect characteristic scale –see also [Lindeberg], scale-space Recover affine deformation that fits local image data best [Mikolajczyk+Schmid] normalization canonical view elliptical support correspondence

16. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 16 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Scaled Harris Corner Detector Harris Laplace [Mikolajczyk+Schmid, Mikolajczyk et al.]

17. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 17 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Scaled Hessian Detector Hessian Laplace [Mikolajczyk+Schmid, Mikolajczyk et al.]

18. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 18 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Harris Affine Detector Harris affine [Mikolajczyk+Schmid, Mikolajczyk et al.]

19. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 19 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Hessian Affine Detector Hessian affine [Mikolajczyk+Schmid, Mikolajczyk et al.]

20. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 20 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Qualitative comparison of detectors (1) Harris Harris affineHarris Laplace Hessian affineHessian Laplace

21. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 21 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Qualitative comparison of detectors (2) Kadir/Brady morphological MSERSUSAN

22. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 22 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Descriptors (1) Representation of salient regions descriptive features feature vector There are many possibilities ! Categorization vs. specific OR, matching –Sufficient descriptive power –Not too much emphasis on specific individuals –Performance is often category-specific feature vector extracted from patch P n vs. AR ?

23. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 23 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Descriptors (2) Grayvalues –Raw pixel values of a patch P –local appearance-based description –local affine frame LAF [Obdržálek+Matas] for MSER General moments of order p+q: Moment invariants: –Central moments μ pq : invariant to translation

24. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 24 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Descriptors (3) Moment invariants: –Normalized central moments –Translation, rotation, scale invariant moments Φ1... Φ7 [Hu] –Geometric/photometric, color invariants [vanGool et al.] Filters –local jets [Koenderink+VanDoorn] –Gabor banks, steerable filters, discrete cosine transform DCT

25. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 25 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Descriptors (4) SIFT descriptors [Lowe] –Scale invariant feature transform –Calculated for local patch P: 8x8 or 16x16 pixels –Subdivision into 4x4 sample regions –Weighted histogram of 8 gradient directions: 0º, 45º, … –SIFT vector dimension: 128 for a 16x16 patch [Lowe]

26. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 26 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Algorithms Point correspondences –Salient point detection –Local descriptors Matrix decompositions –RQ decomposition –Singular value decomposition - SVD Estimation –Systems of linear equations –Solving systems of linear equations Direct Linear Transform – DLT Normalization Iterative error / cost minimization Outliers Robustness, RANSAC –Pose estimation Perspective n-point problem – PnP

27. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 27 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz RQ Decomposition (1) Remember camera projection matrix P P can be decomposed, e.g. finite projective camera

28. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 28 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz RQ Decomposition (2) Unfortunately: R refers to upper triangular, Q to rotation… Givens rotations: How to decompose a given 3 x 3 matrix (say M) ? –MQ x enforcing M 32 = 0, first column of M unchanged, last two columns replaced by linear combinations of themselves –MQ x Q y enforcing M 31 = 0, 2nd column unchanged (M 32 remains 0) –MQ x Q y Q z enforcing M 21 = 0, first two columns replaced by linear combinations of themselves, thus M 31 and M 32 remain 0 MQ x Q y Q z = R, M = RQ x T Q y T Q z T, where R is upper triangular How to enforce e.g. M 21 = 0 ?

29. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 29 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Singular Value Decomposition - SVD Given a square matrix A (e.g. 3x3) A can be decomposed into where U and V are orthogonal matrices, and D is a diagonal matrix with –non-negative entries, –entries in descending order. the column of V corresponding to the smallest singular value the last column of V

30. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 30 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz SVD (2) SVD is also possible when A is non-square (e.g. m x n, mn) A can again be decomposed into where U is m x n with orthogonal columns (U T U=I nxn ), D is an n x n diagonal matrix with –non-negative entries, –entries in descending order, V is an n x n orthogonal matrix.

31. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 31 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz SVD for Least-Squares Solutions Overdetermined system of linear equations Find least-squares ( algebraic error! ) solution Algorithm: 1.Find the SVD 2.Set 3.Find 4.The solution is Even easier for Ax=0: x is the last column of V

32. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 32 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Algorithms Point correspondences –Salient point detection –Local descriptors Matrix decompositions –RQ decomposition –Singular value decomposition - SVD Estimation –Systems of linear equations –Solving systems of linear equations Direct Linear Transform – DLT Normalization Iterative error / cost minimization Outliers Robustness, RANSAC –Pose estimation Perspective n-point problem – PnP

33. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 33 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz (2n x 9) / (2n x 12) matrix representing correspondences Systems of Linear Equations (1) Estimation of –A homography H: –The fundamental matrix: –The camera projection matrix: By finding n point correspondences –between 2 images –between image and scene And solving a system of linear equations –Typical form: 9 - vector representing H, F 12 - vector representing P

34. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 34 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Systems of Linear Equations (2) How to obtain ? Homography H –3 x 3 matrix, 8 DoF, non-singular –at least 4 point correspondences are required Fundamental matrix F –3 x 3 matrix, 7 DoF, rank 2 –at least 7 point correspondences are required Camera projection matrix P –3 x 4 matrix, 11 DoF, decomposition into K, R, t –at least 5-1/2 (6) point correspondences are required why ?

35. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 35 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Homography Estimation (1) Equation defining the computation of H Point correspondences Some notation: Simple rewriting:

36. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 36 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Homography Estimation (2) A i is a 3 x 9 matrix, h is a 9-vector the system describes 3 equations the equations are linear in the unknown h elements of A i are quadratic in the known point coordinates only 2 equations are linearly independent thus, the 3rd equation is usually omitted [Sutherland 63]: A i is a 2 x 9 matrix, h is a 9-vector

37. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 37 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Homography Estimation (3) 1 point correspondence defines 2 equations H has 9 entries, but is defined up to scale 8 degrees of freedom at least 4 point correspondences needed 4 x 2 equations General case: overdetermined n point correspondences 2n equations, A is a 2n x 9 matrix

38. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 38 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Camera Projection Matrix Estimation homography H:projection matrix P very similar ! n point correspondences 2n equations that are linear in elements of P A is a 2n x 12 matrix, entries are quadratic in point coordinates p is a 12-vector P has only 11 degrees of freedom a minimum of 11 equations is required 5-1/2 (6) point correspondences

39. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 39 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Fundamental (Essential) Matrix Estimation (1) solving is different from solving each correspondence gives only one equation in the coefficients of F ! for n point matches we again obtain a set of linear equations (linear in f 1 -f 9 )

40. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 40 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Fundamental (Essential) Matrix Estimation (2) F is a 3 x 3 matrix, has rank 2, |F| = 0 F has only 7 degrees of freedom at least 7 point correspondences are required to estimate F Back to the solution of systems of linear equations ! similar systems of equations, but: different constraints

41. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 41 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz SVD for Least-Squares Solutions Overdetermined system of linear equations Find least-squares ( algebraic error! ) solution Algorithm: 1.Find the SVD 2.Set 3.Find 4.The solution is Even easier for Ax=0: x is the last column of V This is also called direct linear transform – DLT

42. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 42 15.4.2008 Augmented Reality VU 3 Algorithms Axel Pinz Relevant Issues in Practice Poor condition of A Normalization Algebraic error vs. geometric error, Iterative minimization nonlinearities (lens dist.) Outliers Robust algorithms (RANSAC)