Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Algorithms Point correspondences –Salient point detection –Local descriptors Matrix decompositions –RQ decomposition –Singular value decomposition - SVD Estimation –Systems of linear equations –Solving systems of linear equations Direct Linear Transform – DLT Normalization Iterative error / cost minimization Outliers Robustness, RANSAC –Pose estimation Perspective n-point problem – PnP

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Point Correspondences - Example 1 Structure and motion from natural landmarks [Schweighofer] Stereo reconstruction of Harris corners Stereo reconstruction of Harris corners

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Point Correspondences - Example 2 [Mikolajczyk+Schmid] normalization canonical view elliptical support correspondence

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) based on 1 st derivatives Autocorrelation of 2D image signal [Moravec] –Approximation by sum of squared differences (SSD) –Window W –Differences between grayvalues in W and a window shifted by (Δx,Δy) –Four different shift directions f i (x,y): –A corner is detected, when f Moravec >th

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) based on 1 st derivatives Autocorrelation (second moment) matrix: –Avoids various shift directions –Approximate I(x w +Δx,y w +Δy) by Taylor expansion: –Rewrite f(x,y): second moment matrix M

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) based on 1 st derivatives Autocorrelation (second moment) matrix: –M can be used to derive a measure of cornerness –Independent of various displacements (Δx,Δy) –Corner: significant gradients in >1 directions rank M = 2 –Edge: significant gradient in 1 direction rank M = 1 –Homogeneous region rank M = 0 Several variants of this corner detector: –KLT corners, Förstner corners

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) based on 1 st derivatives Harris corners –Most popular variant of a detector based on M –Local derivatives with derivation scale σ D –Convolution with a Gaussian with integration scale σ I –M Harris for each point x in the image –Cornerness c Harris does not require to compute eigenvalues –Corner detection: c Harris > t Harris

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) based on 1 st derivatives Harris corners

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) based on 2 nd derivatives Hessian determinant –Local maxima of det H [Beaudet] –Zero crossings of det H [Dreschler+Nagel] –Detectors are related to curvature –Invariant to rotation –Similar cornerness measure: local maxima of K [Kitchen+Rosenfeld]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) based on 2 nd derivatives DoG / LoG [Marr+Hildreth] –Zero crossings –Mexican hat, Sombrero –Edge detector ! Lowes DoG keypoints [Lowe] –Edge zero-crossing –Blob at corresponding scale: local extremum ! –Low contrast corner suppression: threshold –Assess curvature distinguish corners from edges –Keypoint detection:

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) without derivatives Morphological corner detector [Laganière] –4 structuring elements: +,, x, –Assymetrical closing

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) without derivatives SUSAN corners [Smith+Brady] –Sliding window –Faster than Harris

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) without derivatives Kadir/Brady saliency [Kadir+Brady] –Histograms –Shannon entropy –Scale selection –Used in constellation model [Fergus et al.]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Salient points (corners) without derivatives MSER – maximally stable extremal regions [Matas et al.] –Successive thresholds –Stability: regions survive over many thresholds

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Affine covariant corner detectors Locally planar patch affine distortion Detect characteristic scale –see also [Lindeberg], scale-space Recover affine deformation that fits local image data best [Mikolajczyk+Schmid] normalization canonical view elliptical support correspondence

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Scaled Harris Corner Detector Harris Laplace [Mikolajczyk+Schmid, Mikolajczyk et al.]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Scaled Hessian Detector Hessian Laplace [Mikolajczyk+Schmid, Mikolajczyk et al.]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Harris Affine Detector Harris affine [Mikolajczyk+Schmid, Mikolajczyk et al.]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Hessian Affine Detector Hessian affine [Mikolajczyk+Schmid, Mikolajczyk et al.]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Qualitative comparison of detectors (1) Harris Harris affineHarris Laplace Hessian affineHessian Laplace

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Qualitative comparison of detectors (2) Kadir/Brady morphological MSERSUSAN

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Descriptors (1) Representation of salient regions descriptive features feature vector There are many possibilities ! Categorization vs. specific OR, matching –Sufficient descriptive power –Not too much emphasis on specific individuals –Performance is often category-specific feature vector extracted from patch P n vs. AR ?

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Descriptors (2) Grayvalues –Raw pixel values of a patch P –local appearance-based description –local affine frame LAF [Obdržálek+Matas] for MSER General moments of order p+q: Moment invariants: –Central moments μ pq : invariant to translation

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Descriptors (3) Moment invariants: –Normalized central moments –Translation, rotation, scale invariant moments Φ1... Φ7 [Hu] –Geometric/photometric, color invariants [vanGool et al.] Filters –local jets [Koenderink+VanDoorn] –Gabor banks, steerable filters, discrete cosine transform DCT

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Descriptors (4) SIFT descriptors [Lowe] –Scale invariant feature transform –Calculated for local patch P: 8x8 or 16x16 pixels –Subdivision into 4x4 sample regions –Weighted histogram of 8 gradient directions: 0º, 45º, … –SIFT vector dimension: 128 for a 16x16 patch [Lowe]

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Algorithms Point correspondences –Salient point detection –Local descriptors Matrix decompositions –RQ decomposition –Singular value decomposition - SVD Estimation –Systems of linear equations –Solving systems of linear equations Direct Linear Transform – DLT Normalization Iterative error / cost minimization Outliers Robustness, RANSAC –Pose estimation Perspective n-point problem – PnP

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz RQ Decomposition (1) Remember camera projection matrix P P can be decomposed, e.g. finite projective camera

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz RQ Decomposition (2) Unfortunately: R refers to upper triangular, Q to rotation… Givens rotations: How to decompose a given 3 x 3 matrix (say M) ? –MQ x enforcing M 32 = 0, first column of M unchanged, last two columns replaced by linear combinations of themselves –MQ x Q y enforcing M 31 = 0, 2nd column unchanged (M 32 remains 0) –MQ x Q y Q z enforcing M 21 = 0, first two columns replaced by linear combinations of themselves, thus M 31 and M 32 remain 0 MQ x Q y Q z = R, M = RQ x T Q y T Q z T, where R is upper triangular How to enforce e.g. M 21 = 0 ?

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Singular Value Decomposition - SVD Given a square matrix A (e.g. 3x3) A can be decomposed into where U and V are orthogonal matrices, and D is a diagonal matrix with –non-negative entries, –entries in descending order. the column of V corresponding to the smallest singular value the last column of V

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz SVD (2) SVD is also possible when A is non-square (e.g. m x n, mn) A can again be decomposed into where U is m x n with orthogonal columns (U T U=I nxn ), D is an n x n diagonal matrix with –non-negative entries, –entries in descending order, V is an n x n orthogonal matrix.

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz SVD for Least-Squares Solutions Overdetermined system of linear equations Find least-squares ( algebraic error! ) solution Algorithm: 1.Find the SVD 2.Set 3.Find 4.The solution is Even easier for Ax=0: x is the last column of V

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Algorithms Point correspondences –Salient point detection –Local descriptors Matrix decompositions –RQ decomposition –Singular value decomposition - SVD Estimation –Systems of linear equations –Solving systems of linear equations Direct Linear Transform – DLT Normalization Iterative error / cost minimization Outliers Robustness, RANSAC –Pose estimation Perspective n-point problem – PnP

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz (2n x 9) / (2n x 12) matrix representing correspondences Systems of Linear Equations (1) Estimation of –A homography H: –The fundamental matrix: –The camera projection matrix: By finding n point correspondences –between 2 images –between image and scene And solving a system of linear equations –Typical form: 9 - vector representing H, F 12 - vector representing P

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Systems of Linear Equations (2) How to obtain ? Homography H –3 x 3 matrix, 8 DoF, non-singular –at least 4 point correspondences are required Fundamental matrix F –3 x 3 matrix, 7 DoF, rank 2 –at least 7 point correspondences are required Camera projection matrix P –3 x 4 matrix, 11 DoF, decomposition into K, R, t –at least 5-1/2 (6) point correspondences are required why ?

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Homography Estimation (1) Equation defining the computation of H Point correspondences Some notation: Simple rewriting:

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Homography Estimation (2) A i is a 3 x 9 matrix, h is a 9-vector the system describes 3 equations the equations are linear in the unknown h elements of A i are quadratic in the known point coordinates only 2 equations are linearly independent thus, the 3rd equation is usually omitted [Sutherland 63]: A i is a 2 x 9 matrix, h is a 9-vector

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Homography Estimation (3) 1 point correspondence defines 2 equations H has 9 entries, but is defined up to scale 8 degrees of freedom at least 4 point correspondences needed 4 x 2 equations General case: overdetermined n point correspondences 2n equations, A is a 2n x 9 matrix

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Camera Projection Matrix Estimation homography H:projection matrix P very similar ! n point correspondences 2n equations that are linear in elements of P A is a 2n x 12 matrix, entries are quadratic in point coordinates p is a 12-vector P has only 11 degrees of freedom a minimum of 11 equations is required 5-1/2 (6) point correspondences

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Fundamental (Essential) Matrix Estimation (1) solving is different from solving each correspondence gives only one equation in the coefficients of F ! for n point matches we again obtain a set of linear equations (linear in f 1 -f 9 )

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Fundamental (Essential) Matrix Estimation (2) F is a 3 x 3 matrix, has rank 2, |F| = 0 F has only 7 degrees of freedom at least 7 point correspondences are required to estimate F Back to the solution of systems of linear equations ! similar systems of equations, but: different constraints

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz SVD for Least-Squares Solutions Overdetermined system of linear equations Find least-squares ( algebraic error! ) solution Algorithm: 1.Find the SVD 2.Set 3.Find 4.The solution is Even easier for Ax=0: x is the last column of V This is also called direct linear transform – DLT

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, Augmented Reality VU 3 Algorithms Axel Pinz Relevant Issues in Practice Poor condition of A Normalization Algebraic error vs. geometric error, Iterative minimization nonlinearities (lens dist.) Outliers Robust algorithms (RANSAC)