1. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 1 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Algorithms Point correspondences –Salient point detection –Local descriptors Matrix decompositions –RQ decomposition –Singular value decomposition - SVD Estimation –Systems of linear equations –Solving systems of linear equations Direct Linear Transform – DLT Normalization Iterative error / cost minimization Outliers Robustness, RANSAC –Pose estimation Perspective n-point problem – PnP

2. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 2 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Relevant Issues in Practice Poor condition of A Normalization Algebraic error vs. geometric error, Iterative minimization nonlinearities (lens dist.) Outliers Robust algorithms (RANSAC)

3. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 3 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Normalization (1) Homography H: Entries of A are quadratic in point coordinates SVD is not robust against coordinate transform ! –change of coordinate system (translation, scaling) will influence result ! –algebraic vs. geometric error ! Normalization recommended, e.g.: –translate origin (0,0,1) image center –isotropic scaling such that: either average distance to (0,0,1) is, or average point is (1,1,1)

4. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 4 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Normalization (2) Fundamental Matrix F: poor condition of A T A Normalization is mandatory normalized 8-point algorithm to estimate F [Hartley95] in defense of the 8-point algorithm note: some algorithms use eigenvalues of A T A instead of singular values (SVD) !

5. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 5 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Iterative Minimization DLT minimizes algebraic error geometric distance is more complex Lens distortion is non-linear Standard approach: –estimate linear parameters by DLT initialization for –subsequent iterative minimization over all parameters E.g.: gold standard for estimation of H

6. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 6 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Gold Standard for Estimation of H (1) [Hartley+Zisserman] Objective Given n4 2D to 2D point correspondences {x i x i }, determine the 2D homography matrix H such that x i =Hx i Algorithm (i) (i)For each correspondence x i x i compute A i. Usually only two first rows needed. (ii) (ii)Assemble n 2x9 matrices A i into a single 2 n x9 matrix A (iii) (iii)Obtain SVD of A. Solution for h is last column of V (iv) (iv)Determine H from h [adapted from Pollefeys course] DLT algorithm to estimate H:

7. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 7 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Gold Standard for Estimation of H (2) [Hartley+Zisserman] [adapted from Pollefeys course] normalized DLT algorithm to estimate H: Objective Given n4 2D to 2D point correspondences {x i x i }, determine the 2D homography matrix H such that x i =Hx i Algorithm (i) (i)Normalize points (ii) (ii)Apply DLT algorithm to (iii) (iii)Denormalize solution

8. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 8 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Gold Standard for Estimation of H (3) [Hartley+Zisserman] [adapted from Pollefeys course] Objective Given n4 2D to 2D point correspondences {x i x i }, determine the Maximum Likelihood Estimation of H Algorithm (i) (i)Initialization: compute an initial estimate using normalized DLT or RANSAC (ii) (ii)Geometric minimization of either Sampson error: Minimize the Sampson error Minimize using Levenberg-Marquardt over 9 entries of h or Gold Standard error: compute initial estimate for subsidiary minimize cost over if many points, use sparse method

9. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 9 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Robust Estimation (RANSAC) [Hartley+Zisserman] Handling of outliers ! RANSAC = RANdom Sample Consensus

10. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 10 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz RANSAC Algorithm [Hartley+Zisserman] Objective Robust fit of model to data set S which contains outliers Algorithm (i) (i)Randomly select a sample of s data points from S and instantiate the model from this subset. (ii) (ii)Determine the set of data points S i which are within a distance threshold t of the model. The set S i is the consensus set of samples and defines the inliers of S. (iii) (iii)If the subset of S i is greater than some threshold T, re- estimate the model using all the points in S i and terminate (iv) (iv)If the size of S i is less than T, select a new subset and repeat the above. (v) (v)After N trials the largest consensus set S i is selected, and the model is re-estimated using all the points in the subset S i [adapted From Pollefeys course]

11. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 11 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz RANSAC Algorithm [Hartley+Zisserman] proportion of outliers e s5%10%20%25%30%40%50% 2235671117 33479111935 435913173472 54612172657146 64716243797293 748203354163588 8592644782721177 sample size vs. proportion of outliers: [adapted from Pollefeys course]

12. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 12 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz More Problems critical cases ! e.g. [Torr+Murray, IJCV 1997]

13. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 13 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Pose Estimation X Y Z calibrated camera, known K C xCxC yCyC zCzC R, t Camera xVxV yVyV zVzV Visualization (screen, HMD) R, t determine camera pose: R, t

14. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 14 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Perspective n-Point Problem – PnP (1) Calibrated camera –Known K Known points P i in the scene Given n point correspondences –pi Pi–pi Pi What can be measured with one calibrated camera? angle θ between two lines of sight papa pbpb PbPb PaPa d ab θ C

15. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 15 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Perspective n-Point Problem – PnP (2) PnP uses just this information: P3P will give up to 4 solutions P4P is already overdetermined –Perform 4 x P3P –Find consensus papa pbpb PbPb PaPa d ab θ C

16. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 16 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Pose Estimation Tracking In theory, tracking is simple ! –Calibrate your camera (K) –Measure some points P i in the scene (ground truth) –Perform pose estimation in real-time (for each frame) In practice, tracking is a hard problem ! –Point detection –Correspondence –Motion prediction –Occlusion –Unknown scene –…–… Many solutions have been proposed ! Tracking beyond 15 minutes of thought SIGGRAPH 2001 Turorial #15 [Allen, Bishop, Welch] An introduction to the Kalman filter SIGGRAPH 2001 Turorial #8 [Welch, Bishop]

17. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 17 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Tracking Systems (vision-based / hybrid) some of my own contributions (1) Hybrid inside out magnetic + stereo vision [Auer 1999]

18. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 18 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Tracking Systems (vision-based / hybrid) some of my own contributions (2) stereo vision outside in [Ribo ca. 2000]

19. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 19 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Tracking Systems (vision-based / hybrid) some of my own contributions (3) inertial inside out hybrid inertal + vision [many 2000-2004]

20. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 20 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Tracking Systems (vision-based / hybrid) some of my own contributions (4) vision (stereo or mono) inside out speed solves correspondence ! [Mühlmann 2005]

21. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 21 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Our Current View [Schweighofer 2008] stereo vision inside out structure and motion

22. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 22 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz Summary In these four lectures, I gave an introduction to: Projective geometry Perspective cameras Homographies, camera projection matrices, fundamental and essential matrices Algorithms that are typically applied to solve for –Camera calibration –Stereo reconstruction –Camera pose estimation I consider this the basis for further reading in topics including: Vision-based pose tracking Structure and motion analysis (sometimes termed SLAM) Many aspects were, of course, not covered, but would also be important !

23. Institut für Elektrische Meßtechnik und Meßsignalverarbeitung Professor Horst Cerjak, 19.12.2005 23 22.4.2008 Augmented Reality VU 4 Algorithms + Tracking Axel Pinz What could not be covered ? Self calibration (see Pollefeys, absolute conic,…) Bundle adjustment Levenberg-Marquardt The full presentation of algorithms for the estimation of H, P, K, F, … –see the Hartley, Zisserman book for all about multiple view geometry Tracking in general, Kalman filter (two UNC Siggraph 2001Tutorials) Several prominent variants of vision-based tracking algorithms/systems: –KLT –Rapid, RoRapid –Condensation, ICondensation –[Lu, Hager] –[Ansar, Daniilidis] –[Wunsch, Hirzinger] –[Klein, Murray] –… Another reference to Pollefeys: http://www.cs.unc.edu/~marc/tutorial/node159.html interested in more detail ? 2 VO Image based measurement WS 1 LU Image based measurement SS seminar, project, bachelor, diploma, PhD, …