1. Registration for Robotics Kurt Konolige Willow Garage Stanford University Patrick Mihelich JD Chen James Bowman Helen Oleynikova Freiburg TORO group: Giorgio Grisetti Cyrill Stachness Rainer Kummerle

2. Outline Applications Overview of Registration Front end and image matching Visual Odometry Place recognition Global SBA Extensions

3. Applications of Registration Photo tourism [Snavely, Seitz, Szeliski 2006]

4. Applications of Registration Object Modeling [Lovi et al. 2010] [Newcombe and Davison 2010]

5. Applications of Registration Face Tracking [WATSON: Morency 2003]

6. Applications of Registration Visual Odometry [Willow Garage]

7. Applications of Registration Visual SLAM [Willow Garage]

8. Outline Applications Overview of Registration Front end, image matching Visual Odometry Place recognition Global SBA Extensions

9. Registration Elements Incremental Extract keypoints Compute features Match against local views Estimate pose Optimize Global Add new view to map Match against global views [Place recognition] Estimate pose Optimize

10. Mapping Systems MonoSLAM [Davison 2003] Parallel Tracking and Mapping [Klein and Murray 2007] View-based Maps [Konolige et al. 2009] Map type World points Views and world points Views [and implicit world points]

11. Map Representations Map type World points Views and world points Views [and implicit world points] Covariance Matrix over points p0 p1 p2 … Extended Kalman Filter update

12. MonoSLAM [Davison 2003]

13. Map Representations Map type World points Views and world points Views [and implicit world points] Information matrix over views and world points Nonlinear least squares update H  x = -g c0 c1 c2 … p0 p1 p2 … c0 c1 c2 … p0 p1 p2 …

14. Parallel Tracking and Mapping

15. Map Representations Map type World points Views and world points Views [and implicit world points] Information matrix over views c0 c1 c2 … Nonlinear least squares update H  x = -g

16. View-based Maps

17. Comparison of Mapping Methods Map type World points Useful in small spaces - World points for global matching - World point Covar matrix update Not as accurate [Strasdat et al. 2010] Views and world points Useful in small spaces - World points for global matching - View and point Info matrix update Highly accurate Views [and implicit world points] Useful in larger spaces - View matching - View Info matrix update Not as accurate

18. Outline Applications Overview of Registration Front end and image matching Visual Odometry Place recognition Global SBA Extensions

19. Interest points FAST [Rosten and Drummond 2006] Harris SIFT [Lowe 1999] SUSAN, CenSure, MSER, … Adaptive threshold Gridding for spatial diversity

20. Feature matching SIFT / SURF features Planar patches Random-tree signatures [Calonder et al. 2009] … Tracking with motion model Windowed brute-force KD tree nearest-neighbor 930 features354 matched183 inliers

21. Outlier detection and pose estimation RANSAC to estimate pose –3 points for stereo (3D-3D) –3-point PnP for monocular (2D-3D) Outlier rejection Refinement –Least-squares –Reprojection error Scavenging 930 features354 matched183 inliers

22. Outline Applications Overview of Registration Front end and image matching Visual Odometry Place recognition Global SBA Extensions

23. Visual Odometry – Sliding Window 2-view pose estimates are unreliable –Triangulation is narrow-baseline Multi-view estimates are more accurate –Triangulation is wide-baseline –More points, wider baseline => more accuracy Keyframes –Many close-together views don’t add much –Too far apart leads to low inlier count Sliding window of views –Track points as long as possible –~20 – 40 keyframes in window

24. Visual Odometry – Bundle Adjustment Measurement model: For Gaussian distributions ( is covariance) [Gauss-Newton / Normal] Sparseness of H H 20 camera views 5000 features H is (20x6 + 5000x3) 2 Efficient - ~10ms c0 c1 c2 … p0 p1 p2 … c0 c1 c2 … p0 p1 p2 …

25. Visual Odometry examples [courtesy Andrew Comport, INRIA] Outdoor sequence in Versailles 1 m stereo baseline, narrow FOV ~400 m sequence Average frame distance: 0.6 m Max frame distance: 1.1 m

26. 26 Visual Odometry examples [courtesy Andrew Comport, INRIA] Indoor Willow Garage sequence 10 cm stereo, wide FOV ~100 m sequence Average frame distance: 0.3 m

27. Outline Applications Overview of Registration Front end and image matching Visual Odometry Place recognition Global SBA Extensions

28. Place Recognition K-nearest neighbor feature matching across database of images –KD tree [Lowe 1999, Eade and Drummond 2008, Williams et al. 2007] Bag-of-words –visual vocabulary [Stewenius and Nister 2006, Cummins and Newman 2008] DB test DB test

29. Place Recognition: Vocabulary Trees [Nister and Stewenius CVPR06] - “Bag of words” retrieval - Vocab tree created offline - For recognition: - Image keypoints extracted - Tree encodes approximate NN search - Inverted index of images at leaves [Image from Nister and Stewenius CVPR06] [Cummins and Newman ICRA07 Cullmer et al. ACRA08 Fraundorfer et al. IROS07]

30. Place Recognition: Vocabulary Trees Performance on Indoor dataset

31. View-based Maps [Konolige et al. 2009] Performance on Indoor dataset

32. View-based Maps [Konolige et al. 2009] Performance on Indoor dataset

33. Trajectory synthesis

34. Outline Applications Overview of Registration Front end and image matching Visual Odometry Place recognition Global SBA Extensions

35. Comparison of Mapping Methods Map type World points Useful in small spaces - World points for global matching - World point Covar matrix update Not as accurate [Strastdat 2010] Views and world points Useful in small spaces - World points for global matching - View and point Info matrix update Highly accurate Views [and implicit world points] Useful in larger spaces - View matching - View Info matrix update Not as accurate

36. Fast SBA [Konolige BMVC 2010, Jeong et al. CVPR 2010] Take advantage of sparse secondary structure of Hessian Use fast linear solvers –Davis’ CHOLMOD –Block preconditioned conjugate gradient

37. Fast SBA in VSLAM

38. WG Projected Texture Stereo Device Paint the scene with texture from a projector vs. single camera with structured light Advantages: Simple projector Standard algorithms Full frame rates (640x480) Dynamic scenes

39. Fast SBA in Reconstruction

40. Open-Source in ROS sba package for fast SBA frame_common and posest for front-end and pose estimation vocabulary_tree for place recognition vslam_system for Visual Odometry and VSLAM

41. Solved vs. Unsolved Front end –Keypoint matching –Pose estimation (2D and 3D) Visual odometry –Stereo and monocular –Real time Place recognition –Visual scene static –>10K images Large areas –SBA –Frame-based –Stereo and monocular Object modeling –Registration of textured objects –Surface reconstruction Front end –Faster, more robust descripters –Blur, low texture, lines Visual odometry –RGB-D devices (3D + 2D) Place recognition –Dynamic scenes Large areas –Manifolds Object modeling –Untextured objects –Realtime Surface reconstruction