Registration for Robotics Kurt Konolige Willow Garage Stanford University Patrick Mihelich JD Chen James Bowman Helen Oleynikova Freiburg TORO group: Giorgio Grisetti Cyrill Stachness Rainer Kummerle

Outline Applications Overview of Registration Front end and image matching Visual Odometry Place recognition Global SBA Extensions

Applications of Registration Photo tourism [Snavely, Seitz, Szeliski 2006]

Applications of Registration Object Modeling [Lovi et al. 2010] [Newcombe and Davison 2010]

Applications of Registration Face Tracking [WATSON: Morency 2003]

Applications of Registration Visual Odometry [Willow Garage]

Applications of Registration Visual SLAM [Willow Garage]

Outline Applications Overview of Registration Front end, image matching Visual Odometry Place recognition Global SBA Extensions

Registration Elements Incremental Extract keypoints Compute features Match against local views Estimate pose Optimize Global Add new view to map Match against global views [Place recognition] Estimate pose Optimize

Mapping Systems MonoSLAM [Davison 2003] Parallel Tracking and Mapping [Klein and Murray 2007] View-based Maps [Konolige et al. 2009] Map type World points Views and world points Views [and implicit world points]

Map Representations Map type World points Views and world points Views [and implicit world points] Covariance Matrix over points p0 p1 p2 … Extended Kalman Filter update

MonoSLAM

Map Representations Map type World points Views and world points Views [and implicit world points] Information matrix over views and world points Nonlinear least squares update H  x = -g c0 c1 c2 … p0 p1 p2 … c0 c1 c2 … p0 p1 p2 …

Parallel Tracking and Mapping

Map Representations Map type World points Views and world points Views [and implicit world points] Information matrix over views c0 c1 c2 … Nonlinear least squares update H  x = -g

View-based Maps

Comparison of Mapping Methods Map type World points Useful in small spaces - World points for global matching - World point Covar matrix update Not as accurate [Strasdat et al. 2010] Views and world points Useful in small spaces - World points for global matching - View and point Info matrix update Highly accurate Views [and implicit world points] Useful in larger spaces - View matching - View Info matrix update Not as accurate

Outline Applications Overview of Registration Front end and image matching Visual Odometry Place recognition Global SBA Extensions

Interest points FAST [Rosten and Drummond 2006] Harris SIFT [Lowe 1999] SUSAN, CenSure, MSER, … Adaptive threshold Gridding for spatial diversity

Feature matching SIFT / SURF features Planar patches Random-tree signatures [Calonder et al. 2009] … Tracking with motion model Windowed brute-force KD tree nearest-neighbor 930 features354 matched183 inliers

Outlier detection and pose estimation RANSAC to estimate pose –3 points for stereo (3D-3D) –3-point PnP for monocular (2D-3D) Outlier rejection Refinement –Least-squares –Reprojection error Scavenging 930 features354 matched183 inliers

Outline Applications Overview of Registration Front end and image matching Visual Odometry Place recognition Global SBA Extensions

Visual Odometry – Sliding Window 2-view pose estimates are unreliable –Triangulation is narrow-baseline Multi-view estimates are more accurate –Triangulation is wide-baseline –More points, wider baseline => more accuracy Keyframes –Many close-together views don’t add much –Too far apart leads to low inlier count Sliding window of views –Track points as long as possible –~20 – 40 keyframes in window

Visual Odometry – Bundle Adjustment Measurement model: For Gaussian distributions ( is covariance) [Gauss-Newton / Normal] Sparseness of H H 20 camera views 5000 features H is (20x x3) 2 c0 c1 c2 … p0 p1 p2 … c0 c1 c2 … p0 p1 p2 …

Visual Odometry examples [courtesy Andrew Comport, INRIA] Outdoor sequence in Versailles 1 m stereo baseline, narrow FOV ~400 m sequence Average frame distance: 0.6 m Max frame distance: 1.1 m

26 Visual Odometry examples [courtesy Andrew Comport, INRIA] Indoor Willow Garage sequence 10 cm stereo, wide FOV ~100 m sequence Average frame distance: 0.3 m

Outline Applications Overview of Registration Front end and image matching Visual Odometry Place recognition Global SBA Extensions

Place Recognition K-nearest neighbor feature matching across database of images –KD tree [Lowe 1999, Eade and Drummond 2008, Williams et al. 2007] Bag-of-words –visual vocabulary [Stewenius and Nister 2006, Cummins and Newman 2008] DB test DB test

Place Recognition: Vocabulary Trees [Nister and Stewenius CVPR06] - “Bag of words” retrieval - Vocab tree created offline - For recognition: - Image keypoints extracted - Tree encodes approximate NN search - Inverted index of images at leaves [Image from Nister and Stewenius CVPR06] [Cummins and Newman ICRA07 Cullmer et al. ACRA08 Fraundorfer et al. IROS07]

Place Recognition: Vocabulary Trees Performance on Indoor dataset

View-based Maps [Konolige et al. 2009] Performance on Indoor dataset

View-based Maps [Konolige et al. 2009] Performance on Indoor dataset

Trajectory synthesis

Outline Applications Overview of Registration Front end and image matching Visual Odometry Place recognition Global SBA Extensions

Comparison of Mapping Methods Map type World points Useful in small spaces - World points for global matching - World point Covar matrix update Not as accurate [xxx and Davison 2010] Views and world points Useful in small spaces - World points for global matching - View and point Info matrix update Highly accurate Views [and implicit world points] Useful in larger spaces - View matching - View Info matrix update Not as accurate

Fast SBA [Konolige BMVC 2010, Jeong et al. CVPR 2010] Take advantage of sparse secondary structure of Hessian Use fast linear solvers –Davis’ CHOLMOD –Block preconditioned conjugate gradient

Fast SBA in VSLAM

Fast SBA in Reconstruction

Open-Source in ROS sba package for fast SBA frame_common and posest for front-end and pose estimation vocabulary_tree for place recognition vslam_system for Visual Odometry and VSLAM