Parallel Tracking and Mapping for Small AR Workspaces Vision Seminar 2008. 9. 4 (Thu) Young Ki Baik Computer Vision Lab.

References Parallel Tracking and Mapping for Small AR Workspaces Georg Klein, David Murray, ISMAR 2007 Visual Odometry David Nister et. al. , CVPR 2004

Outline What is AR? Previous works Proposed methods Demo Conclusion

What is AR? A AR : Augment Reality Full 3D? (O) Expensive? (O) ugument reality(AR) uses the real scene as the background, and makes applications with putting 3D objects to the background. Since the cost of augment reality is not as expensive as full 3D virtual reality(VR), AR has been very popular research topic. A Full 3D? (O) Expensive? (O) Full 3D (X) Expensive? (X)

Demonstration

The Aim AR with a hand-held camera Visual Tracking provides registration

The Aim AR with a hand-held camera Visual Tracking provides registration Track without prior model of world

The Aim AR with a hand-held camera Visual Tracking provides registration Track without prior model of world Challenges Speed Accuracy Robustness Interaction with real world

Existing attempts : SLAM SLAM : Simultaneous Localization and Mapping can use many different types of sensor to acquire observation data used in building the map such as laser rangefinders, sonar sensors and cameras. Well-established in robotics (using a rich array of sensors) Demonstrated with a single hand-held camera by Davison at 2003 (Mono-SLAM). Mono-SLAM was applied to AR system at 2004.

Existing attempts : Model based tracking Model-based tracking is More robust More accurate Proposed by Lepetit et. al. at ISMAR 2003

Frame by Frame SLAM Why? is SLAM fundamentally harder? Many DOF Time One frame Find features Many DOF Update camera pose and entire map Draw graphics

Frame by Frame SLAM SLAM Proposed approach Updating entire map every frame is so expensive!!! Needs “sparse map of high-quality features” - A. Davison Proposed approach Use dense map (of low quality features) Don’t update the map every frame : Keyframes Split the tracking and mapping into two threads

Parallel Tracking and Mapping Proposed method - Split the tracking and mapping into two threads Time Thread #2 Mapping Update map One frame Thread #1 Tracking Find features Update camera pose only Simple & easy Draw graphics

Parallel Tracking and Mapping Tracking thread: Responsible estimation of camera pose and rendering augmented graphics Must run at 30 Hz Make as robust and accurate as possible Mapping thread: Responsible for providing the map Can take lots of time per key frame Make as rich and accurate as possible

Tracking thread Overall flow Map Pre-process frame Project points Measure points Measure points Update Camera Pose Update Camera Pose Coarse stage Fine stage Draw Graphics

Pre-process frame Make for pyramid levels 80x60 640x480 320x240

Pre-process frame Make for pyramid levels Detect Fast corners E. Rosten et al (ECCV 2006) 80x60 640x480 320x240 160x120

Project Points Use motion model to update camera pose Constant velocity model Estimated current Pt+1 Previous pos Pt ∇t’ Previous pos Pt-1 ∇t Vt =(Pt – Pt-1)/∇t Pt+1=Pt+∇t’(Vt)

Project Points Choose subset to measure ~50 1000 ~ 50 biggest features for coarse stage 1000 randomly selected for fine stage ~50 1000 80x60 640x480 320x240 160x120

Measure Points Generate 8x8 matching template (warped from source key-frame:map) Search a fixed radius around projected position Use Zero-mean SSD Only search at Fast corner points

? Update caemra pose 6-DOF problem Obtain by SFM (Three-point algorithm) ?

Dray graphics What can we draw in an unknown scene? Assume single plane visible at start Run VR simulation on the plane

Stereo Initialization Mapping thread Overall flow Stereo Initialization Tracker Wait for new key frame Add new map points Optimize map Map maintenance

Stereo Initialization Use five-point-pose algorithm D. Nister et. al. 2006 Requires a pair of frames and feature correspondences Provides initial map User input required: Two clicks for two key-frames Smooth motion for feature correspondence

Wait for new key frame Key frames are only added if : There is a sufficient baseline to the other key frame Tracking quality is good Key frame (4 level pyramid images and its corners) When a key frame is added : The mapping thread stops whatever it is doing All points in the map are measured in the key frame New map points are found and added to the map

Add new map points Want as many map points as possible Check all maximal FAST corners in the key frame : Check score Check if already in map Epipolar search in a neighboring key frame Triangulate matches and add to map Repeat in four image pyramid levels

Optimize map Use batch SFM method: Bundle Adjustment Adjusts map point positions and key frame poses Minimize reprojection error of all points in all keyframes (or use only last N key frames)

Map maintenance When camera is not exploring, mapping thread has idle time Data association in bundle adjustment is reversible Re-attempt outlier measurements Try measure new map features in all old key frames

Comparison to EKF-SLAM More Accurate More robust Faster tracking < SLAM based AR Proposed AR

System and Results Environment Tracking speed Desktop PC (Intel Core 2 Duo 2.66 GHz) OS : Linux Language : C++ Tracking speed Total 19.2 ms Key frame preparation 2.2 ms Feature Projection 3.5 ms Patch search 9.8 ms Iterative pose update 3.7 ms

System and Results Mapping scalability and speed 150 key frames Practical limit 150 key frames 6000 points Bundle adjustment timing Key frames 2-49 50-99 100-149 Local Bundle Adjustment 170 ms 270 ms 440 ms Global Bundle Adjustment 380 ms 1.7 s 6.9 s

Demonstration

Remaining problem Outlier management Still brittle in some scenario Repeated texture Passive stereo initialization Occlusion problem Relocation problem

Conclusion Conclusion Contribution Opinion Parallel tracking and mapping process are presented using multi-thread. Contribution Visual odometry system was well presented. Overcome computation by multi-thread Opinion The proposed algorithm can be applied to our research Navigation system 3D tracking system

Q & A