1. Parallel Tracking and Mapping for Small AR Workspaces Vision Seminar
(Thu)
Young Ki Baik
Computer Vision Lab.

2. References Parallel Tracking and Mapping for Small AR Workspaces
Georg Klein, David Murray, ISMAR 2007
Visual Odometry
David Nister et. al. , CVPR 2004

3. Outline
What is AR?
Previous works
Proposed methods
Demo
Conclusion

4. What is AR? A AR : Augment Reality Full 3D? (O) Expensive? (O)
ugument reality(AR) uses the real scene as the background, and
makes applications with putting 3D objects to the background.
Since the cost of augment reality is not as expensive as full 3D virtual reality(VR), AR has been very popular research topic. A
Full 3D? (O) Expensive? (O)
Full 3D (X)
Expensive? (X)

5. Demonstration

6. The Aim AR with a hand-held camera
Visual Tracking provides registration

7. The Aim AR with a hand-held camera
Visual Tracking provides registration
Track without prior model of world

8. The Aim AR with a hand-held camera
Visual Tracking provides registration
Track without prior model of world
Challenges
Speed
Accuracy
Robustness
Interaction with real world

9. Existing attempts : SLAM
SLAM : Simultaneous Localization and Mapping
can use many different types of sensor to acquire observation data used in building the map such as laser rangefinders, sonar sensors and cameras.
Well-established in robotics (using a rich array of sensors)
Demonstrated with a single hand-held camera by Davison at 2003 (Mono-SLAM).
Mono-SLAM was applied to
AR system at 2004.

10. Existing attempts : Model based tracking
Model-based tracking is
More robust
More accurate
Proposed by Lepetit et. al.
at ISMAR 2003

11. Frame by Frame SLAM Why? is SLAM fundamentally harder? Many DOF Time
One frame
Find features
Many DOF
Update camera pose and entire map
Draw graphics

12. Frame by Frame SLAM SLAM Proposed approach
Updating entire map every frame is so expensive!!!
Needs “sparse map of high-quality features”
- A. Davison
Proposed approach
Use dense map (of low quality features)
Don’t update the map every frame : Keyframes
Split the tracking and mapping into two threads

13. Parallel Tracking and Mapping
Proposed method
- Split the tracking and mapping into two threads
Time
Thread #2
Mapping
Update map
One frame
Thread #1
Tracking
Find features
Update camera pose only
Simple & easy
Draw graphics

14. Parallel Tracking and Mapping
Tracking thread:
Responsible estimation of camera pose and rendering augmented graphics
Must run at 30 Hz
Make as robust and accurate as possible
Mapping thread:
Responsible for providing the map
Can take lots of time per key frame
Make as rich and accurate as possible

15. Tracking thread Overall flow Map Pre-process frame Project points
Measure points
Measure points
Update Camera Pose
Update Camera Pose
Coarse stage
Fine stage
Draw Graphics

16. Pre-process frame Make for pyramid levels 80x60 640x480 320x240

17. Pre-process frame Make for pyramid levels Detect Fast corners
E. Rosten et al (ECCV 2006)
80x60
640x480
320x240
160x120

18. Project Points Use motion model to update camera pose
Constant velocity model
Estimated current Pt+1
Previous pos Pt
∇t’
Previous pos Pt-1 ∇t
Vt =(Pt – Pt-1)/∇t
Pt+1=Pt+∇t’(Vt)

19. Project Points Choose subset to measure ~50 1000
~ 50 biggest features for coarse stage
1000 randomly selected for fine stage
~50
1000
80x60
640x480
320x240
160x120

20. Measure Points
Generate 8x8 matching template (warped from source key-frame:map)
Search a fixed radius around projected position
Use Zero-mean SSD
Only search at Fast corner points

21. ? Update caemra pose 6-DOF problem
Obtain by SFM (Three-point algorithm) ?

22. Dray graphics What can we draw in an unknown scene?
Assume single plane visible at start
Run VR simulation on the plane

23. Stereo Initialization
Mapping thread
Overall flow
Stereo Initialization
Tracker
Wait for new key frame
Add new map points
Optimize map
Map maintenance

24. Stereo Initialization
Use five-point-pose algorithm
D. Nister et. al. 2006
Requires a pair of frames and feature correspondences
Provides initial map
User input required:
Two clicks for two key-frames
Smooth motion for feature correspondence

25. Wait for new key frame Key frames are only added if :
There is a sufficient baseline to the other key frame
Tracking quality is good
Key frame (4 level pyramid images and its corners)
When a key frame is added :
The mapping thread stops whatever it is doing
All points in the map are measured in the key frame
New map points are found and added to the map

26. Add new map points Want as many map points as possible
Check all maximal FAST corners in the key frame :
Check score
Check if already in map
Epipolar search in a neighboring key frame
Triangulate matches and add to map
Repeat in four image pyramid levels

27. Optimize map Use batch SFM method: Bundle Adjustment
Adjusts map point positions and key frame poses
Minimize reprojection error of all points in all keyframes (or use only last N key frames)

28. Map maintenance
When camera is not exploring, mapping thread has idle time
Data association in bundle adjustment is reversible
Re-attempt outlier measurements
Try measure new map features in all old key frames

29. Comparison to EKF-SLAM
More Accurate
More robust
Faster tracking
<
SLAM based AR
Proposed AR

30. System and Results Environment Tracking speed
Desktop PC (Intel Core 2 Duo 2.66 GHz)
OS : Linux
Language : C++
Tracking speed
Total
19.2 ms
Key frame preparation
2.2 ms
Feature Projection
3.5 ms
Patch search
9.8 ms
Iterative pose update
3.7 ms

31. System and Results Mapping scalability and speed 150 key frames
Practical limit
150 key frames
6000 points
Bundle adjustment timing
Key frames
2-49
50-99
Local Bundle Adjustment
170 ms
270 ms
440 ms
Global Bundle Adjustment
380 ms
1.7 s
6.9 s

32. Demonstration

33. Remaining problem Outlier management Still brittle in some scenario
Repeated texture
Passive stereo initialization
Occlusion problem
Relocation problem

34. Conclusion Conclusion Contribution Opinion
Parallel tracking and mapping process are presented using multi-thread.
Contribution
Visual odometry system was well presented.
Overcome computation by multi-thread
Opinion
The proposed algorithm can be applied to our research
Navigation system
3D tracking system

35. Q & A