1. Karel Lebeda, Simon Hadfield, Richard Bowden
2D Or Not 2D: Bridging the Gap Between Tracking and Structure from Motion
Karel Lebeda, Simon Hadfield, Richard Bowden
ACCV 2014

2. TWININGS Testing trackers since 2009
Babenko et al.: Visual Tracking with Online
Multiple Instance Learning. CVPR 2009.

3. LIVER RUN Long-term tracking sequence
Many challenges: extreme length, disappearance, varying illumination,...
No significant viewpoint changes!
Lebeda et al.: Long-Term Tracking
through Failure Cases. VOT 2013.

4. MOTIVATION Real-world sequences contain out-of-plane rotations
Tracker often has to rapidly learn a change in appearance...
But the object does not change!
Only the viewpoint has changed, why not model it explicitly?

5. - bringing together Tracking, SfM & SLAM
TMAGIC
racking,
odelling Reconstruction nd
aussian-process
nference
ombined T M A G I C
- bringing together Tracking, SfM & SLAM

6. Three Basic Principles (I)M A G I C
Tracking:
- given a video-sequence
and a 3D model of the object,
we can estimate camera pose
- model-based 3D tracking, SLAM

7. Three Basic Principles (II)M A G I C
Modelling:
- given a videosequence
and the camera poses,
we can estimate a feature cloud
and model the shape
- Structure from Motion, SLAM
...

8. Three Basic Principles (III)M A G I C
Gaussian-process Inference:
- given a video-sequence and a shape model, we can validate newly detected features

9. Three Basic PrinciplesM A G I C
Combined:
- estimating everything simultaneously

10. Algorithm Overview T M GI 2D TRACKING CAMERA ESTIMATION BUNDLE
ADJUSTMENT
MODEL
TRAINING
FEATURE
GENERATION T M GI

11. Algorithm Overview T M GI 2D TRACKING CAMERA ESTIMATION BUNDLE
ADJUSTMENT
MODEL
TRAINING
FEATURE
GENERATION T M GI

12. 2D Features & Tracking - point features (DoG+HAff)
- line features (LSD)
- tracked independently
- geometric verification
Lebeda et al.: Long-Term Tracking
through Failure Cases. VOT 2013.

13. Algorithm Overview T M GI 2D TRACKING CAMERA ESTIMATION BUNDLE
ADJUSTMENT
MODEL
TRAINING
FEATURE
GENERATION T M GI

14. 3D Features & Camera Pose Estimation
- PnP? PnL?
- optimisation approach
(conditional gradient method)
- fixed during tracking
- updating using bundle adjustment

15. Algorithm Overview T M GI 2D TRACKING CAMERA ESTIMATION BUNDLE
ADJUSTMENT
MODEL
TRAINING
FEATURE
GENERATION T M GI

16. Surface Shape Modelling
- Gaussian Process Regression
- modelling radii
- GP advantages: few training data, non-parametric, probabilistic, no over-fitting

17. Algorithm Overview T M GI 2D TRACKING CAMERA ESTIMATION BUNDLE
ADJUSTMENT
MODEL
TRAINING
FEATURE
GENERATION T M GI

18. Feature Generation & Validation
- tracking:
only a small part of the image
- back-projecting extracted features
- must intersect the shape model
- 3D position initialisation

19. Algorithm Overview T M GI 2D TRACKING CAMERA ESTIMATION BUNDLE
ADJUSTMENT
MODEL
TRAINING
FEATURE
GENERATION T M GI

20. Outputs of the Tracker - camera trajectory - surface shape model
2D trajectory
(segmentation)

21. Results Example

22. Results Example

23. Final Shape Model

24. Results - localisation error & overlap
- on standard sequences comparable to SOTA (22 %)
- on sequences with strong out-of-plane rotation highly superior (58 %)
(out-of-plane rotations beneficial)

25. Summary - explicitly 3D tracker
- combining approaches from tracking, SfM and SLAM
- modelling the object shape as a Gaussian Process
- limitations (= future work):
- non-rigid objects
- long-term tracking, redetection
- dense reconstruction

26. Summary