Karel Lebeda, Simon Hadfield, Richard Bowden 2D Or Not 2D: Bridging the Gap Between Tracking and Structure from Motion Karel Lebeda, Simon Hadfield, Richard Bowden ACCV 2014

TWININGS Testing trackers since 2009 Babenko et al.: Visual Tracking with Online Multiple Instance Learning. CVPR 2009.

LIVER RUN Long-term tracking sequence Many challenges: extreme length, disappearance, varying illumination,... No significant viewpoint changes! Lebeda et al.: Long-Term Tracking through Failure Cases. VOT 2013.

MOTIVATION Real-world sequences contain out-of-plane rotations Tracker often has to rapidly learn a change in appearance... But the object does not change! Only the viewpoint has changed, why not model it explicitly?

- bringing together Tracking, SfM & SLAM TMAGIC racking, odelling Reconstruction nd aussian-process nference ombined T M A G I C - bringing together Tracking, SfM & SLAM

Three Basic Principles (I) M A G I C Tracking: - given a video-sequence and a 3D model of the object, we can estimate camera pose - model-based 3D tracking, SLAM

Three Basic Principles (II) M A G I C Modelling: - given a videosequence and the camera poses, we can estimate a feature cloud and model the shape - Structure from Motion, SLAM ...

Three Basic Principles (III) M A G I C Gaussian-process Inference: - given a video-sequence and a shape model, we can validate newly detected features

Three Basic Principles M A G I C Combined: - estimating everything simultaneously

Algorithm Overview T M GI 2D TRACKING CAMERA ESTIMATION BUNDLE ADJUSTMENT MODEL TRAINING FEATURE GENERATION T M GI

Algorithm Overview T M GI 2D TRACKING CAMERA ESTIMATION BUNDLE ADJUSTMENT MODEL TRAINING FEATURE GENERATION T M GI

2D Features & Tracking - point features (DoG+HAff) - line features (LSD) - tracked independently - geometric verification Lebeda et al.: Long-Term Tracking through Failure Cases. VOT 2013.

Algorithm Overview T M GI 2D TRACKING CAMERA ESTIMATION BUNDLE ADJUSTMENT MODEL TRAINING FEATURE GENERATION T M GI

3D Features & Camera Pose Estimation - PnP? PnL? - optimisation approach (conditional gradient method) - fixed during tracking - updating using bundle adjustment

Algorithm Overview T M GI 2D TRACKING CAMERA ESTIMATION BUNDLE ADJUSTMENT MODEL TRAINING FEATURE GENERATION T M GI

Surface Shape Modelling - Gaussian Process Regression - modelling radii - GP advantages: few training data, non-parametric, probabilistic, no over-fitting

Algorithm Overview T M GI 2D TRACKING CAMERA ESTIMATION BUNDLE ADJUSTMENT MODEL TRAINING FEATURE GENERATION T M GI

Feature Generation & Validation - tracking: only a small part of the image - back-projecting extracted features - must intersect the shape model - 3D position initialisation

Algorithm Overview T M GI 2D TRACKING CAMERA ESTIMATION BUNDLE ADJUSTMENT MODEL TRAINING FEATURE GENERATION T M GI

Outputs of the Tracker - camera trajectory - surface shape model 2D trajectory (segmentation)

Results Example

Results Example

Final Shape Model

Results - localisation error & overlap - on standard sequences comparable to SOTA (22 %) - on sequences with strong out-of-plane rotation highly superior (58 %) (out-of-plane rotations beneficial)

Summary - explicitly 3D tracker - combining approaches from tracking, SfM and SLAM - modelling the object shape as a Gaussian Process - limitations (= future work): - non-rigid objects - long-term tracking, redetection - dense reconstruction

Summary