1. Lecture 10 Causal Estimation of 3D Structure and Motion
MASKS © 2004
Invitation to 3D vision

2. machine to INTERACT with the environment
VISION as a SENSOR
machine to INTERACT with the environment
NEED to estimate relative 3D MOTION
3D SHAPE (TASK)
REAL-TIME
CAUSAL processing
representation of SHAPE
(only supportive of representation of motion)
POINT-FEATURES
MASKS © 2004
Invitation to 3D vision

3. TRADEOFFS SFM EASY CORRESPONDENCE HARD/IMPOSSIBLE LARGE BASELINE
MASKS © 2004
Invitation to 3D vision

4. TRADEOFFS SFM CORRESPONDENCE LARGE BASELINE EASY HARD/IMPOSSIBLE SMALL
MASKS © 2004
Invitation to 3D vision

5. TRADEOFFS SFM CORRESPONDENCE LARGE BASELINE EASY HARD/IMPOSSIBLE
GLOBAL
SMALL
BASELINE
HARD/IMPOSSIBLE
EASY
LOCAL
MASKS © 2004
Invitation to 3D vision

6. WHAT DOES IT TAKE ? SFM CORRESPONDENCE LARGE BASELINE EASY
HARD/IMPOSSIBLE
GLOBAL
SMALL
BASELINE
HARD/IMPOSSIBLE
EASY
LOCAL
INTEGRATE
visual information
OVER TIME
OCCLUSIONS!
MASKS © 2004
Invitation to 3D vision

7. Setup and notation MASKS © 2004 Invitation to 3D vision
Temporal evolution:
MASKS © 2004
Invitation to 3D vision

8. Structure and motion as a filtering problem
Given measurements of the “output” (feature point positions)
Given modeling assumptions about the “input” (acceleration = noise)
Estimate the “state” (3D structure, pose, velocity)
MASKS © 2004
Invitation to 3D vision

9. Model is non-linear (output map = projection)
Difficulties …
Model is non-linear (output map = projection)
State-space is non-linear! (SE(3))
Noise: need to specify what we mean by “estimate”
Even without noise: model is not observable!
MASKS © 2004
Invitation to 3D vision

10. Observability
Equivalent class of state-space trajectories generate the same measurements
Fix, e.g., the direction of 3 points, and the depth of one point (Gauge transformation)
MASKS © 2004
Invitation to 3D vision

11. Local coordinatization of the state space
MASKS © 2004
Invitation to 3D vision

12. Minimal realization Now it looks very much like:
And we are looking for (a point statistic of):
MASKS © 2004
Invitation to 3D vision

13. For single rigid body/static scene, expect unimodal posterior
EFK vs. particle filter?
For single rigid body/static scene, expect unimodal posterior
No need to estimate entire density; point estimate suffices
Robust (M-) version of EKF works well in practice …
… and in real time for a few hundred feature points
MASKS © 2004
Invitation to 3D vision

14. Adding/removing features (subfilters)
In practice …
Adding/removing features (subfilters)
Multiple motions/outliers (M-filter, innovation tests)
Tracking drift (reset with wide-baseline matching)
Switching the reference features (hard! Causes unavoidable global drift)
Global registration (maintain DB of lost features)
MASKS © 2004
Invitation to 3D vision

15. MASKS © 2004
Invitation to 3D vision